Why Charlie Wilson Could’ve Used His ‘Masked Singer’ Costume at Coachella (Exclusive)

Even after five decades in the music industry, Charlie Wilson found The Masked Singer to be vocally challenging.

“They had me singing it upstairs and they could see if I could sing it. I said, ‘Well, can you bring it down some? It’s a little high,’” Wilson, 71, tells PEOPLE of testing his range as Ugly Sweater. “I think they got tired of that. It’s like, ‘No, you got to sing these songs in the same key everybody else is singing it in.’ So I was like, ‘OK.'”

The co-founder of The Gap Band was eliminated on Queen Night on April 17 after singing “I Want to Break Free.”

“I wish I could have went all the way, but it just didn’t work that way for me” he says. “And I was fine with it. Somebody got to bite the dust, so it looks like to me, ‘Ugly Sweater, you’re the one going to bite the dust.’ ”

Away from stage, “I wouldn’t wear an ugly sweater,” the 13-time Grammy nominee jokes.

Wilson, who released his latest single “Superman” in December, shares which past Masked Singer winner inspired him to go on the show, how it felt when Robin Thicke guessed his identity and how his wife of 29 years, Mahin, pushed back against Kanye West in the recording studio.

What appealed to you about doing The Masked Singer at this point in your career?

When I did see Masked Singer back in the day, it was when T-Pain won the whole thing. I was so excited for him. When they had him sing again, I was like, “Oh man, I’ve never heard that voice actually before, except for when he was writing a song for me back in the day.” But I’d almost forgot it without the auto-tune on it. So it looked like it was a lot of fun and I wanted to do it too.

How did performing as Ugly Sweater differ from what you’re used to?

The ugly sweater costume was so, so heavy. They had to take some of the hair out of the top of it, the feathers, because I think each one of those feathers weighed 20 lbs. or something like that. If I bent down, it was like I was going to fall over. The suit weighed about 200 and something lbs. I was already weighing 180 lbs., plus the 200 lbs., that was almost 300 lbs. I barely could walk.

What did you make of the guesses the judges made for you, which included Smokey Robinson, Ronald Isley, Eddie Levert and Aaron Neville?

I was in there laughing.

And then Robin guessed you pretty early on.

When I started singing, Robin Thicke said, “That’s Charlie Wilson right there.” He said, “I know his voice. I’ve studied him when I was a kid and I know his voice like the back of my hand, and I know that’s Charlie Wilson’s voice right there.” And I was like, “Wow. I’m toast.”

Had you been getting messages from friends and family members who suspected you were Ugly Sweater?

Oh, no. I wouldn’t tell them anyway. I’m looking at the YouTube, I’m looking at everybody saying, “Charlie Wilson, Charlie Wilson, Charlie Wilson, Charlie Wilson.” I’m telling my friends and my family members, “I don’t know because I don’t watch The Masked Singer. I watched it a long time ago and that’s when T-Pain won it. Everybody’s like, “Well, they’re saying it’s you.” I said, “Well, let them guess.”

In your clue package, there were a lot of references to some of the artists you’ve worked with throughout your career. Who surprised you the most in terms of being different from their public persona?

Kanye. He was the one that shocked me the most. Because he used to have a kind of a situation when you’d come to the studio, everybody had to be suited and booted. All of the engineers had to be dressed in suit and ties and all of that stuff. So I came in there breaking rules. I didn’t get nothing about none of that. But he allowed me. I had my phone and nobody’s supposed to have a phone. Nobody was supposed to ever record, and I was up in there recording. And my wife shocked him. She’s very opinionated. She said whatever she wanted to say, and sometimes he didn’t like it, but he couldn’t say nothing because I was sitting there. He got to remember that’s my wife and he couldn’t say too many things. So we ended up being really, really, really close.

And it was the same thing with Snoop Dogg. They was trying to figure out, “Man, who is this lady, man, telling us we can’t smoke and can’t do this here?” Because I was recovering the first time I got with Snoop from drugs and stuff.

Did you enjoy performing with Tyler, the Creator at Coachella during Weekend One?

It was amazing. It was just freezing. I was so cold and I was so high up above the audience. That wind was blowing like about 25, 30 miles an hour and plus, like 39 or 40 degrees. I was just shaking and couldn’t get my lyrics, couldn’t get my phone out. It was rough. Now I know if I ever go back there again, I know to have on big turtlenecks and my big pants and some boots, scarves and a big coat. Because those tents that they had, man, didn’t work for me. That cold air [was] taking my throat away. It’s like singing in a refrigerator. If I’d had the Ugly Sweater on, I could have came out there and wrecked the place!

You mentioned on the show how your wife didn’t know who you were when you two first met. What has it meant to have her by your side?

My wife, she’s Persian. I used to hear, “Charlie Wilson’s got a white wife.” My wife is not white, she’s Persian! And she didn’t know anything about me or that I was a singer. As a matter of fact, her sisters in Iran at the time thought I was as big as Michael Jackson, because they didn’t know anything about me either. They only knew who Michael Jackson was. I ended up with my wife and she was the one to walk me through the highs and lows. She don’t stand for no BS. So I’m happy. I’m having a good time being who I am, and I’m not trying to be anybody else.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

The Masked Singer airs Wednesdays at 8 p.m. ET on Fox.