Following the news that host Steve Jones and judges Paula Abdul and Nicole Scherzinger had been axed ahead of the second season, rumours have been rife over who will replace the trio - with Queen B and Mariah Carey tipped to land roles on the panel.
However, during an interview with Entertainment Weekly, Cowell was keen to state that he has no plans to bring in some of the biggest names around to find the next American popstar.
The music mogul explained: "When we first launched [Pop] Idol in the UK years ago, we didn't think for one second there had to be an A-list star on the panel.
"Everyone on the panel was really unknown. Firstly, they are incredibly expensive - that is millions you lose on something else. Secondly, they have to make the time commitment and on my show, they have to work five days a week."
He continued: "Thirdly, they have to be interesting… I'm not playing this game where we have to compete on the level of A-lister stars… you can lose focus on what the show is all about, and at the end of the day the most important thing is the contestants."
Speaking of the speculation surrounding Bey and Mariah's involvement, Cowell claimed: "I read that online that I offered [Beyoncé] $100 million (£63 million) a year. The budget for the show is $100 million.
"That would never, ever happen. It's absolute nonsense… Even if you had that kind of money. I wouldn't spend $100 million on one person.
"I would love [Mariah]. But you can't expect someone who's just had two kids to make the kind of commitment you would need on this show."
In other related news, Nicole Scherzinger has revealed she had "no regrets" when it comes to leaving the talent show behind her.
In an interview with Access Hollywood, the former Pussycat Dolls singer said: "I had a wonderful experience on the show. I loved being a part of my X Factor family.
"Nothing could obviously prepare me for everything I had to go through, all the emotions and stuff, but I have no regrets. I've always believed in never looking back and just moving forward and I've got an amazing year to look forward to.
"What happened? Well, all I know is that I gave 110% to that show. I know everybody's talking about it, they're saying good, bad, they love me or hate me, but the one thing you can't say is that I didn't give it my all."
Unlike Nicole, ex-Def Jam boss L.A. Reid survived The X Factor USA shake-up and will return alongside SiCo for this year's series of the Fox show.
Watch MTV News on the hour every hour on MTV - Sky Channel 126 and Virgin Channel 134