Simon Cowelland show execs worried about ending up with "a Steve Brookstein-style embarrassment".
A source said: "It's a shame because he's so sweet. They're determined to get him kicked off the show. It's crazy. The show is in a spin and Cowell is really panicking. They will be trying anything now to get people to not like this guy."
Meanwhile the line-up for the show's arena tour has been announced - and it seems only the last seven acts will be playing to crowds of thousands.
That means James Arthur, Ella Henderson, Jahmene Douglas, Rylan Clark, Christopher Maloney and groups Union J and District3 have got the nod - even though there were reports that a spot had been saved for Lucy Spraggan.
The tour starts toward the end of January and will visit Brighton, Cardiff, Nottingham, Liverpool, London, Birmingham, Sheffield, Newcastle, Glasgow, Aberdeen, Dublin and Belfast.
Several of the stops will feature afternoon performances as well as evening ones.

[uuid=52440b59-a391-4181-80ec-003ec4154c8e]THE X FACTOR IS BACK FOR 2012! CHECK OUT PICS HERE[/uuid]