The Irishman's decision to take last weekend's sing-off to deadlock meant Carolynne Poole was sent home while Rylan remains to fight another day.
But when images emerged of a producer whispering in Louis' ear before the result, accusations of a fix quickly followed.
The long-standing judge has now defended his dithering in an open diary published on the Mail Online.
He said of Sunday's show: "I jump for joy when all my acts go through. I'm not sure who I'm going to choose between Rylan Clark and Carolynne Poole.
"Producer Richard Holloway comes over to talk to us before they perform. He does that every week to tell us what order to go in when we give our verdict because they have to get the cameras set up right. He tells me I will be last."
Louis goes on to admit he feels like he is "looking stupid, like a rabbit in the headlights" as host Dermot O'Leary pushed him for a decision, but added: "That's the thing about the X Factor, it's live, it's real, anything can happen."
The music manager also revealed he woke up to "a huge fuss brewing" the following morning.
He wrote: "People are Tweeting that last night was a fix, and a picture is published showing Richard talking to me. The story is that he's telling me how to vote - but that's not true.
"Not only has no one ever told any of us what to do, but that picture was taken before the artists sang, so no one could have known how their performance was going to be judged."
Watch MTV News on the hour every hour on MTV - Sky Channel 126 and Virgin Channel 134