The Earthquake hitmaker - who is the only non-talent show act to be signed to Cowell's SyCo record label - was quoted by The Metro as saying that he would never be a judge on the series following Kelly Rowland's exit.
When asked by the newspaper if he would ever consider joining the panel, Lab apparently exclaimed: "What do you think I should do? For me, I think it’s absolute s**t."
The 23-year old is then said to have stated that The X Factor was nothing more than a way "to get a couple more Twitter fans", before adding he would be an "idiot" if he signed up to join the ITV1 show.
However, Lab has since commented on The Metro's article, telling his fans on Twitter that his remarks were taken out of context.
He tweeted: "I've gotta give it to the metro paper for there amazing ability to reconstruct words in controversial bulls**t…
"Come on now why would I s**t on my own family? Lesson learned. It was a joke ha ha #detentionforlab."
Labrinth added: "Come on man I got my break a different way but The X Factor gives people a chance and U gotta respect that still."