In an extract from Tom Bower's unauthorised biography The Intimate Life Of Simon Cowell - published by The Sun newspaper over the weekend - it was claimed the pair were "genuinely in love", despite denying romance rumours in 2008.
With neither SiCo or Minogue commenting on Bower's accusations, PR guru Clifford has now set the record straight, telling Exclusive TV: "Yes - he did have a relationship with Dannii."
"Simon was on his own at the time and she was on her own too. There was a natural chemistry between them. It's not something he wanted to come out but it has - there is nothing unusual about it at all."
He was also quick to jump to Cowell's defence, adding that the 52-year old has never before discussed his personal life in the press, stating: "Everyone who knows Simon knows he keeps his private life as private as possible. But [it's] the truth."
Slamming Bower, Clifford added: "It was Simon's choice, but my attitude is you don't talk to someone like Tom Bower, he's only really looking for the controversial stuff. It was a mistake."
According to The Metro, Max also revealed that Cowell was forced to call Minogue to explain himself after his revelations about their relationship were made public in Bower's book.
Despite the reports, it seems there are no hard feelings between Dannii and Simon after the 40-year old singer was spotted backstage at Cowell's boybandOne Direction's gig in Melbourne, Australia on Tuesday night.