After pictures emerged of the 17-year old heartthrob leaving Flack's North London home where he spent the night, his mother Anne is said to be upset her son has fallen for Caroline - who is fifteen years older than him.
A source told Star magazine: "Anne doesn't want Harry to end up getting hurt. Caroline is a worldly woman, who is never short of male attention."
They added: "Anne just feels Harry needs to stick to girls his own age and says Caroline should back off!"
The Xtra Factor host recently addressed her romance with Harry, telling Now magazine: "I feel like I shouldn’t have to worry about what I do. But it’s a social thing that people aren’t accepting of big age gaps."
“I keep thinking, ‘What have I done wrong?’ But I haven’t done anything wrong. What’s hard for me to get my head around is people saying it’s disgusting. I don’t think it is.”
The 32-year old continued: "Some people tend to see the negative immediately and that’s what I find strange. If two people like each other and get on, why does anyone else find a negative in it?"
“I know it’s human nature for everyone to gossip. But why say it’s bad? No one’s being hurt.”
Flack has since received death threats from Harry's devoted 1D fans, with Styles telling her she should ignore the online abuse.
Watch MTV News on the hour every hour on MTV - Sky Channel 126 and Virgin Channel 134