Now The Sun claims that his wife has flown to Australia and is demanding to know why ITV left him in the jungle for so long.
A source told the paper: "She couldn’t believe how much he'd changed after 10 days in the jungle. He looks really unwell and vulnerable.
"He's lost a lot of weight and is exhausted. Doctors have prescribed lots of rest until he recovers. Anne-Marie was hugely supportive of Brian's decision to go on the show, but now you can't blame her wishing he hadn't.
"She wants to know how this happened and why bosses did not intervene earlier."
Brian, 51, has a well-documented history of alcohol abuse and depression - and got upset following a row with boxer David Haye over the campfire shortly before his exit.
The source added: "We pride ourselves on ensuring celebrities are equipped for the challenge, but it seems one slipped through the net."
And ITV told the paper: "The safety and welfare of our contestants is of paramount importance. All contestants are assessed by independent medical experts.
"We also have contact where necessary with the celebrities' GPs. Medical personnel are also available 24 hours a day to provide support if it is needed."
Meanwhile Hugo Taylor from Made In Chelsea was popular with his campmates after winning nine out of 11 meals in the Savage Sewer task.
He had been voted to undergo the Bushtucker Trial by the public, giving Helen Flanagan a rare day off.