Olly who was runner-up in 2009 is tipped to replace Louis Walsh on the panel or be added as a fifth judge according to the Star on Sunday.
Other options being discussed include the popular singer replacing, or joining, Dermot O’Leary as the main show presenter.
Simon Cowell, and X Factor producers have already had several meetings to discuss a major makeover.
Our source revealed: “Simon has been incredibly vocal that The X Factor needs to see some major changes.
“It is too early to confirm what they are yet but one thing is for certain – everybody wants Olly to be a part of the show.
“Olly’s become a huge in- ternational star off the back of the show, he’s well liked from his time presenting the Xtra Factor and is massively popular with the all-important younger fans.
“X Factor producers and ITV bosses are keen for him to replace Louis. “They think Olly would be a much bigger ratings hit than Louis.
“If Simon returns again this year – which he is expected to – Louis is unlikely to be sacked, although the music mogul is keen on a major role for Olly.
“Replacing Dermot has also been suggested, which would certainly be a major shock as he is such a popular host.
“Simon has never been scared to axe people out of the blue though and Dermot knows no one on the show is safe.”
The source added that Olly, and Dermot, could co-host the show, saying: “It is much more likely a judging role will be created, though.”
