The former presenter of The X Factor has defended the ITV show after a flurry of mistreatment allegations