The Conservative leader said recent departures from the party were ‘not about policy differences or ideology, they are about character.’