The Defense Secretary has been fighting to defend himself since the fresh allegations came to light on Sunday