Socialite Lady Victoria Hervey, who previously dated Andrew, has made her opinion on Giuffre's posthumous memoir very clear