Of all the public criticism levelled at the Duke of Sussex in recent days, the comments from his old friend Ben Goldsmith were possibly the most significant.