A friend of the Prince of Wales said he no longer cares what his brother says about the Royal Family to the public.