The Duke of Sussex spent less than an hour with his father in Clarence House where it is understood they had a private tea