The sitcom star is thought to split his time between London and a quaint Scottish village, with which he has a 'very special relationship'