The historic castle has been visited by celebrities like Gok Wan and Gordon Ramsay and featured on TV