The couple's building work at their Oxfordshire home has irked neighbours over the years and their bid to keep a pergola and gazebo in their garden was refused by the council.