The former Top Gear host says the past year on Diddly Squat Farm was so tough he had to ‘send the cameras away’