One particular fruit that's beloved by Brits is destroying the environment so refuse to eat them, says the gardening expert