The popular presenting duo admitted there was a point when they were no longer enjoying the experience and seriously considered walking away