The ITV series hosted by Holly Willoughby and Stephen Mulhern has seen a decline in audience, losing over a million viewers in the past two years