‘I got her phone number that night and we’ve been in each other’s pockets ever since,’ ‘Pointless’ star said