David Harbour was accused of having an affair with costume designer Natalie Tippett aka ‘Madeline’ while he was married to Lily Allen