Scammers have swindled money from one in five adults in the past year, a worrying report today reveals.