“For too long, a small but powerful group has dominated the debate about Lambeth’s future,” an open letter written to residents has said