No one talks like a human being in Mike Flanagan’s ambitious but confused three-part tale, which stars everyone from Chiwetel Ejiofor to Mark Hamill