Knocking down the long-derelict listed building is considered "the only feasible option" after it was served with a danger notice.