The health secretary insisted the number of forced removals ‘should be low’ but admitted he was not comfortable with the plan