"If these trades workers lose their jobs, we may never get them back," Pierre Poilievre said, pointing to growing layoffs in homebuilding