We report a 4.8$\sigma$ measurement of the cross-correlation signal between the cosmic microwave background (CMB) lensing convergence reconstructed from measurements of the CMB polarization made by the Polarbear experiment and the infrared-selected galaxies of the Herschel-ATLAS survey. This is the first measurement of its kind. We infer a best-fit galaxy bias of , corresponding to a host halo mass of at an effective redshift of z $\sim$ 2 from the cross-correlation power spectrum. Residual uncertainties in the redshift distribution of the submillimeter galaxies are subdominant with respect to the statistical precision. We perform a suite of systematic tests, finding that instrumental and astrophysical contaminations are small compared to the statistical error. This cross-correlation measurement only relies on CMB polarization information that, differently from CMB temperature maps, is less contaminated by galactic and extragalactic foregrounds, providing a clearer view of the projected matter distribution. This result demonstrates the feasibility and robustness of this approach for future high-sensitivity CMB polarization experiments.