We report the fast neutron flux and spectrum in the PE room at the China Jinping Underground Laboratory (CJPL) with a liquid scintillator detector doped with 0.5% gadolinium (Gd-Ls). The difference between nuclear recoil signals and electronic recoil signals is used to select neutron events in a liquid scintillator detector doped with gadolinium. The signature of gadolinium-captured thermal neutrons is used to time discrimination. The fast neutron flux range of 1~10 MeV measured in the PE room at the CJPL is 4.45×10-8 cm-2 s-1 based on 206 days of data. A similar selection method is used to analyze the degree of α contamination in the Gd-Ls. The α contamination of the natural radioactivity series including 238U, 235U, and 232Th are (0.0227±0.0001) s-1, (0.0329±0.0001) s-1, and (0.1471±0.0002) s-1, respectively. Compared with the fast neutron flux of other deep laboratories around the world, the fast neutron flux of CJPL is at an extremely low level, which is an ideal place for rare-events measurement experiments.