The linear structure light sensor is used to measure the three-dimensional error of the gear, which has advantages of fast, full information, and high repeatability. However, the position and posture relationship between the sensor and the gear under test is the key issue that has influence on the accuracy of gear measurement. To optimize the position and posture relationship between the probe and the gear, the quantitative evaluation index of the position and posture of the sensor is proposed. To evaluate the effectiveness and correctness of the proposed method, measurement experiments of the same gear product of different posture parameters are carried out, and compared and analyzed with the Klingberg P26 gear measurement center. Compared with the conventional measurement method, results show that the point cloud density ratio Λp of the optimization method is 33. 6% higher than the index, and the inclination ratio υp is 30. 2% higher than the index. The measured tooth profile unit is more complete and more in line with the data requirements of the international standard ISO1328. The individual gear error evaluation results are also more accurate. By using this method, parameters of the position and posture of the linear structured light sensor are estimated, which are used for gear measurement. Thereby, the accuracy of gear measurement is improved. It provides a way to solve the problem of non-contact measurement of steep tooth surfaces.