To improve the measurement accuracy of the spindle rotation error of precision machine tools, the multi-turn coincidence method is used to process the sensor data in the spindle rotation error measurement, which is based on the four-point matrix algorithm. The convergence speed of the traditional genetic algorithm is improved and the dependence of the optimization results on the initial value is reduced. The crossover and mutation probability factor formulas are updated, and the improved genetic algorithm is used to optimize the sensor installation angle and output weight coefficients. By using the improved genetic algorithm, the convergence rate is about 50% higher than the value of the traditional genetic algorithm. The spindle radial rotation error measurement and separation experiment are implemented by a multifunctional inclined rail CNC lathe. Compared with the calibration value, the deviation of the standard mandrel shape after separation is within 5% , and the error repeatability is less than 5% . Results show that the accuracy of the separation results is high. Thus, the correctness and feasibility of the proposed algorithm are verified.