The calibration of the tool-robot end coordinate system position relationship in the vision-guided robot grinding system is the key to determine the grinding accuracy. In order to solve the problem of poor calibration accuracy and low efficiency in the calibration of the robot tool coordinate system, this paper proposes a calibration scheme based on the grinding contour deviation correction of the test block. The proposed method begins with high-precision hand-eye calibration using a combined optimization algorithm. Next, a line-structured optical sensor scans the standard test block before and after grinding to establish the ideal and actual positional relationship between the tool and the robot base coordinate system. A deviation matrix is introduced to compensate for discrepancies in the tool-to-robot end coordinate system, effectively achieving tool coordinate system calibration. Simulation and experimental results show that the position deviation of the tool coordinate system is within 0.25 mm, and the attitude deviation of the three axes of the coordinate system is less than 0.01°. Furthermore, pipe seam grinding experiments reveal that the residual height of the weld seam after grinding is within 0.2 mm, meeting most industrial requirements.