The precise feedback of the end position of the micro-manipulation actuator is of great significance in the micro-automation operation, and the existing research cannot overcome the problem of accurate tracking of the end of the actuator in a complex interference environment. Aiming at the above mentioned problem, a method for detecting and tracking the end position of the actuator based on the semantic segmentation model is proposed. Firstly, an end-to-end semantic segmentation model of the actuator image is built. Secondly, the contour inflection point detection algorithm is used to track the end position of the actuator in the segmented mask image. In order to further improve the tracking accuracy and robustness of the algorithm in a complex environment, 2D Kalman filter algorithm is used to process the occlusion situation, and the position tracking is realized when the end of the actuator is occluded. The experiment results show that the semantic segmentation model can achieve the segmentation accuracy of 62. 4% for the actuator, and the maximum average error of tracking the end position of the actuator in a complex environment is 1. 51 pixels, which provides a basis for improving the manipulation accuracy of the micro end effector.