Aiming at the low accuracy of common target recognition and tracking algorithms caused by camera high frequency sloshing under dynamic view, this paper proposes an adaptive window target tracking algorithm based on Canny and GrabCut. Firstly, the speeded up robust features (SURF) algorithm is used to learn the picture library and remember the picture features. The memory target recognition algorithm based on SURF algorithm is designed. Then, GrabCut adaptive optimization algorithm is used to segment the region of interest to achieve rough tracking of the target. Finally, a windowed algorithm based on Canny is achievedtoaccurately track the target. The experimental results show that the algorithm designed in this paper can quickly identify the target and accurately outline its contour. Moreover, the target can be stably tracked. Compared with other algorithms, the algorithm has obvious improvement in computation efficiency and accuracy.