To deal with the serious performance degradation of target tracking algorithms in complex tracking scenes, such as background clutter, occlusion, thermal crossover, and target deformation, a real-time infrared single object tracking algorithm based on the adaptive label and sparse-learning correlation filter is proposed. First, sample labels are constructed based on the target response adaptively, and the discrimination ability of the correlation filter is enhanced by training with adaptive sample labels, which suppresses the pollution of the interference region to the tracking model. Secondly, the sparse learning strategy is introduced to suppress the multi-peak distribution of the response map in complex tracking scenes by its L1 norm, resulting in improving the robustness of the tracking algorithm. Compared with the baseline algorithm, the precision and AUC of the proposed algorithm are improved by 19. 3% and 39. 8% , respectively. Experimental results on datasets GTOT, RGBT234, and VOT-2016TIR show that the proposed algorithm has a strong ability to deal with the above complex tracking scenes. Its running speed is over 35 fps, and its comprehensive performance is better than the compared tracking algorithms.