To address the problems of high-speed railway perimeter intrusion detection such as complex surroundings and a large number of small targets, an improved ByteTrack algorithm is proposed to realize the identification and tracking of perimeter intrusion. The model is improved by integrating YOLOv7-X and the data association method of BYTE. The convolution block attention module is introduced to improve the recognition effect of foreground targets in complex surroundings. The space-to-depth layer and the non-strided convolution layer are used to optimize the step convolution and pooling layers to improve the loss of fine-grained information caused by down-sampling in small target recognition. The railway perimeter intrusion dataset is established for experiments. The experimental results show that the AP of the improved module is 95. 6% , an increase of 9. 4% , and has improved the AP of target recognition for large, small, and medium-sized targets, especially for small targets, with a significant improvement of 22. 2% . The improved ByteTrack algorithm can realize the identification and tracking of intrusion behavior in the complex environment of high-speed railway perimeter, and provide technical support for high-speed railway perimeter protection.