Aiming at the enormous threat that railway intrusion obstacles pose to train operation safety, and the existing railway obstacle detection models have difficulty balancing detection accuracy and speed and poor multi-scale object detection robustness in complex railway environments, this article proposes an all-weather high-precision real-time multi-scale railway obstacle detection model. The model improves the feature extraction speed of the model by using dual-branch structure and linear operation. By modifying the Transformer structure, the lightweight model can model global contextual information. By designing high richness feature fusion structure and lightweight attention mechanism, the model′s multi-scale object detection ability is further improved. In addition, we embed the model and develop an intelligent detection system. The experimental results show that the proposed model has a detection accuracy and speed of 94. 93% and 132 fps in the dataset collected from actual railway scenes, respectively, which is 3. 09% higher than YOLOv5s. It can meet the application requirements of high-precision real-time detection of multi-scale obstacles in complex railway scenes.