The railway obstacles in front of the train have great threat to traffic safety. The existing railway object detection algorithms are difficult to balance the detection accuracy and speed, which are susceptible to complex environment and difficult to deploy in embedded equipment. To address these issues, the lightweight and adaptive multiscale convolutional neural network is proposed in this article. The model simplifies the computation of redundant feature maps in feature extraction process by means of feature map linear transformation, and the adaptive multi-scale feature fusion is used to optimize the ability and further improve the accuracy of foreign obstacles detection. In addition, combined with NVIDIA Jetson TX2, an autonomous intrusion detection system for railway traffic scenes is developed. Experimental results show that the proposed model performs a great compromise between detection speed and accuracy. The detection speed of LAM-NET on the NVIDIA GeForce GTX1080Ti is 297 FPS, and the detection accuracy is 92. 96% (7. 72% higher than that of YOLOv4-tiny), which can well realize the high precision, real-time and high robustness detection for railway obstacles.