The highspeed railway clearance intrusion detection system is used to detect whether there is object intruding the safety clearance of the highspeed railway. To enhance the reliability of the system, a new CNN based fast feature extraction algorithm is proposed. Aiming at the problem of slow feature calculation speed, a simplified full connected network structure is proposed, and the structure of the neural network is simplified to two full connected convolutional layers. To avoid the accuracy decreasing caused by simplifying network structure, the convolutional kernels of the convolutional layers are pretrained. Finally, in order to prevent the symmetric feature extraction caused by full connection, fast feature extraction algorithm with sparse parameters added is proposed, and the network is trained with sparse coding algorithm. The improved CNN accelerates the calculation speed while ensures the accuracy. At the same time, the new algorithm satisfies the requirements of real time capability and high accuracy. Experiment result shows that the speed of processing single image is 0.15 s and the accuracy is 99.5%.