The detection of rail defect is of great significance for guaranteeing railway safety. In the background of studying the nondestructive testing and evaluation technology of rail, this paper comprehensively reviews the methods of rail defect detection adopted at home and abroad, including physical detection method and machine vision inspection method. The application of rail defect evaluation method by machine vision is presented and analyzed. Meanwhile, the nondestructive testing technology and evaluation technology for rail are compared. The differetiation of physical inspection by radiation inspection and the machine vision detection by image processing is discussed and summarized. The modern nondestructive testing, evaluation technology, and related technical problems involved in the development are analyzed and discussed. The future development of nondestructive testing and evaluation technology for rail defect is assumed.