It is important for the surface quality of hot rolled steel strips to make final product. Therefore, it is necessary to strictly control the defects on the surface of hot rolled steel strips. The current you only look once (YOLO) v4 algorithm has low detection accuracy and poor performance on small-scale information. To address these issues, an improved YOLOv4 automatic detection method is proposed. First, to improve detection speed, enhance detection target feature extraction and reduce gradient vanishing, the feature extraction network CSPDarknet53 in YOLOv4 is replaced with the lightweight deep neural network MobileNetv3. Secondly, to improve the learning efficiency and accelerate the convergence speed, the K-Means clustering is utilized to generate a prior box to suit for this experiment. Finally, the confidence loss is redefined and a loss function is proposed that can adapt to the multi-scale to solve the problem of poor detection effect due to the imbalance of positive and negative samples. Compared with the original YOLOv4 model for the surface defect detection of the hot rolled steel strip, experimental results show that the proposed method enhance the mean average precision and the speed about 7. 94% and 4. 52 f / s, respectively. The accuracy of this model is improved effectively while ensuring the detection speed.