Internal combustion engine fault diagnosis using vibration response signal meets the challenging of strong coupling and weak fault characteristics. A fault diagnosis method is proposed based on texture feature extraction of internal combustion engine vibration spectrum. In order to clearly characterize the nonstationary timevarying components in the timefrequency distribution of internal combustion engine vibration signal, the variational mode decomposition (VMD) is combined with the Rihaczek complex energy density distribution method. Thus, the vibration spectrum image can be obtained with good timefrequency clustering and no crossterm interference. Considering the parameter selection in the VMD decomposition process, Shannon entropy is introduced as the objective function and the successive grid search technique is employed to identify the optimal model parameters, to improve the adaptability of VMD decomposition. To realize the automatic recognition and fault diagnosis of the vibration spectrum of internal combustion engine, an improved local binary model (ILBP) is presented to analyze the texture information contained in the vibration spectrum. The lowdimensional feature parameters are extracted and the nearest neighbor classifier is adopted to identify the vibration spectrum under different working conditions. The proposed method is applied to the fault diagnosis of internal combustion engine. The results show that the method can effectively extract the weak fault characteristics of the vibration signal and realize the automatic diagnosis of internal combustion engine failure.