Knee osteoarthritis (OA) is one of the main causes of activity limitation and physical disability in the elderly. Early diagnosis and intervention of knee osteoarthritis can help patients slow down the deterioration of OA. At present, the early diagnosis of knee osteoarthritis is detected by X-rays and scored according to the Kellgren-Lawrence (KL) grade. However, doctors′ scores are relatively subjective and vary from doctor to doctor. Grade classification of knee osteoarthritis is a matter of orderly classification. The ordinal penalty loss function assigns higher penalty weights to the classes that are further away from the ground truth, which is more suitable for knee osteoarthritis classification. In existing works, the penalty weights no longer change during training procedure, so the training model often fails to reach the expected results. In this paper, an adaptive ordinal penalty adjustment strategy is proposed to address the shortcomings of the ordinal penalty loss, in which the penalty weights are automatically tuned in reverse according to the confusion matrix obtained at each stage (epoch). Furthermore, the performance of the proposed method is validated on several classical CNN models such as ResNet, VGG, DenseNet and Inception by X-ray image data from Osteoarthritis Initiative (OAI). Experimental results show that the adaptive ordinal penalty adjustment strategy proposed in this paper can effectively improve the classification accuracy (AC) and mean absolute error (MAE) of the model when the initial weight score difference is small.