In order to automatically classify weld surface defect in header pipe joint, Computer Vision based defect classification is studied. The texture features of different weld defects are analyzed, Grey level cooccurrence matrix (GLCM) is applied to extract features from digital images, and 15 types of statistical indexes are obtained to characterize the weld surface defects. Backpropagation artificial neural network method is used for defect classification. The influence of GLCM parameters, the neural network structure and the number and variety of input parameters on the defect classification performance is analyzed, and optimal neural network structure and input parameters are selected. In further, the optimized network is utilized for training and classifying the images of different weld defects acquired by industrial endoscope. The results show that weld defects detection rate of overall classification can be up to 91%. The proposed method can be used for automatic classification of weld surface defect in header pipe joint.