In complex environment, there are many kinds of defects on the surface of metal. Existing metal surface defect segmentation algorithm has the deficiencies of low efficiency and narrow application scope, to solve these problems, an adaptive segmentation algorithm for metal surface defects is proposed in this paper. In this algorithm, firstly, the graylevel images of the metal surfaces are transformed from eight directions. And then, according to the graylevel fluctuation of the multiple images, the threshold and step length in neighborhood gray level difference segmentation algorithm are adaptively changed, and corresponding image in each direction is segmented. Lastly, all the processed images are compressed to a single image with PCA algorithm. Experiment results indicate that compared with existing segmentation algorithms, the proposed algorithm not only can be applied to segment various kinds of metal surface defects, but also has high segmentation accuracy.