In order to improve the adaptive ability and segmentation accuracy of video object segmentation algorithm in various complex scenes, an object segmentation algorithm based on motion saliency map and optical flow vector analysis is proposed in this paper. Firstly, the proposed algorithm extracts the rough target region based on motion saliency map. Then, the motion boundaries of the motion object and background region are determined based on the optical flow vector between pairs of subsequent frames. The above information is combined to acquire the accurate pixels inside the moving objects with the pointinpolygon principle from the computational geometry. Finally, to refine the spatial accuracy of object segmentation in the previous step, per frame superpixels are acquired with oversegmenting method. And these superpixels are labeled as foreground or background based on confidence level and statistical model. The proposed algorithm was compared with typical algorithms in different scenes, and the results indicate that the proposed algorithm can effectively deal with the moving object segmentation on a variety of challenging scenes, and has higher segmentation accuracy than other existing algorithms.