The mismatch of image features affects the basic matrix calculation and leads to poor estimation accuracy of SLAM visual odometry. To address this issue, an optimization method of visual odometry based on RANSAC is proposed. First, the initial matching is roughly filtered by the minimum distance threshold method with an appropriate threshold, and the relative transformation relationship between images is then calculated by RANSAC. The result that conforms to the transformation relationship is considered an interior point. The iteration result with most interior points is the correct matching result. Then, the homographic transformation between images is calculated, and the basic matrix is derived from the calculated results. The interior points are determined by epipolar geometric constraints and the fundamental matrix with most interior points is obtained. Finally, the TUM data set is used to validate the performance of the Visual Odometry optimization algorithm from characteristic matching and basic matrix calculation. The experiment results show that the optimized RANSAC algorithm not only effectively improves operation efficiency and removes the mismatched feature points, but also improves the accuracy of image feature point matching by 7. 7% . Meanwhile, the interior-point rate of the basic matrix estimation algorithm in this paper is increased by 3% while improving the basic matrix calculation accuracy. This algorithm provides the theoretical basis for improving the accuracy of visual odometer estimation.