The intrinsic magnetism calibration of the marine vector magnetometer is the core technology for magnetometer application system, especially when the magnetism of the carrier changes; the insitu calibration algorithm is the key to influence its application effectiveness. This paper analyzes the main magnetic interference sources of the underwater exploration system carrying three axis fluxgate magnetometer firstly, describes corresponding magnetic calibration model, and introduces the ellipsoid fitting algorithm, particle swarm optimization algorithm, genetic algorithm and particle swarm optimizationgenetic hybrid algorithm briefly. These algorithms are realized with MATLAB respectively. Finally, the field operation data of the underwater vehicle in the southwest Indian ocean were used to verify the functions of the above algorithms. From the results of the data fluctuation before and after calibration, rootmeansquare error and relative error, it can be seen that the hybrid algorithm can get better calibration effect and improve the measurement effectiveness of the magnetometer sensor in the insitu magnetic calibration.