By establishing the error model of heading angle for triaxial magnetometer that widely used in magnetoinertial navigation system of unmanned aerial vehicle (UAV), the heading misalignment error that consists of constant error and half circle compass error is concluded. Therefore, a novel calibration algorithm is proposed, which is based on dual inner products. The principle of proposed method makes use of the invariance of geomagnetic vector and itself inner product, as well as the constant inner product of geomagnetic vector and gravity vector. This method can eliminate the mutual misalignment of triaxial magnetometer that cannot be identified using the invariance of vector norm only, and thus it can achieve complete calibration of triaxial magnetometer. Numerical simulations and experiment results show that the proposed method has better performance compared with scalar checking, dot product invariance and twostep methods. It can effectively reduce the norm error of geomagnetic vector and heading angle error of UAV, and it has preferable robustness against sensor noise.