An attitude estimation method in the presence of unknown correspondences is proposed by utilizing inertial aid tracking. Unlike methods that optimize prior pose selection using inertial information at the data processing level, this approach combines image and inertial information at the data acquisition level, which could enhance the accuracy and robustness of attitude estimation and feature point matching. The proposed method introduces a new hybrid tracking approach that combines feature point tracking and motion vector constraint tracking. And an initial matching weight matrix for two-dimensional and three-dimensional point sets is established. An optimization objective function is formulated based on this weight matrix, and an improved soft assignment algorithm is employed to iteratively optimize the estimation of attitude and the correspondences between the 2D/3D point sets. Algorithm performance is tested by using a 2D precision turntable with cooperative stereo targets. The results show that the success rate of this method is 99% when the yaw/pitch angle is ±50°, and the error of attitude estimation is less than 0.15°. The single measurement time is approximately 17 ms, and the method maintains a high success rate even in visual occlusion.