To address the issue of low accuracy and poor robustness in single-feature sound source tracking under strong indoor reverberation and low signal-to-noise ratio (SNR), a robust tracking algorithm using multi-feature optimization mechanism is presented in this paper. This algorithm establishes a multi-feature optimization mechanism based on a time-delay estimation multi-hypothesis model, overcoming the poor localization performance of single-feature tracking in reverberant noise environments. Moreover, To enhance the robustness of the multi-feature optimization mechanism against random movements of the speaker, we introduce an improved Interacting Multiple Model (IMM) particle filter algorithm. By real-time adjustment of model noise variance and model probability, the robustness of the multi-feature optimization mechanism is improved. Simulation analysis and actual test results indicate that the average root mean square error (RMSE) of the position is reduced by approximately 12% using the proposed algorithm, compared with the existing literature, under the multi-feature optimization mechanism. Based on the improved IMM algorithm, the average RMSE of the position is reduced by nearly 89.6% through the proposed algorithm, compared with the other algorithms. The proposed algorithm significantly eliminates the adverse effects of reverberation and noise, and improves the accuracy and robustness of sound source localization and tracking.