To meet the test requirements of large out-of-band suppression filter transmission and reflection characteristics, this article proposes an in-suit enhancement method for the dynamic range of a VNA. The method has add a low-noise amplifier to the test chain without changing the hardware architecture of the instrument, extend dynamic range by amplifying tiny signals, and the cross-correlation signal processing algorithm is adopted. The noise floor of the instrument is further reduced by performing multiple operations on the twochannel test data. To improve the test accuracy and solve the problem that the error term of the receiving port cannot be directly obtained due to the introduction of the low noise amplifier, this article designs a two-step calibration method based on a 12-term error model. The extraction and de-embedding of the connection adapter parameters is finished. Finally, the characteristics of the large out-of-band rejection filters in two different frequency bands are measured by using the proposed method. Experimental results show that the dynamic range of the IF bandwidth of 200 Hz can reach 130 dB, which can be improved by more than 20 dB compared to the dynamic range of conventional instruments. The effectiveness of the effectiveness of the proposed method is verified.