To overcome the limitations of small fieldview and nonintelligent of traditional monitoring systems, this work combines panoramic imaging and computer vision technology to establish an automatic intrusion detection system in unmanned environment. The proposed system can achieve fast and accurate moving object detection and tracking in panoramic monitoring field. To solve the key challenge on effectively extracting moving objects in complex and dynamic background, this paper presents an adaptive background updating algorithm based on dynamic matching between feature blocks. On the basis of target detection using the fusion algorithm with frame differencing background subtraction, the torque information is utilized for tracking the target so as to avoid the lack of color and contour features under the panoramic view. The feature block si then extracted according to the outline and position of the target, in which the local matching of feature area is conducted between video sequences of each frame image and the initial background image. The color feature of the feature block is firstly analyzed, and a RGB color histogram is established based on interval statistics. Thus, the colour feature sequences are obtained.wWhether the region background updates is then determined by calculating the correlation between the two sequences to reducethe computation of a single pixel to be updated. Experiments show that the proposed algorithm is robust and feasible, and it can effectively improve the stability of the monitoring system.