In the application of object tacking,there are some typical problemsin complex scenario,such as light changing, shadow, occupancy and moving background. To overcome light changing and shadow, the algorithm of shadow tolerant local binary similarity pattern is proposed, which is based on the local binary pattern background model. Then, the distance discriminator which is calculated between the current target bounding box and the target bounding box in history is utilized to solve the problems of occupancy and moving background. The distance discriminator is also adopted to detect target fragments. Based on this detection result, the merging procedure can be achieved. After merging, the optical flow of each target bounding box is calculated.The differencesin a block and amongblocks are calculated. The bounding box segmentation procedure is processed based on the calculated similarity information. Finally, the structure support vector machineis used to design the target association procedure. Experimental results on multiple target tracking benchmark show that the proposed algorithm can achieve excellent tracking precision and robustness.