For light-weight unmanned surface vessel(USV), the observability of course is low when sailing at low speed(7 knots). In terrible ocean environment and strong surge, It is difficult to meet the measuring accuracy requirement because the compass is not stable. In order to realize the smoothing filtering of course angle of the USV, the article estimate the process noise of course angle effectively according to the system control frequency and the boat body movement performance. We use an α-β filter to smoothing the course of USV based on the estimated process noise. In order to eliminating the boundary dis-continuous of course angle which will cause singular value when using α-β filter, we proposed a boundary filtering strategy to realize the full range smooth filtering for the course angle. Real-time data playback verified the effectiveness of the method on the navigation control for the USV.