Structure from motion (SFM) refers to a process in which the 3D structure is created by analyzing 2D image sequences, which is very important in many applications of computer vision. Feature tracking is one of the core components of largescale SFM. However, the robustness and time efficiency of the existing algorithms are needed to be improved. To address these issues, a fast and robust feature tracking (FRFT) algorithm is presented. Firstly, images moments are used to define a main direction for AGAST feature point, which can help to construct a rotation invariance descriptor. Secondly, in the space of the difference of Gaussian, the difference between the center point and its neighbor points is used to construct a descriptor for the OAGAST keypoint, which can avoid the influence of illumination and scale change on the feature matching. Thirdly, to improve the time efficiency of feature matching, the large feature set is clustered to some small ones, and KDTree is used to accelerate feature matching for improving the time efficiency of FRFT. Finally, the proposed method is evaluated with four ways, and compared with the stateoftheart methods. Experimental results show that the proposed FRFT method outperforms the stateofthe art ones on robustness and time efficiency.