To address space complexity and time difference of human action, an adaptive recognition method of human skeleton action with spatial-temporal tensor fusion is proposed. Firstly, the spatial-temporal feature tensors of adjacent frames are established by making full use of the intra-frame spatial relationship and the inter-frame temporal relationship of the human action skeleton sequences. Secondly, the difference of spatial-temporal feature tensors of adjacent frames is calculated to achieve the spatial-temporal feature tensors of key adjacent frames and compose the behavior spatial-temporal feature tensors. Then, the spatial feature difference of the action spatial-temporal feature tensors and the multi-scale temporal convolution is used to construct the adaptive attention mechanism of the behavior spatio-temporal feature tensors to complete the fusion of action spatial-temporal features. Finally, a deep stochastic configuration network is used to recognize human action according to the spatial-temporal feature fusion tensor of action. The NTU RGB-D data set was used for experimental simulation, and the recognition accuracy reached 84. 57%. The corresponding system is designed for practical application verification. The results show that the proposed method is suitable for dealing with the space complexity and time difference of human action recognition.