Spatio-Temporal Attention-Based LSTM Networks for
3D Action Recognition and Detection

Human action analytics has attracted a lot of attention for decades in computer vision. It is important to extract discriminative spatio-temporal features to model the spatial and temporal evolutions of different actions. In this work, we propose a spatial and temporal attention model to explore the spatial and temporal discriminative features for human action recognition and detection from skeleton data. We build our networks based on Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. The learned model is capable of selectively focusing on discriminative joints of skeletons within each input frame and paying different levels of attention to the outputs of different frames. To ensure effective training of the network for action recognition, we propose a regularized cross-entropy loss to drive the learning process and develop a joint training strategy accordingly. \M{Moreover, based on temporal attention, we develop a method to generate action temporal proposals for action detection. We evaluate the proposed method on the SBU Kinect Interaction Dataset, the NTU RGB+D Dataset and the PKU-MMD Dataset, respectively. Experiment results demonstrate the effectiveness of our proposed model on both action recognition and action detection.