Multi-Modality Multi-Task Recurrent Neural Network
for Online Action Detection

Online action detection is a brand new challenge and plays a critical role in the visual surveillance analytics. It goes one step further over conventional action recognition task, which recognizes human actions from well-segmented clips. Online action detection is desired to identify the action type and localize action positions on the fly from the untrimmed stream data. In this paper, we propose a Multi-Modality Multi-Task Recurrent Neural Network (MM-MT RNN), which incorporates both RGB and Skeleton networks. We design different temporal modeling networks to capture specific characteristics from vari- ous modalities. Then, a deep Long Short-Term Memory (LSTM) subnetwork is utilized effectively to capture the complex long- range temporal dynamics, naturally avoiding the conventional sliding window design and thus ensuring high computational efficiency. Constrained by a multi-task objective function in the training phase, this network achieves superior detection perfor- mance and is capable of automatically localizing the start and end points of actions more accurately. Furthermore, embedding subtask of regression provides the ability to forecast the action prior to its occurrence. We evaluate the proposed method and several other methods in action detection and forecasting on the Online Action Detection Dataset (OAD) and Gaming Action Dataset (G3D) datasets. Experimental results demonstrate that our model achieves the state-of-the-art performance on both two tasks.