Deep Learning for Human Activity Recognition on 3D Human Skeleton: Survey and Comparative Study

Overview

Journal Sensors (Basel)

Publisher MDPI

Specialty Biotechnology

Date 2023 Jun 10

PMID 37299848

Authors

Hung-Cuong Nguyen

Thi-Hao Nguyen

Rafal Scherer

Van-Hung Le

Affiliations

Soon will be listed here.

Abstract

Human activity recognition (HAR) is an important research problem in computer vision. This problem is widely applied to building applications in human-machine interactions, monitoring, etc. Especially, HAR based on the human skeleton creates intuitive applications. Therefore, determining the current results of these studies is very important in selecting solutions and developing commercial products. In this paper, we perform a full survey on using deep learning to recognize human activity based on three-dimensional (3D) human skeleton data as input. Our research is based on four types of deep learning networks for activity recognition based on extracted feature vectors: Recurrent Neural Network (RNN) using extracted activity sequence features; Convolutional Neural Network (CNN) uses feature vectors extracted based on the projection of the skeleton into the image space; Graph Convolution Network (GCN) uses features extracted from the skeleton graph and the temporal-spatial function of the skeleton; Hybrid Deep Neural Network (Hybrid-DNN) uses many other types of features in combination. Our survey research is fully implemented from models, databases, metrics, and results from 2019 to March 2023, and they are presented in ascending order of time. In particular, we also carried out a comparative study on HAR based on a 3D human skeleton on the KLHA3D 102 and KLYOGA3D datasets. At the same time, we performed analysis and discussed the obtained results when applying CNN-based, GCN-based, and Hybrid-DNN-based deep learning networks.

Citing Articles

The Millennia-Long Development of Drugs Associated with the 80-Year-Old Artificial Intelligence Story: The Therapeutic Big Bang?.

Crouzet A, Lopez N, Riss Yaw B, Lepelletier Y, Demange L Molecules. 2024; 29(12).

PMID: 38930784 PMC: 11206022. DOI: 10.3390/molecules29122716.

Biosensor-Based Multimodal Deep Human Locomotion Decoding via Internet of Healthcare Things.

Javeed M, Abdelhaq M, Algarni A, Jalal A Micromachines (Basel). 2023; 14(12).

PMID: 38138373 PMC: 10745656. DOI: 10.3390/mi14122204.

DSA-Net: Infrared and Visible Image Fusion via Dual-Stream Asymmetric Network.

Yin R, Yang B, Huang Z, Zhang X Sensors (Basel). 2023; 23(16).

PMID: 37631634 PMC: 10459630. DOI: 10.3390/s23167097.

Multi-Camera-Based Human Activity Recognition for Human-Robot Collaboration in Construction.

Jang Y, Jeong I, Younesi Heravi M, Sarkar S, Shin H, Ahn Y Sensors (Basel). 2023; 23(15).

PMID: 37571779 PMC: 10422633. DOI: 10.3390/s23156997.

References

Guan M, Cho S, Petro R, Zhang W, Pasche B, Topaloglu U . Natural language processing and recurrent network models for identifying genomic mutation-associated cancer treatment change from patient progress notes. JAMIA Open. 2019; 2(1):139-149. PMC: 6435007. DOI: 10.1093/jamiaopen/ooy061. View

Ren B, Liu M, Ding R, Liu H . A Survey on 3D Skeleton-Based Action Recognition Using Learning Method. Cyborg Bionic Syst. 2024; 5:0100. PMC: 11096730. DOI: 10.34133/cbsystems.0100. View

Koniusz P, Wang L, Cherian A . Tensor Representations for Action Recognition. IEEE Trans Pattern Anal Mach Intell. 2021; 44(2):648-665. DOI: 10.1109/TPAMI.2021.3107160. View

Morshed M, Sultana T, Alam A, Lee Y . Human Action Recognition: A Taxonomy-Based Survey, Updates, and Opportunities. Sensors (Basel). 2023; 23(4). PMC: 9963970. DOI: 10.3390/s23042182. View

Wang M, Ni B, Yang X . Learning Multi-View Interactional Skeleton Graph for Action Recognition. IEEE Trans Pattern Anal Mach Intell. 2020; 45(6):6940-6954. DOI: 10.1109/TPAMI.2020.3032738. View

Wang L, Huynh D, Koniusz P . A Comparative Review of Recent Kinect-Based Action Recognition Algorithms. IEEE Trans Image Process. 2019; 29:15-28. DOI: 10.1109/TIP.2019.2925285. View

Gupta N, Gupta S, Pathak R, Jain V, Rashidi P, Suri J . Human activity recognition in artificial intelligence framework: a narrative review. Artif Intell Rev. 2022; 55(6):4755-4808. PMC: 8763438. DOI: 10.1007/s10462-021-10116-x. View

Liu J, Shahroudy A, Perez M, Wang G, Duan L, Kot A . NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding. IEEE Trans Pattern Anal Mach Intell. 2019; 42(10):2684-2701. DOI: 10.1109/TPAMI.2019.2916873. View

Liu J, Shahroudy A, Wang G, Duan L, Kot A . Skeleton-Based Online Action Prediction Using Scale Selection Network. IEEE Trans Pattern Anal Mach Intell. 2019; 42(6):1453-1467. DOI: 10.1109/TPAMI.2019.2898954. View

10.

Qin Z, Liu Y, Ji P, Kim D, Wang L, McKay R . Fusing Higher-Order Features in Graph Neural Networks for Skeleton-Based Action Recognition. IEEE Trans Neural Netw Learn Syst. 2022; 35(4):4783-4797. DOI: 10.1109/TNNLS.2022.3201518. View

11.

Wen J, Guillen L, Abe T, Suganuma T . A Hierarchy-Based System for Recognizing Customer Activity in Retail Environments. Sensors (Basel). 2021; 21(14). PMC: 8309534. DOI: 10.3390/s21144712. View

12.

Ke Q, Bennamoun M, Rahmani H, An S, Sohel F, Boussaid F . Learning Latent Global Network for Skeleton-based Action Prediction. IEEE Trans Image Process. 2019; . DOI: 10.1109/TIP.2019.2937757. View

13.

Song Y, Zhang Z, Shan C, Wang L . Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition. IEEE Trans Pattern Anal Mach Intell. 2022; 45(2):1474-1488. DOI: 10.1109/TPAMI.2022.3157033. View

14.

Hochreiter S, Schmidhuber J . Long short-term memory. Neural Comput. 1997; 9(8):1735-80. DOI: 10.1162/neco.1997.9.8.1735. View

15.

Arshad M, Bilal M, Gani A . Human Activity Recognition: Review, Taxonomy and Open Challenges. Sensors (Basel). 2022; 22(17). PMC: 9460866. DOI: 10.3390/s22176463. View

16.

Yang H, Yan D, Zhang L, Sun Y, Li D, Maybank S . Feedback Graph Convolutional Network for Skeleton-Based Action Recognition. IEEE Trans Image Process. 2021; 31:164-175. DOI: 10.1109/TIP.2021.3129117. View

17.

Bianchi F, Grattarola D, Livi L, Alippi C . Graph Neural Networks With Convolutional ARMA Filters. IEEE Trans Pattern Anal Mach Intell. 2021; 44(7):3496-3507. DOI: 10.1109/TPAMI.2021.3054830. View

18.

Islam M, Nooruddin S, Karray F, Muhammad G . Human activity recognition using tools of convolutional neural networks: A state of the art review, data sets, challenges, and future prospects. Comput Biol Med. 2022; 149:106060. DOI: 10.1016/j.compbiomed.2022.106060. View

19.

Zhang P, Xue J, Lan C, Zeng W, Gao Z, Zheng N . EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks. IEEE Trans Image Process. 2019; . DOI: 10.1109/TIP.2019.2937724. View

20.

Li M, Chen S, Chen X, Zhang Y, Wang Y, Tian Q . Symbiotic Graph Neural Networks for 3D Skeleton-Based Human Action Recognition and Motion Prediction. IEEE Trans Pattern Anal Mach Intell. 2021; 44(6):3316-3333. DOI: 10.1109/TPAMI.2021.3053765. View