Deep multi-model fusion for human activity recognition using evolutionary algorithms

Kamal Kant Verma, Brij Mohan Singh

Research output: Contribution to journalArticlepeer-review

19 Citations (Scopus)


Machine recognition of the human activities is an active research area in computer vision. In previous study, either one or two types of modalities have been used to handle this task. However, the grouping of maximum information improves the recognition accuracy of human activities. Therefore, this paper proposes an automatic human activity recognition system through deep fusion of multi-streams along with decision-level score optimization using evolutionary algorithms on RGB, depth maps and 3d skeleton joint information. Our proposed approach works in three phases, 1) space-time activity learning using two 3D Convolutional Neural Network (3DCNN) and a Long Sort Term Memory (LSTM) network from RGB, Depth and skeleton joint positions 2) Training of SVM using the activities learned from previous phase for each model and score generation using trained SVM 3) Score fusion and optimization using two Evolutionary algorithm such as Genetic algorithm (GA) and Particle Swarm Optimization (PSO) algorithm. The proposed approach is validated on two 3D challenging datasets, MSRDailyActivity3D and UTKinectAction3D. Experiments on these two datasets achieved 85.94% and 96.5% accuracies, respectively. The experimental results show the usefulness of the proposed representation. Furthermore, the fusion of different modalities improves recognition accuracies rather than using one or two types of information and obtains the state-of-art results.

Original languageEnglish
Pages (from-to)44-58
Number of pages15
JournalInternational Journal of Interactive Multimedia and Artificial Intelligence
Issue number2
Publication statusPublished - 2021

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Statistics and Probability
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence


Dive into the research topics of 'Deep multi-model fusion for human activity recognition using evolutionary algorithms'. Together they form a unique fingerprint.

Cite this