Description of the project

Human motion analysis is crucial for studying people and understanding how they behave, communicate and interact with real world environments. Due to the complex nature of body movements as well as the high cost of motion capture systems, acquisition of human motion is not straightforward and thus constraints data production. Hopefully, recent approaches estimating human poses from videos offer new opportunities to analyze skeleton-based human motion. While skeleton-based human motion analysis has been extensively studied for behavior understanding like action recognition, some efforts are yet to be done for the task of human motion generation. Particularly, the automatic generation of motion sequences is beneficial for rapidly increasing the amount of data and improving Deep Learning-based analysis algorithms. Besides, generative models facilitate the understanding of data intrinsic features and thus allow the generation of new variable contents. Particularly, these considerations can be very beneficial and promising in the medical domain and especially in physical rehabilitation.

The DELEGATION project aims to propose a Deep Learning-based framework for generating various expressive skeleton-based human motion sequences. At the heart of this framework, continuous latent sub-spaces will allow controlling the generation process through meaningful input parameters such as action/activity type, emotion, motion style, target human morphology, etc. By selecting some of these parameters, a user can then generate new motion sequences corresponding to each of the criteria. The idea of DELEGATION is depicted in Figure 1 where the trained latent action and style sub-spaces allow the generation of a new action (e.g. walk) with a style (e.g. proudly). Moreover, such a framework will be extended to consider complex motion sequences combining shorter motion primitives in order to generate realistic movements like activities of daily living (ADL) (e.g. prepare a meal).

Thus, we propose a Deep Learning-based framework to disentangle aforementioned features from human motion sequences and approximate corresponding latent sub-spaces. The following challenges will be addressed during the DELEGATION project:

• Deep generative model for noisy skeleton data: Skeleton data that we are considering in the project are estimated from videos using recent Deep Learning-based approaches like OpenPose and BlazePose. We will then investigate which deep architectures should be employed to better handle such noisy and incomplete data to characterize spatio-temporal features within our generative model.
• Extracting dynamics features: Expressiveness in motions is characterized by very fine variations of the skeleton data. Hence differentiating these fine variations is not straightforward and a deep analysis is required. We will investigate how deep generative models can disentangle human motion sequences and learn different interpretable sub-spaces.
• Generating complex motion sequences: Human motion is by nature very complex and can simulate different levels of complexity of movements characterized by varying duration and dynamics and involving specific parts of the body. In particular, activities of daily living (ADL) are characterized by very long sequences and a various combination of short motion primitives. We will then study if the proposed architecture can be extended for combining short motion patterns and generating complex activities.
• Considering rehabilitation exercises: When it comes to applying theoretical to real-world problems, the adaptation is not effortless, particularly in the context of physical rehabilitation. We aim to adapte theoretical findings to rehabilitation motion assessment. First, we will leverage our generative model to overcome the time consuming and restrictive acquisition process in order to produce new rehabilitation sequences. Second, disentangling physical limitations (due to injuries, pain or neurodegenerative disease) from the motion is crucial for a robust assessment. We will investigate how the approach is able to accomplish this.