Research Project

MoHuCo: Modeling Humans in Context

Type

National Project

Start Date

01/09/2021

End Date

31/08/2024

Project Code

PID2020-120049RB-I00

Project illustration

Staff

Project Description

Project PID2020-120049RB-I00 funded by MCIN/ AEI /10.13039/501100011033

Recent advances in computer vision and deep learning have shown impressive results in modelling different aspects of humans. Given a single image or a video sequence, these models provide detailed reconstructions of the body shape and clothes, predict future movements and understand human behaviour, emotions and intentions. However, one essential factor that has been obviated so far, is the fact that most of these human characteristics are inherently driven by interactions with objects and/or other people in the environment. For instance, the body trajectory is highly constrained by the spatial distribution of the rest of objects in the environment; a particular facial expression (e.g. ‘fear’) may respond to a specific circumstance occurring in the surrounding (e.g. ‘danger’). Understanding these types of human-context connections would allow going beyond current state-of-the-art and perform robust human reasoning under complex situations such as partial observations (e.g. crowded scenes, heavy occlusions) or indirect observations (predicting human characteristics from contextual clues).

The goal of MoHuCo is therefore to develop novel computer vision tools to discover interrelations between person’s properties and the context. For this purpose, we will split the project in three main blocks:

1) Observing the human: consolidation and pushing to the limits the algorithms for 3D body/cloth reconstruction, motion prediction and behaviour analysis given direct observations of the person.

2) Observing the context: research on novel algorithms to extract heterogeneous information (both geometric and semantic) of the environment;

3) Build joint human-context models: bringing the representations of humans and environment into a single model, allowing to indirectly reason about the human from direct observations of the context.

Project Publications

Journal Publications

  • E. Corona, G. Alenyà, G. Pons-Moll and F. Moreno-Noguer. LayerNet: high-resolution semantic 3D reconstruction of clothed people. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2): 1257-1272, 2024.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Dhamanaskar, M. Dimiccoli, E. Corona, A. Pumarola and F. Moreno-Noguer. Enhancing egocentric 3D pose estimation with third person views . Pattern Recognition, 138(109358), 2023.

    Open/Close abstract Abstract Info Info pdf PDF

Conference Publications

  • A. Agudo. Detail-aware uncalibrated photometric stereo, 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, 2023, Rhodes Island, Greece, pp. 1-5.

    Open/Close abstract Abstract Info Info pdf PDF

  • D.F. Ordoñez, M. Martin, A. Agudo and F. Moreno-Noguer. On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis, 2023 Robotics: Science and Systems Conference, 2023, Daegu, Republic of Korea.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Urdapilleta and A. Agudo. Comparative study of feature localization methods for endoscopy image matching, 2023 IEEE International Conference on Image Processing Challenges and Workshops, 2023, Kuala Lumpur, Malaysia, pp. 3719-3723, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • R. Pérez, A. Espersen and A. Agudo. Robust wind turbine blade segmentation from RGB images in the wild, 2023 IEEE International Conference on Image Processing, 2023, Kuala Lumpur, Malaysia, pp. 1025-1029.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Pérez and A. Agudo. Sensor-agnostic multimodal fusion for multiple object tracking from camera, radar, lidar and V2X, 2023 FISITA 2023 World Congress, 2023, Barcelona, to appear.

    Open/Close abstract Abstract Info Info pdf PDF

  • M. Pérez and A. Agudo. Robust multimodal and multi-object tracking for autonomous driving applications, 2023 International Conference on Advanced Robotics, 2023, Abu Dhabi, UAE, pp. 100-106, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • G.D. Delmas, P. Weinzaepfel, F. Moreno-Noguer and G. Rogez. PoseFix: correcting 3D human poses with natural language, 2023 International Conference on Computer Vision, 2023, Paris, France, pp. 14972-14982.

    Open/Close abstract Abstract Info Info pdf PDF

  • D.F. Ordoñez, M. Martin, A. Agudo and F. Moreno-Noguer. Morphological symmetries in robot learning, 2023 RSS Workshop on Symmetries in Robot Learning, 2023, Daegu (South Korea), pp. 1-5.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Caselles, E. Ramon, J. Garcia, X. Giro-i-Nieto, F. Moreno-Noguer and G. Triginer. SIRA: Relightable Avatars from a Single Image, 2023 IEEE Winter Conference on Applications of Computer Vision, 2023, Waikoloa, Hawaii, pp. 775-784.

    Open/Close abstract Abstract Info Info pdf PDF

  • P. Estevez and A. Agudo. Uncalibrated, unified and unsupervised specular-aware photometric stereo, 2022 ICPR Workshop on Towards a Complete Analysis of People: From Face and Body to Clothes, 2022, Montreal (Canada), pp. 7-20.

    Open/Close abstract Abstract Info Info pdf PDF

  • G.D. Delmas, P. Weinzaepfel, T. Lucas, F. Moreno-Noguer and G. Rogez. PoseScript: 3D human poses from natural language, 17th European Conference on Computer Vision, 2022, Tel Aviv (Israel), in Computer Vision – ECCV 2022 , Vol 13666 of Lecture Notes in Computer Science, pp. 346-362, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Corona, G. Pons-Moll, G. Alenyà and F. Moreno-Noguer. Learned Vertex Descent: a new direction for 3D human model fitting, 17th European Conference on Computer Vision, 2022, Tel Aviv (Israel), in Computer Vision – ECCV 2022 , Vol 13666 of Lecture Notes in Computer Science, pp. 146--165, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Shen, A. Agudo, F. Moreno-Noguer and A. Ruiz. Conditional-Flow NeRF: Accurate 3D modelling with reliable uncertainty quantification, 17th European Conference on Computer Vision, 2022, Tel Aviv (Israel), in Computer Vision – ECCV 2022 , Vol 13666 of Lecture Notes in Computer Science, pp. 540-557, 2022.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Pérez and A. Agudo. Matching and recovering 3D people from multiple views, 2022 IEEE Winter Conference on Applications of Computer Vision, 2022, Waikoloa, Hawaii, USA, pp. 1184-1193, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Safari from visual signals: Recovering volumetric 3D shapes, 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, 2022, Singapore, pp. 2495-2499.

    Open/Close abstract Abstract Info Info pdf PDF

  • D.F. Ordoñez, A. Agudo, F. Moreno-Noguer and M. Martin. An adaptable approach to learn realistic legged locomotion without examples, 2022 IEEE International Conference on Robotics and Automation, 2022, Philadelphia, Pennsylvania, USA, pp. 4671-4678.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Agudo. Spline human motion recovery, 2022 IEEE International Conference on Image Processing, 2022, Bordeaux, France, pp. 4138-4142, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF

  • N. Ugrinovic, A. Pumarola, A. Sanfeliu and F. Moreno-Noguer. Single-view 3d body and cloth reconstruction under complex poses, 17th International Conference on Computer Vision Theory and Applications, 2022, Online.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Ruiz, A. Agudo and F. Moreno-Noguer. Generating attribution maps with disentangled masked backpropagation, 2021 International Conference on Computer Vision, 2021, Montreal, Canada, pp. 885-894.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Sanchez, A. Pumarola and F. Moreno-Noguer. PhysXNet: A customizable approach for learning cloth dynamics on dressed people, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 879-888.

    Open/Close abstract Abstract Info Info pdf PDF

  • E. Ramon, G. Triginer, J. Escur, A. Pumarola, J. García, X. Giro-i-Nieto and F. Moreno-Noguer. H3D-Net: Few-shot high-fidelity 3D head reconstruction, 2021 International Conference on Computer Vision, 2021, Montreal, Canada, pp. 5600-5609.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Hernandez Ruiz, A. Vilalta and F. Moreno-Noguer. Neural Cellular Automata manifold, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville, TN, USA (Virtual), pp. 10015-10023, Computer Vision Foundation.

    Open/Close abstract Abstract Info Info pdf PDF

  • N. Ugrinovic, A. Ruiz, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. Body size and depth disambiguation in multi-person reconstruction from single images, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 53-63.

    Open/Close abstract Abstract Info Info pdf PDF

  • J. Shen, A. Ruiz, A. Agudo and F. Moreno-Noguer. Stochastic Neural Radiance Fields: Quantifying uncertainty in implicit 3D representations, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 972-981.

    Open/Close abstract Abstract Info Info pdf PDF

  • A. Chatziagapi, S. Athar, F. Moreno-Noguer and D. Samaras. SIDER: Single-image neural optimization for facial geometric detail recovery, 2021 International Conference on 3D Vision, 2021, London, UK (Virtual), pp. 815-824.

    Open/Close abstract Abstract Info Info pdf PDF