Research Project

IntellAct: Intelligent observation and execution of Actions and manipulations

Type

European Project

Start Date

01/03/2011

End Date

30/05/2015

Project Code

FP7-ICT-2009-6-269959

Project illustration

Staff

Project Description

IntellAct addresses the problem of understanding and exploiting the meaning (semantics) of manipulations in terms of objects, actions and their consequences for reproducing human actions with machines. It comprises three building blocks: (1) Learning: Abstract, semantic descriptions of manipulations are extracted from video sequences showing a human demonstrating the manipulations; (2) Monitoring: In the second step, observed manipulations are evaluated against the learned, semantic models; (3) Execution: Based on learned, semantic models, equivalent manipulations are executed by a robot. The analysis of low-level observation data for semantic content (Learning) and the synthesis of concrete behavior (Execution) constitute the major scientific challenge of IntellAct. Based on the semantic interpretation and description and enhanced with low-level trajectory data for grounding, two major application areas are addressed by IntellAct: First, the monitoring of human manipulations for correctness (e.g., for training or in high-risk scenarios) and second, the efficient teaching of cognitive robots for manipulation (e.g. in the context of assembly processes in industrial settings and on a space station). To achieve these goals, IntellAct brings together recent methods for (1) parsing scenes into spatio-temporal graphs and so called ‘semantic event chains’, (2) probabilistic models of objects and their manipulation, (3) probabilistic rule learning, and (4) dynamic motion primitives for trainable and flexible descriptions of robotic motor behavior. Its implementation employs a concurrent-engineering approach that includes virtual-reality-enhanced simulation as well as physical robots. Its goal culminates in the demonstration of a robot understanding, monitoring and reproducing human action.

Project Publications

Journal Publications

  • S. Thiusius, B. Anders, S. Christian, N. Wantia, J. Rossmann, D. Martínez, G. Alenyà, C. Torras, U. Ales, N. Bojan, K. Aljaž, F. Wörgötter, E.E. Aksoy, J. Papon, S. Haller, J. Piater and N. Krüger. Teaching a robot the semantics of assembly tasks. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 48(5): 670-692, 2018.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà and C. Torras. Relational reinforcement learning with guided demonstrations. Artificial Intelligence, 247: 295–312, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • C. Torras. Service robots for citizens of the future. European Review, 24(1): 17-30, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Husain, B. Dellen and C. Torras. Consistent depth video segmentation using adaptive surface models. IEEE Transactions on Cybernetics, 45(2): 266-278, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà and C. Torras. Planning robot manipulation to clean planar surfaces. Engineering Applications of Artificial Intelligence, 39: 23-32, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, C. Torras and F. Moreno-Noguer. DaLI: Deformation and light invariant descriptor. International Journal of Computer Vision, 115(2): 136–154, 2015.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa, G. Alenyà, F. Moreno-Noguer and C. Torras. Learning RGB-D descriptors of garment parts for informed robot grasping. Engineering Applications of Artificial Intelligence, 35: 246-258, 2014.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Husain, B. Dellen and C. Torras. Robust surface tracking in range image sequences. Digital Signal Processing, 35: 37-44, 2014.

    Open/Close abstract Abstract Info Info pdf PDF
  • L. Rozo, P. Jiménez and C. Torras. A robot learning from demonstration framework to perform force-based manipulation tasks. Intelligent Service Robotics, 6(1): 33-51, 2013.

    Open/Close abstract Abstract Info Info pdf PDF
  • P. Jiménez. Survey on assembly sequencing: A combinatorial and geometrical perspective. Journal of Intelligent Manufacturing, 24(2): 235-250, 2013.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Penate-Sanchez, J. Andrade-Cetto and F. Moreno-Noguer. Exhaustive linearization for robust camera pose and focal length estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(10): 2387-2400, 2013.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Abramov, K. Pauwels, J. Papon, F. Wörgötter and B. Dellen. Real-time segmentation of stereo videos on a portable system with a mobile GPU. IEEE Transactions on Circuits and Systems for Video Technology, 22(9): 1292-1305, 2012.

    Open/Close abstract Abstract Info Info pdf PDF
  • S. Ulbrich, V. Ruiz de Angulo, C. Torras, T. Asfour and R. Dillman. Kinematic Bézier maps. IEEE Transactions on Systems, Man and Cybernetics: Part B, 42(4): 1215-1230, 2012.

    Open/Close abstract Abstract Info Info pdf PDF

Conference Publications

  • F. Husain, A. Colomé, B. Dellen, G. Alenyà and C. Torras. Realtime tracking and grasping of a moving object from range video, 2014 IEEE International Conference on Robotics and Automation, 2014, Hong Kong, China, pp. 2617-2622.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà, P. Jiménez, C. Torras, J. Rossmann, N. Wantia, A. Eren Erdal, S. Haller and J. Piater. Active learning of manipulation sequences, 2014 IEEE International Conference on Robotics and Automation, 2014, Hong Kong, China, pp. 5671-5678.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé and C. Torras. Dimensionality reduction and motion coordination in learning trajectories with dynamic movement primitives, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014, Chicago, IL, USA, pp. 1414-1420.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà and C. Torras. Finding safe policies in model-based active learning, 2014 IROS Machine Learning in Planning and Control of Robot Motion Workshop, 2014, Chicago, pp. 1-6.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Husain, B. Dellen and C. Torras. Recognizing point clouds using conditional random fields, 22nd International Conference on Pattern Recognition, 2014, Stockhom, Sweden, pp. 4257-4262, IEEE Computer Society.

    Open/Close abstract Abstract Info Info pdf PDF
  • T.R. Savarimuthu, A.G. Buch, Y. Yang, W. Mustafar, S. Haller, J. Papon, D. Martínez and E.E. Aksoy. Manipulation monitoring and robot intervention in complex manipulation sequences, 2014 RSS Workshop on Robotic Monitoring, 2014, Berkeley, CA, USA.

    Open/Close abstract Abstract Info Info pdf PDF
  • B. Dellen, F. Husain and C. Torras. Joint segmentation and tracking of object surfaces in depth movies along human/robot manipulations, 8th International Conference on Computer Vision Theory and Applications, 2013, Barcelona, pp. 244-251.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, A. Quattoni, C. Torras and F. Moreno-Noguer. A joint model for 2D and 3D pose estimation from a single image, 2013 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2013, Portland, OR, USA, pp. 3634--3641.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé, D. Pardo, G. Alenyà and C. Torras. External force estimation during compliant robot manipulation, 2013 IEEE International Conference on Robotics and Automation, 2013, Karlsruhe, Germany, pp. 3535-3540.

    Open/Close abstract Abstract Info Info pdf PDF
  • S. Reich, A. Abramov, J. Papon, F. Wörgötter and B. Dellen. A novel real-time edge-preserving smoothing filter, 8th International Conference on Computer Vision Theory and Applications, 2013, Barcelona, pp. 1-11.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé, G. Alenyà and C. Torras. Handling high parameter dimensionality in reinforcement learning with dynamic motor primitives, 2013 ICRA Workshop on Novel Methods for Learning and Optimization of Control Policies and Trajectories for Robotics, 2013, Karlsruhe, Germany.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé and C. Torras. Positioning two redundant arms for cooperative manipulation of objects, 6th International Workshop on Computational Kinematics, 2013, Barcelona, in Computational Kinematics, Vol 15 of Mechanisms and Machine Science, pp. 121-129, 2014, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà and C. Torras. Planning surface cleaning tasks by learning uncertain drag actions outcomes, 2013 ICAPS Workshop on Planning and Robotics, 2013, Rome, pp. 106-111.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa and C. Torras. Large-scale image classification using ensembles of nested dichotomies, 16th Catalan Conference on Artificial Intelligence, 2013, Vic, Spain, in Artificial Intelligence Research and Development, Vol 256 of Frontiers in Artificial Intelligence and Applications, pp. 87-90, 2013, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa, G. Alenyà, F. Moreno-Noguer and C. Torras. FINDDD: A fast 3D descriptor to characterize textiles for robot manipulation, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013, Tokyo, Japan, pp. 824-830.

    Open/Close abstract Abstract Info Info pdf PDF
  • L. Rozo, S. Calinon, D. Caldwell, P. Jiménez and C. Torras. Learning collaborative impedance-based robot behaviors, 27th AAAI Conference on Artificial Intelligence, 2013, Bellevue, pp. 1422-1428.

    Open/Close abstract Abstract Info Info pdf PDF
  • L. Rozo, P. Jiménez and C. Torras. Force-based robot learning of pouring skills using parametric hidden Markov models, 9th International Workshop on Robot Motion and Control, 2013, Wasowo, Poland, pp. 227-232.

    Open/Close abstract Abstract Info Info pdf PDF
  • C. Torras. From the Turing test to science fiction: The challenges of social robotics, 16th Catalan Conference on Artificial Intelligence, 2013, Vic, Spain, in Artificial Intelligence Research and Development, Vol 256 of Frontiers in Artificial Intelligence and Applications, pp. 5-7, 2013, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Alenyà, A. Ramisa, F. Moreno-Noguer and C. Torras. Characterization of textile grasping experiments, 2012 ICRA Workshop on Conditions for Replicable Experiments and Performance Comparison in Robotics Research, 2012, St Paul, Minnesota, USA, pp. 1-6.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé, D. Pardo, G. Alenyà and C. Torras. External force estimation for textile grasp detection, 2012 IROS Workshop Beyond Robot Grasping: Modern Approaches for Learning Dynamic Manipulation, 2012, Vilamoura, Portugal, pp. 1-1.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Pardo, L. Rozo, G. Alenyà and C. Torras. Dynamically consistent probabilistic model for robot motion learning, 2012 IROS Workshop on Learning and Interaction in Haptic Robots, 2012, Algarve, Portugal, pp. 1-2.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Alexey, K. Pauwels, J. Papon, F. Wörgötter and B. Dellen. Depth-supported real-time video segmentation with the Kinect, 2012 IEEE Workshop on Applications of Computer Vision, 2012, Breckenridge, CO, USA, pp. 457-464.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa, G. Alenyà, F. Moreno-Noguer and C. Torras. Using depth and appearance features for informed robot grasping of highly wrinkled clothes, 2012 IEEE International Conference on Robotics and Automation, 2012, Saint Paul, MN, USA, pp. 1703-1708, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • S. Foix, S. Kriegel, S. Fuchs, G. Alenyà and C. Torras. Information-gain view planning for free-form object reconstruction with a 3D ToF camera, 14th International Conference on Advanced Concepts for Intelligent Vision Systems, 2012, Brno, Czech Republic, in Advanced Concepts for Intelligent Vision Systems, Vol 7517 of Lecture Notes in Computer Science, pp. 36-47, 2012, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Rigual, A. Ramisa, G. Alenyà and C. Torras. Object detection methods for robot grasping: Experimental assessment and tuning, 15th Catalan Conference on Artificial Intelligence, 2012, Alacant, in Artificial Intelligence Research and Development, Vol 248 of Frontiers in Artificial Intelligence and Applications, pp. 123-132, 2012, IOS Press.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, A. Ramisa, G. Alenyà, C. Torras and F. Moreno-Noguer. Single image 3D human pose estimation from noisy observations, 2012 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2012, Providence, RI, USA, pp. 2673-2680, IEEE Press.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé and C. Torras. Redundant inverse kinematics: Experimental comparative review and two enhancements, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, Vilamoura, Portugal, pp. 5333-5340, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • P. Monsó, G. Alenyà and C. Torras. POMDP approach to robotized clothes separation, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, Vilamoura, Portugal, pp. 1324-1329, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • L. Rozo, P. Jiménez and C. Torras. Robot learning from demonstration in the force domain, 2011 IJCAI Workshop on Agents Learning Interactively from Human Teachers, 2011, Barcelona, Spain, pp. 1-6.

    Open/Close abstract Abstract Info Info pdf PDF
  • L. Rozo, P. Jiménez and C. Torras. Robot learning from demonstration of force-based tasks with multiple solution trajectories, 15th International Conference on Advanced Robotics, 2011, Tallin, Estonia, pp. 124-129, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • E.E. Aksoy, B. Dellen, M. Tamosiunaite and F. Wörgötter. Execution of a dual-object (pushing) action with semantic event chains, 11th IEEE-RAS International Conference on Humanoid Robots, 2011, Bled, Slovenia, pp. 576-583, IEEE Press.

    Open/Close abstract Abstract Info Info pdf PDF
  • A.A. Ortega and J. Andrade-Cetto. Segmentation of dynamic objects from laser data, 5th European Conference on Mobile Robots, 2011, Örebro, Sweden, pp. 115-121, Örebro Univ.

    Open/Close abstract Abstract Info Info pdf PDF
  • E.H. Teniente and J. Andrade-Cetto. FaMSA: Fast multi-scan alignment with partially known correspondences, 5th European Conference on Mobile Robots, 2011, Örebro, Sweden, pp. 139-144, Örebro Univ.

    Open/Close abstract Abstract Info Info pdf PDF
  • E.H. Teniente, R. Valencia and J. Andrade-Cetto. Dense outdoor 3D mapping and navigation with Pose SLAM, 2011 Workshop de Robótica Experimental, 2011, Seville, pp. 567-572.

    Open/Close abstract Abstract Info Info pdf PDF

Other Publications

  • A. Colomé and C. Torras. Reinforcement Learning of Bimanual Robot Skills. Volume 134 of Springer Tracts in Advanced Robotics. Springer, 2020.

    Open/Close abstract Abstract Info Info
  • G. Alenyà, S. Foix and C. Torras. ToF cameras for eye-in-hand robotics. In Optical Imaging Devices: New Technologies and Applications, 117-147. CRC Press - Francis Taylor Group, 2015.

    Open/Close abstract Abstract Info Info pdf PDF