Research Project

RobInstruct: Instructing robots using natural communication skills

Type

National Project

Start Date

01/01/2015

End Date

31/12/2017

Project Code

TIN2014-58178-R

Project illustration

Staff

Project Description

The demographic projections for the EU (low birth rate and an ageing population) open a series of social, medical and economical challenges that demand for new technologies for the assistance of elderly and disabled people, both indoors and outdoors. A key aspect which should be researched is that of devising such new user-centric technologies in a way that they can easily adapt to the specific needs of the end-users.

In robotics, the goal of allowing an inexperienced end-user to program a robot with a new desired behavior has been pursued using Learning from Demonstration approaches, where the human teaches the robot by simply showing it how to perform the task. A typical set-up consists of a manipulator arm, teleoperated by the user through a haptic device to, for instance, pour water into a glass. The demonstration is usually repeated several times so the robot can learn the different steps, the essence, and the variability of the task. Yet, this procedure is not as natural as we would expect. Ideally, we would like to get rid of any specialized device and teach the robot as we teach another person, just by showing how the human performs the task or simply narrating the steps needed. The robot should be able to perceive the meaningful actions, identify the tools being used, and extract the essential knowledge of the process to be able to actually perform the entire task itself. The first main objective of the RobInstruct project is thus to move a step forward from existing approaches in robot learning, and develop the technology to instruct a general-purpose robot in a natural and human-like manner.

Additionally, most general-purpose robotic assistants designed so far, are restricted to indoor and non-urban scenarios, ignoring the important part of the therapy that represents the social interaction out-of-home. As a second objective of the project we will therefore pursue a number of assistance tasks in urban areas, such as streets, a university campus or a shopping mall.

To tackle these problems, we will put together tools from computer vision, machine learning, natural language processing and robotics. Specifically, to teach robots in a natural and human-like manner we will first develop parsers to represent both video and natural language data using an intermediate abstraction level. We will then investigate learning approaches to discover mappings between the visual/textual content and the robot action space. These algorithms should address several issues: heterogeneity of possible end- users, differences between the human and the robot kinematics, and large amounts of perceptual noise (as in outdoor settings), an element typically overlooked in the past.

In order to bring these assistive robots to outdoor and uncontrolled scenarios, the human-to- robot communication skills will be combined with new algorithms to reliably localize the robot in very large 3D maps and for long periods of time, even in GPS-denied areas. For this purpose, we will integrate novel computer vision pose estimation algorithms with inertial sensors.

The two main objectives we pursue are commercially and socially relevant robotics technologies, as endorsed by our three EPOs. With the development of such technologies, the project will contribute to more flexible and general use of robots in assistive tasks, as well as more robust service robots able to navigate and interact in previously unknown and unstructured environments.

Project Publications

Journal Publications

  • M. Villamizar, J. Andrade-Cetto, A. Sanfeliu and F. Moreno-Noguer. Boosted random ferns for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Colomé and C. Torras. Dual REPS: A generalization of relative entropy policy search exploiting bad experiences. IEEE Transactions on Robotics, 2017, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Santamaria-Navarro, P. Grosch, V. Lippiello, J. Solà and J. Andrade-Cetto. Uncalibrated visual servo for unmanned aerial manipulation. IEEE/ASME Transactions on Mechatronics, 2017, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Combining local-physical and global-statistical models for sequential deformable shape from motion. International Journal of Computer Vision, 122(2): 371-387, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • R. Rossi, A. Santamaria-Navarro, J. Andrade-Cetto and P. Rocco. Trajectory generation for unmanned aerial manipulators through quadratic programming. IEEE Robotics and Automation Letters, 2(2): 389-396, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, J.M. Martínez, L. Agapito and B. Calvo. Modal space: A physics-based model for sequential estimation of time-varying shape from monocular video. Journal of Mathematical Imaging and Vision, 57(1): 75–98, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, C. Torras and F. Moreno-Noguer. 3D human pose tracking priors using geodesic mixture models. International Journal of Computer Vision, 2017, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • J. Deray, J. Solà and J. Andrade-Cetto. Word ordering and document adjacency for large loop closure detection in 2D laser maps. IEEE Robotics and Automation Letters, 2(3): 1532-1539, 2017.

    Open/Close abstract Abstract Info Info pdf PDF
  • M. Villamizar, A. Garrell Zulueta, A. Sanfeliu and F. Moreno-Noguer. Interactive multiple object learning with scanty human supervision. Computer Vision and Image Understanding, 149: 51-64, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Moreno-Noguer and J.M. Porta. A Bayesian approach to simultaneously recover camera pose and non-rigid shape from monocular images. Image and Vision Computing, 52: 141-153, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Sanromà, A. Penate-Sanchez, R. Alquézar Mancho, F. Serratosa, F. Moreno-Noguer, J. Andrade-Cetto and M.A. González. MSClique: Multiple structure discovery through the maximum weighted clique problem. PLOS One, 11(1): e0145846, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • V. Lippiello, J. Cacace, A. Santamaria-Navarro, J. Andrade-Cetto, M.A. Trujillo, Y. Rodriguez and A. Viguria. Hybrid visual servoing with hierarchical task composition for aerial manipulation. IEEE Robotics and Automation Letters, 1(1): 259-266, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • C. Torras. Service robots for citizens of the future. European Review, 24(1): 17-30, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • J.G. Hoyos, F. Prieto, G. Alenyà and C. Torras. Execution Fault Recovery in Robot Programming by Demonstration Using Multiple Models. IEEE Latin America Transactions , 14(2): 517-523, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, F. Moreno-Noguer, B. Calvo and J.M. Martínez. Real-time 3D reconstruction of non-rigid shapes with a single moving camera. Computer Vision and Image Understanding, 153(12): 37–54, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Husain, H. Schulz, B. Dellen, C. Torras and S. Behnke. Combining semantic and geometric features for object class segmentation of indoor scenes. IEEE Robotics and Automation Letters, 2(1): 49-55, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • J.G. Hoyos, F. Prieto, G. Alenyà and C. Torras. Incremental learning of skills in a task-parameterized Gaussian mixture model. Journal of Intelligent and Robotic Systems, 82(1): 81-99, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, F. Moreno-Noguer, B. Calvo and J.M. Martínez. Sequential non-rigid structure from motion using physical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(5): 979-994, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • F. Husain, B. Dellen and C. Torras. Action recognition based on efficient deep feature learning in the spatio-temporal domain. IEEE Robotics and Automation Letters, 1(2): 984-991, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa, G. Alenyà, F. Moreno-Noguer and C. Torras. A 3D descriptor to detect task-oriented grasping points in clothing. Pattern Recognition, 60: 936-948, 2016.

    Open/Close abstract Abstract Info Info pdf PDF

Conference Publications

  • A. Agudo and F. Moreno-Noguer. DUST: Dual union of spatio-temporal subspaces for monocular multiple object 3D reconstruction, 2017 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2017, Honolulu, USA, IEEE, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Global model with local interpretation for dynamic shape reconstruction, 2017 IEEE Winter Conference on Applications of Computer Vision, 2017, Santa Rosa (California), USA, IEEE, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Pumarola, A. Vakhitov, A. Agudo, A. Sanfeliu and F. Moreno-Noguer. PL-SLAM: Real-time monocular visual SLAM with points and lines, 2017 IEEE International Conference on Robotics and Automation, 2017, Singapore, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Gabás, E. Corona, G. Alenyà and C. Torras. Robot-aided cloth classification using depth information and CNNs, 9th Conference on Articulated Motion and Deformable Objects, 2016, Palma de Mallorca, Spain, in Articulated Motion of Deformable Objects, Vol 9756 of Lecture Notes in Computer Science, pp. 16-23, 2016, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Recovering pose and 3D deformable shape from multi-instance image ensembles, 13th Asian Conference on Computer Vision, 2016, Taipei, Taiwan, in Computer Vision – ACCV 2016, Vol 10114 of Lecture Notes in Computer Science, pp. 291-307, 2017, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Canal, G. Alenyà and C. Torras. Personalization framework for adaptive robotic feeding assistance, 8th International Conference on Social Robotics, 2016, Kansas City, USA, in Social Robotics, Vol 9979 of Lecture Notes in Artificial Intelligence, pp. 22-31, 2016, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Jevtić, A. Colomé, G. Alenyà and C. Torras. User evaluation of an interactive learning framework for single-arm and dual-arm robots, 8th International Conference on Social Robotics, 2016, Kansas City, USA, in Social Robotics, Vol 9979 of Lecture Notes in Artificial Intelligence, pp. 52-61, 2016, Springer.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo, J.M. Martínez, B. Calvo and F. Moreno-Noguer. Mode-shape interpretation: Re-thinking modal space for recovering deformable shapes, 2016 IEEE Winter Conference on Applications of Computer Vision, 2016, Lake Placid, USA, pp. 1-8, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • D. Martínez, G. Alenyà, C. Torras, T. Ribeiro and K. Inoue. Learning relational dynamics of stochastic domains for planning, 26th International Conference on Automated Planning and Scheduling, 2016, London, pp. 235-243.

    Open/Close abstract Abstract Info Info pdf PDF
  • G. Martín, F. Husain, H. Schulz, S. Frintrop, C. Torras and S. Behnke. Semantic segmentation priors for object discovery, 23rd International Conference on Pattern Recognition, 2016, Cancún, Mexico, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Rubio, L. Yu, E. Simo-Serra and F. Moreno-Noguer. BASS: Boundary-aware superpixel segmentation, 23rd International Conference on Pattern Recognition, 2016, Cancún, Mexico, to appear.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Vakhitov, J. Funke and F. Moreno-Noguer. Accurate and linear time pose estimation from points and lines, 14th European Conference on Computer Vision, 2016, Amsterdam, in Computer Vision - ECCV 2016, Vol 9911 of Lecture Notes in Computer Science, pp. 583-599, 2016.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Quattoni, A. Ramisa, P. Swaroop, E. Simo-Serra and F. Moreno-Noguer. Structured prediction with output embeddings for semantic image annotation, 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2016, San Diego, pp. 552-557.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Corominas Murtra, J. Vallvé, J. Solà, I. Flores and J. Andrade-Cetto. Observability analysis and optimal sensor placement in stereo radar odometry, 2016 IEEE International Conference on Robotics and Automation, 2016, Stockholm, Sweden, pp. 3161-3166.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Ramisa, J. Wang, Y. Lu, E. Dellandrea, F. Moreno-Noguer and R. Gaizauskas. Combining geometric, textual and visual features for predicting prepositions in image descriptions, 2015 Conference on Empirical Methods in Natural Language Processing, 2015, Lisbon, pp. 214-220.

    Open/Close abstract Abstract Info Info pdf PDF
  • E. Simo-Serra, E. Trulls Fortuny, L. Ferraz, I. Kokkinos, P. Fua and F. Moreno-Noguer. Discriminative learning of deep convolutional feature point descriptors, 2015 International Conference on Computer Vision, 2015, Santiago de Chile, pp. 118-126, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Penate-Sanchez, L. Porzi and F. Moreno-Noguer. Matchability prediction for full-search template matching algorithms, 2015 International Conference on 3D Vision, 2015, Lyon, pp. 353-361.

    Open/Close abstract Abstract Info Info pdf PDF
  • A. Agudo and F. Moreno-Noguer. Learning shape, motion and elastic models in force space, 2015 International Conference on Computer Vision, 2015, Santiago de Chile, pp. 756-764, IEEE.

    Open/Close abstract Abstract Info Info pdf PDF
  • L.D. Ellebracht, A. Ramisa, P. Swaroop, J.A. Cordero, F. Moreno-Noguer and A. Quattoni. Semantic tuples for evaluation of image sentence generation, 4th Workshop on Vision and Language, 2015, Lisbon.

    Open/Close abstract Abstract Info Info pdf PDF

Other Publications

  • C. Torras. Robot pain: A speculative review of its functions. In Pain and the Conscious Brain, 235-246. Wolters Kluwer, 2016.

    Open/Close abstract Abstract Info Info pdf PDF