OpenDR Project: enhancing healthcare robotics in trials with the TIAGo robot

As the OpenDR project approaches its finale, let’s take a moment to look over the three-year journey of the project and its rewarding achievements. OpenDR has focused on developing open-source software tools for robotics, with PAL Robotics playing a key role in this initiative.

Overview of the OpenDR project and enabling TIAGo to function as an assistant in healthcare settings

The OpenDR project, funded by the European Union, set out with an important goal: to develop open-source software tools that advance the capabilities of robotics in diverse sectors, including healthcare, education, and industrial automation. The aim of this project is creating tools that are not only advanced but also accessible and user-friendly. PAL Robotics’ role in the project was to integrate these deep learning algorithms into our TIAGo robot, equipping the robot with advanced features like human presence recognition, understanding of activities and vocal instructions, and emotional state detection, as a result enabling TIAGo to effectively function as an assistant in healthcare settings, with responsibilities including receiving visitors and handling objects.

Project results including the creation on an open deep learning toolkit for robotics.

A significant result of the OpenDR project is the creation and enhancement of various tools in an open deep learning toolkit for robotics. These tools include advanced perception algorithms and detailed manipulation techniques, all crafted to improve the abilities of robots in different environments. PAL Robotics played a key part in refining these tools, incorporating feedback from their use into our robots for ongoing improvement. This collaboration has led to tools that prove effective in challenging settings such as healthcare.

Tools developed/improved by the OpenDR project:

• Pose Estimation
• High Resolution Pose Estimation
• Fall Detection
• Wave Detection
• Face Detection
• Face Recognition
• 2D Object Detection
• 2D Single Object Tracking
• 2D Object Tracking
• Vision Based Panoptic Segmentation
• Semantic Segmentation BiSeNet
• Semantic Segmentation YOLOv8
• Binary High Resolution
• Image-based Facial Emotion Estimation
• Landmark-based Facial Expression Recognition
• Skeleton-based Human Action Recognition
• Video Human Activity Recognition
• RGB Hand Gesture Recognition
• End-to-End Multi-Modal Object Detection (GEM)
• RGBD Hand Gesture Recognition
• Continual SLAM
• Audiovisual Emotion Recognition
• Speech Command Recognition
• Speech Transcription
• Intent Recognition
• 3D Object Detection Voxel
• 3D Object Tracking AB3DMOT
• LiDAR Based Panoptic Segmentation
• Heart Anomaly Detection

All of these tools are available in both ROS1 and ROS2 interfaces, as well as a C interface for some of them.

PAL Robotics demo in a healthcare setting

In the healthcare sector, PAL Robotics has adeptly demonstrated the practical application of OpenDR tools through its TIAGo robot. This mobile manipulator, equipped with cutting-edge tools from the OpenDR project, has been deployed in a simulated healthcare setting to assist and support staff and patients. TIAGo in OpenDR, with the ability to recognise people, respond to vocal instructions, and even gauge emotional states, stands as a clear example of the success of the project. This use case illustrates how the integration of OpenDR tools has empowered TIAGo to perform tasks ranging from assisting patients to facilitating healthcare workers, marking a significant step forward in multi-modal human-robot interaction.

Looking ahead, the tools developed under the OpenDR project hold great potential for impacting healthcare and other sectors. PAL Robotics sees these innovations as stepping stones to more widespread use of robotics in daily life, contributing positively to society.

The end of the OpenDR project marks not an end, but contributes to what we hope will be the beginning of a new era in robotics, especially in healthcare. PAL Robotics is proud to have been a part of this initiative and is excited about the future possibilities these tools bring.

TIAGo the mobile manipulator robot

TIAGo robot combines perception, navigation, manipulation, and Human-Robot Interaction skills out of the box. TIAGo the mobile manipulator robot is customisable and is available with single and dual arms, and is able to support a wide variety of end effectors (grippers or hands). The robot is one of the most popular research platforms for universities and institutions across Europe and is also the robot of choice in numerous EU-funded projects.
At PAL Robotics we are partners in collaborative projects in healthcare and Ambient Assisted Living, as well as areas such as agri-food, smart cities, factories of the future, Artificial Intelligence, and Industry 4.0 and 5.0. For more information about PAL Robotics and our involvement in other collaborative projects, visit our website and if you have any questions, make sure you get in touch with us.

OpenDR receives funding from the European Horizon 2020 research and innovation programme under grant agreement No 871449.