Implementación de un sistema de interacción humano-máquina que emplee lenguaje multimodal en una plataforma robótica basada en ROS.

Abstract

This project seeks to make a contribution to the research and improvement of human-machine communication, in order to be able to have a much more fluid and natural conversation in which the machine, in addition to having correct and consistent answers with the topic in conversation. , have movements and gestures according to the state of mind that is identified, that is, that the machine is capable of recognizing if the person is in an exalted, calm or neutral state of mind while they are having a dialogue and with respect to that mood recognition , the movements of their hands, arms, torso and other joints and indicators that the robotic platform has, represent the way in which the conversation is taking place.

The process begins with a state of the art, which allows the understanding and study of various investigations related to the environment of robotics and multimodal communication, in order to have clearer ideas about the developments and the important points to take into account. An analysis of the robotic platforms that can be used is also made in order to define which one is the most suitable with respect to the limitations in software and hardware. We proceed to carry out the development and construction of tools such as: the vision that is responsible for collecting the coordinates of people's postures, voice recognition, which converts audio into text and is sent to text analysis (ChatBot ) where a processing of the obtained phrase and gives a corresponding answer. In order to have a more natural communication with the human, it is necessary for the platform to perform body movements, therefore, with the information collected with vision and text analysis, a selection of the robot's movements is made that matches the type of conversation .