Abstract:

In this paper, we present a tool flow which provides an opportunity for users to transfer their gesture and speech to a robot. Based on our design requirements to protect user's privacy and ensure the system's sustainability and compatibility by engaging users as participants, our goal is to create a user-friendly software system which allows a user with less technical background to animate a robot. As a core part of this system, we created a python-based tool flow which utilizes Mediapipe for motion detection and Vosk for automatic speech recognition. The tool flow, we present in this paper, is conceptually generic enough to be adapted for scenarios using various robots. As a proof of concept, we applied this tool flow to realize a robotic teacher by using an intelligent tutoring system (ITS) and a humanoid robot.

Keywords:

Mediapipe, teacher robot, gesture recognition, automatic speech recognition, tool flow, Vosk.