Real-Time Non-Verbal Human-Robot Interaction with Reachy Mini

Abstract

Non-verbal interaction is a fundamental component of natural communication between humans, yet in human-robot interactions, it remains underexplored relative to language-centric approaches. Most existing systems lack believable non-verbal reaction capabilities when interacting with a human, and can not maintain the person’s suspension of disbelief in a free interaction environment. Such systems’ incapacities in performing non-verbal interactions are often attributed to their being unable to infer humans’ underlying emotions and intentions correctly, and produce a reaction in time, given their limited functionalities. In this project, we investigate how a compact humanoid robot, Reachy Mini, can autonomously perceive and react to human dynamic face and body gestures in real time, without relying on spoken language. The project includes three tightly coupled components: facial and head recognition, reaction generation, and a real-time robotic teleoperation interface. First, we study machine learning approaches for recognizing dynamic facial and head expressions using the Chehre dataset, focusing on identity-invariant facial cues. Second, we explore fine-tuning and prompt-engineering strategies for vision-language models to generate appropriate non-verbal reaction signals from structured face and gesture tokens. Third, we develop a real-time robotic puppeteering and awareness system that supports teleoperation, enabling direct comparison between human-operated and model-driven behaviors. The system is implemented and evaluated on both simulated and physical Reachy Mini robots. To assess perceived autonomy, human participants interact with the robot and judge whether its behavior appears teleoperated or autonomous. Through this project, we aim to advance non-verbal human-robot interaction by tightly integrating perception, generative reasoning, and embodied control.