We model six degrees of freedom: virtual eyeball pitch/yaw, physical neck pan/tilt, and optional ear rotation. A weighted, differential inverse-kinematics solver prioritises high-frequency saccades in the eyes while distributing slower corrections to the neck, thereby emulating the human vestibulo-ocular reflex. During larger gestures—nods, shakes, “look-over-there” sweeps—the solver injects additional constraints so that the pupils cling to the target despite head motion.

Every render frame, a 256 × 256 px tile is cropped around the attended point in the RGB-D camera feed, flipped horizontally to preserve left-right semantics and alpha-blended at variable opacity atop the pupil layer. Dynamic reflection scaling ensures distant and close objects remain recognizable. The entire OpenCV shader runs at 60 fps on a single Jetson Orin core, leaving ample head-room for perception and planning tasks. 

Ten primitives sit on a continuum from human-like to “augmented-reality” cues:

1. stylized iris and pupil, 2. reduced pupil size, 3. positive state, 4. negative state, 5. eyes closed, Superhuman augmentations: 6. processing animation, 7. color coding of robot state, 8. focused on a person (blurred), 9. focused on objects at lower reflection opacity + loading animation indicating processing, 10. focus on an object with a brief overexposure (flash) indicating first registration.

With 33 participants, MirrorEyes cut error-interruption time by ~8 % and lifted UEQ-S user-experience scores from “Good” to “Excellent” versus eyes-only baselines—evidence that the design is not just eye-catching but truly helpful.

Please visit our paper website for more details about MirrorEyes.

Head Design

The whole module rides on a high-speed, whisper-quiet pan-tilt unit, enabling expressive nods, shakes, and directional gazing. All structural parts are 3-D-printed in light-grey ABS to match the robot arms; an 80 % - transmittance acrylic cover masks the bezels and adds depth to the screen. Multiple print - test - refine loops optimized stiffness, weight, and maintainability. 

The motor behavior is generated through a hybrid pipeline: LLM prompts propose high-level social cues that are translated into rule-based motion primitives, producing responses that feel both vivid and timely. We will release the complete CAD files, firmware, and bill-of-materials as open-source so other researchers can replicate the head using inexpensive off-the-shelf components. 

For more detail about the LLM-based expression generation, please see our paper webpage.

The perceptive abilities include:

• action detection to recognize what the humans are doing

• open-world perception to recognize novel objects

• inspecting objects to actively inspect objects physically

• person identification to remember previously met humans

These abilities fuel the V/LLM-based framework with the situational information required to decide what to do.The actionable abilities include:

• collaborative actions to physically cooperate

• explainable interaction to inform the human on current plans – or explain observed mistakes and ask for help

• mirror eyes to intuitively and visually transmit the robots’ attention and intention

• flexible turn-taking for fluid interactions with less waiting

• multi-party interaction to provide support within a group

More detail about the VLM-based robotic system, please visit our paper websites: CuriousRobot, AttentiveSupport