May 5, 2025
From our Post #8 analysis, the robot must perceive:
Human arm placement: To ensure proper positioning of the robot’s gripper and force sensor.
Chair location: To align the robot base parallel to the user for effective left-right arm stretching.
These need to be detected at moderate spatial precision within a few inches and with continuous updates (at least every 50–100 ms during active exercises).
Our general approach uses Stretch’s built-in cameras and established computer vision tools. By integrating models such that the robot can recognize the human hand, arm, and chair position in real time. These detections feed into a control loop where the robot checks whether the user is applying pressure to the gripper sensor. If the hand is removed or misaligned, the robot pauses the session and logs the result. This approach provides autonomy and flexibility but can be prone to failure under poor lighting, movement occlusion, or background clutter.
To make perception more reliable and less dependent on ideal conditions, we also propose modifying the physical environment. We plan to use ArUco markers (high-contrast visual tags) placed on the chair to help the robot identify its position on the map quickly and reliably. Additionally, colored floor tape will guide the robot’s straight-line motion, reducing the chance of misalignment during continuous exercises. These physical modifications are inexpensive, easy to implement, and dramatically improve perception reliability in a controlled setting like a clinic.
Recognizing that even robust systems need a fallback, we’ve also designed a human-in-the-loop setup process. Before each session, a doctor will adjust the robot and the user’s chair to ensure proper alignment. We plan to build a simple interface that shows a camera preview, allowing the staff member to view a camera preview and confirm that the user and robot are correctly positioned. This small intervention at the beginning of each session dramatically reduces the chances of error during autonomous operation and doesn’t add significant overhead for staff, allowing a single doctor to control a group of six robot + patient pairs.
To ensure safe operation throughout the session, the robot will continuously monitor a pressure sensor on the gripper. It will check every 50 milliseconds to verify that the user’s hand remains in contact. If the hand is removed or force drops below a threshold, the robot will automatically stop moving, notify the user, and save the session data. This adds a quick fallback in case visual perception fails or unexpected movement from the user occurs.
By combining computer vision, environmental changes, and interactive human setup, we create a perception strategy that is both practical and reliable. Each method complements the others: vision systems provide general autonomy, markers and guides reduce software complexity, and human input ensures fail-safe alignment. This hybrid approach ensures the Stretch robot will be effective and safe in real-world therapy environments, even when operating with multiple users in parallel.