We have programmed the robot to accomplish several different tasks over 10 weeks, however there is still a lot of work that we would have liked to be implemented as shown below:
Improve camera accuracy by using an alternate method of extracting the centerpoint of the desired object
Instead of using Gemini API, we isolate the desired image and use the contours and calculate the centerpoint based on the moments of those contours
Smooth out robotic motion
This will hopefully improve with a better detection model of the objects and their centerpoints. Otherwise, we try adding in torque limits to prevent unnecessary gripper movements.
Implement a hand detection model
We use MediaPipe to track specific points on the hand which will enable better control of where we want the robot to pour the medicine bottle into
Add in more features:
Detect a cup and “pour” the medicine bottle into the cup
Robot drives to human with the medicine bottle
For fun: Robot gives you a fistbump after it completes the task