Collaborators: Joyce Moon, Joyce Wang, Ritika Srivastava, Corey Zhou, Alex Hong

General Topic Outline: Our group has decided to work on teleoperations with the Baxter robot in which we will be developing a controller to move Baxter. We can use Leap Motion to get the movement of the hand, and the robot can consequently move its arms to move the way our hand moved. Baxter robot can automatically adjust the angles of the joint to locate its 0hand in certain position. So, only with information of where the hand should be, it can perform the movement without the information of how to move provided (angle,etc). So leap motion can give us accurate location in 3d space and if we send that information to the robot, it can process it.

Motivation:

• Interest in VR/AR
• Teleoperation is a fun topic
• Allows for us to interact with the robot from a distance, (in the future this can be expanded to various fields)

Idea:

• We use Leap Motion to get the 3D movement of the hand, and the robot will move based on the hand movement. Baxter robot can automatically adjust the angles of the joint to locate its hand in certain position.

Purpose:

• To control Baxter robot more intuitively and from a distance

Shopping List:

• 1 Leap motion controller (need it for capturing hand movements) http://store-us.leapmotion.com/products/leap-motion-controller

• 1 Google cardboard (need it because it is a vr-headset) https://store.google.com/config/google_cardboard?sku=_google_cardboard_single&gl=us

• 1 Dual lens camera (to see what the robots sees so we can control it from a distance)

• 1 Kinect (baxter has 7 deg of freedom which can be further explored with the connect since leap motion only accounts for 6 of them) https://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-Sensor-for-Xbox-One/productID.2267482500

Project Plan

Progress:

Virtual Reality Aspect: Camera on Ceiling

Preprocessing raw Leap Motion data

• Grab data
  • hand location (x, y, z)
  • hand orientation (roll, pitch, yaw)
  • pinch strength
• Convert coordinate system (coodinates & orientation)
  • leap (x+, y+, z+) → ROS (y-, z, x-)
• Communicate with ROS via Tornado websockets

Displacement calculation

• Compare most recent position and orientation data with reference points
• Calculate and scale the difference (displacement)
• Determine the next position for Baxter to move to

RPI Livestreaming -> Local Computer -> Phone

• RPI streams 2 live-videos (left & right camera)
• Computer receives and sends html through local server
  • CSS to format Place the live video side by side.
• Phone placed into Google Cardboard
• Tweaked around camera resolution to decrease the lag live-streaming.
• Placed the “eyes” up to ceiling for 3rd person view
  • Narrow visual angle

Final Thoughts on Project:

We were all exposed to various technologies we had not dealt with before.

• More accurate motion mirroring in terms of both location and speed
• Smoother baxter
• Having two 360 camera to enable wider vision and more immersive experience in Virtual Reality.
• Include Leap Motion hand visualization in VR, so that user can be more aware of their hands in the space
• Integration with Kinect as to account for the 7th degree of freedom

Final Presentation Deck