ROS_VPT package

In this section, I will walk through how to get the ros_vpt package set up in your local environment.

First, clone the 2024 UROP Repository and copy the ros_vpt directory into your ROS workspace.

• git clone https://github.com/UMich-MAVRIC/2024UROP-Robot_VPT.git

• cp 2024UROP-Robot_VPT/ros_vpt  <location/to/ros/workspace>

Then, navigate to the ros_vpt package location and open the model/ directory. Here, follow the download_model.md instructions and place the two models in the directory.

• Make sure you have the robot camera and human camera in the tf tree and the tf tree is being published

• Also, ensure that the robot's camera is publishing its image to a rostopic.

• First, run roscd and navigate to the ros_vpt package's scripts directory: cd src/ros_vpt/scripts

• Then, open the output_tf_location.py file in your favorite text editor/IDE

• Navigate to and fix all of the comments that have a header "CHECKME:" or "TODO:"

  • For the first comment (line 24), replace the rostopic to whatever the robot's camera publishing topic is.

  • For the second and third comments (line 32,33), replace the matrix with the camera matrix of the simulated camera, and then remember to edit cx and cy later.

  • For the fourth comment (line 65), replace the topics with the tf message topics for the human's location and camera's location.

  • Finally, for the last comment (line 83), comment and uncomment the lines to your liking

    • cv2.imshow() will display the image directly to your screen

    • cv2.imsave() will save the image to output.jpg

• Now, run the simulation in ISAAC sim!

• And then, run the command rosrun ros_vpt output_tf_location.py

• Depending on your choices, you might see a window pop up, but if you receive a log message that says "published image to /gaze_hsrb", it should be good to go!

• Finally, open rqt and subscribe the /gaze_hsrb topic through the "Image View" plugin!

• An image should appear with a blue line drawn where the model believes the person is looking!

insert image here

Notes:

• If the blue line's location is slightly offset all the time, edit the cx and cy values in the camera matrix located in output_tf_location.py. (Top right and middle right values)

• If you want to end the program, press "q" on the keyboard and the script should stop running.

Running in a real world environment

TBD