Assignments

FINAL VIDEO & DEMONSTRATION (20 points)

Due: Mar 18th, 12:00noon

1. (16 points) VIDEO PRESENTATION: You final assignment involves creating a video that presents your project, based on the script you prepared in Week #9 blog. Your video should be three minutes or shorter. Your video should both introduce and "sell" your design. The following example videos might inspire you:

• Videos on Kickstarter (e.g. apps, software, robots, web, wearables, or any other) or Indiegogo (e.g. technology, search results for "apps")
• Videos created for projects in previous robotics capstones: 2016, 2017, 2018
• Videos created in HCI courses, e.g. iDrinkSafe, TasteBud, Aqueous, Neighborly, Broccoli for All

Your video script (Week #9 blog) should include a plan for narration and accompanying visuals (footage or images). Your script should be in the form of a three column table.

1. The first should include a section number and a short section title (e.g. "introduction", "demonstration of task 1" or "testimonial from test user").
2. The second should be the a description of visual elements that will be shown in the video (e.g. "talking head of team member A", "video of robot manipulating object", "screen recording of someone using the UI"). Include sufficient details such that someone who reads your description could go and shoot that video for you.
3. The last should be the narration text that will be read voice-over on the video, if the video does not already include relevant audio. This includes what will be said in "talking head" shots.

To allow for creative freedom and avoid obtaining many cookie-cutter videos, we are not constraining the sections in your video. But please make sure that your video follows these guidelines:

• Your video should clearly describe and motivate the problem. Get the viewers to care about the problem.
• Your video should clearly describe your solution and convince the viewer that it addresses the problem.
  • It should be clear what your design is useful for to someone who is hearing about the project for the first time.
  • Your video should demonstrate the use of your robot. This can be done by walking through different tasks that your robot supports. Be sure to set context for the use and demonstrate the outcome (just like in your storyboards).
• Your video should feature positive feedback from potential users regarding your design (quotes from your interviews or new recordings of people saying what they like about your design or describing contexts in which they would use your design if it were real).
• Your video should introduce you, the team.

You will receive feedback about your scripts shortly after submitting them. Afterwards you will have a week to shoot and edit your videos. Try to incorporate any feedback you received about your script.

The evaluation of this assignment will be done by a panel of judges and it has two components:

• Evaluation of the video: In addition they will assess the quality of your video. Besides covering important content, it is important for your video to have appeal.
• Evaluation of the project: The judges will assess the importance and quality of your project. Note that this assessment depends on the judge's ability to understand your project based on the video.

The rating sheet that the judges will use is available here. To avoid file format related challenges and allow judges to view videos remotely, we request that you upload your video to YouTube (you can make it Unlisted if desired) and submit a link to the video on Canvas.

2. (4 points) DEMONSTRATION: In addition to the video, you will demonstrate your project live on the robot during the final presentation event. The judges will expect to see the core functionality shown in the video to work similarly on the robot and will try variations of the demonstration. You will have 10 minutes to prepare and demonstrate the capability. We encourage creating checklists and launch scripts to simplify getting your demo running as well as rehearsing getting it to run. One person from the team should describe the demonstration while others set up the demo environment, run the necessary scripts, keep an eye on the debug screens, or pretend to interact with the robot for the demonstration. If a team's demo does not work during the event, a make up time can be scheduled later during the finals week with no penalty.

FETCH PROJECTS:

1. (4 points) Post #1: Labs 27-29 will show you how to work on perception problems in simulation and introduce you to a package for tracking fiducials in the pointcloud stream from Fetch’s PrimeSense camera. Using this and other functionality developed in previous assignments, you will develop a new system that allows you to: (a) save end-effector poses relative to a fiducial and (b) move the robot’s arm to the same relative poses after the fiducial has been moved. Using this system you will be able to define different actions for manipulating objects that have fiducials attached to them, simply by saving a sequence of end-effector poses and gripper states (open/closed) relative to the fiducial on the object.

The system should allow you to specify end-effector poses by physically moving the robot's arms to the desired poses. It should detect the pose based on the robot's sensor data (i.e., using TF). You will need some kind of interface (command-line or web-based) to save poses after moving the arm to each pose. For each pose, you will need to use your interface to specify which fiducial the pose is relative to (or if the pose is relative to the robot's base). Once you have finished defining an action, you should use your interface to save the action to a file. You also need to build a program executor that can load an action from a file and execute it.

To demonstrate the tools developed in this assignment, assume that you have box shaped object with a fiducial (F1) on its front face. Use of your tool *on the real robot* to define the following three actions:

• Push: Make the object with F1 move side-to-side by pushing it one way
• Poke: Make the object with F1 tip over by poking it from front-to-back
• Pick-and-place: Pick up the object with F1 from the top. Place the object on a second fiducial, F2.

Make a video to show how the tool is used for specifying three different actions as well as the execution of each action in two different initial configurations of the object with F1. Post your video with a brief description of what is being shown.

Your actions should actually succeed. So if you find that the action fails when you test it, you need to go back and re-define your actions differently (e.g. try having a different number of poses, using different pose configurations, or adjusting the motion speed).

2. (3 points) Post #2: Labs 30-32 will get you started on processing the robot's PointCloud stream to detect surfaces and segment objects that are on a surface. For this assignment we would like you to start thinking about using this functionality for your projects. Choose a representative object that needs to be perceived for your project (e.g. tennis ball) and tune the methods you learn in the labs to work for segmenting that particular object. Your blog post should include the following:

• A list of objects/landmarks that need to be perceived for your project. You might already have this in Blog #2, Post #2 (or in later revisions of your technical requirements). Discuss with the instructors if you are unsure what the full list is.
• A screen capture video showing a visualization of the outcome of your segmentation algorithm (similar to the screenshot in Lab 32) in a scene containing the object you chose for this assignment. Use a placeholder for objects that you do not have yet. Demonstrate the outcomes for different configurations of the scene by moving the objects around. Describe how you tuned the method to make it work for the specific object.
• For each object/landmark that needs to be perceived for your project, discuss the robustness of the clustering-based perception system you developed, how it might be improved, and whether it is a better option than attaching a fiducial to the object.

KURI PROJECTS:

1. (4 points) Post #1: Labs 28b-29b will show you how to work on perception problems in simulation and introduce you to a package for tracking fiducials in images from Kusi's camera. You will combine this new capability with functionality developed in previous weeks to allow specifying a sequence of robot configurations relative to a detected fiducial and navigating to those configurations when the fiducial moves. Using this tool you will be able to define robot actions relative to objects (augmented with a fiducial) in its environment.

The system should first detect the fiducial and record its initial pose. All robot poses saved after this should be in the reference frame of the fiducial. The system should allow you to specify robot poses by manually driving the robot to the desired pose and saving through a UI of your choice.

To demonstrate the functionality of this tool you will program Kuri to person two actions relative to a detected fiducial:

• Circle around the object on which the fiducial is attached
• Push the object on which the fiducial is attached for about 10 inches

Make a video to show how the tool is used for specifying two different actions as well as the execution of each action in two different initial configurations of the object relative to the robot. Post your video with a brief description of what is being shown.

Your actions should actually succeed. So if you find that the action fails when you test it, you need to go back and re-define your actions differently (e.g. try having a different number of poses, using different pose configurations, or adjusting the motion speed).

2. (3 points) Post #2: Labs 30b-32b will guide you through a speech processing tool integrated with ROS called Sphinx. You will train a new grammar for Sphinx based on representative verbal interactions between the user and the robot you wish to support in your project. For this try to come up with five verbal dialogs (supporting core functionality of your robot) with 4-10 turns (each utterance by the human or the robot is considered a turn). Once you have trained your grammar, make a video of someone from your team verbally interacting with Kuri to demonstrate the five interactions. Post the vide with a short description.

FETCH PROJECTS:

1. (2 points) Post #1: Labs 19-22 will introduce you to Inverse Kinematics (IK) and motion planning with obstacles and constraints. At the end of Lab 22 you will develop an RViz-based interface that involves an InteractiveMarker, shaped like the robot's gripper, which can be moved around to specify a target gripper pose and send the robot to that pose. When moved around, the marker should change color to reflect whether the corresponding gripper pose is reachable by the robot or not. After testing it in simulation, you will test this interface *on the real robot* to grasp a soft object. To do this, use your interface to (i) send the robot gripper to a "pre-grasp" pose near the object, (ii) then send it to a "grasp" pose, (iii) close the gripper, and (iv) send the gripper to a "lift" pose. Make a video that shows both the use of your RViz interface and the successful grasping the object by the robot. See Lab 26 for additional tips on how to accomplish this part of the assignment. Post this video with a brief description of what it demonstrates.
2. (3 points) Post #2: Labs 23-25 will introduce you to coordinate frame transforms using TF and transformation matrices. At the end of Lab 26 you will have an RViz-based interface that involves an InteractiveMarker corresponding to a world object which can be moved around. You will use transform arithmetic to compute "pre-grasp", "grasp", and "lift" poses for the gripper relative to the current pose of the object. You can use any type of visualization to indicate whether an object pose has reachable gripper poses associated with it (e.g. change color of the object and attach text or separately visualize the associated gripper poses in different colors). After testing it in simulation, you will test this interface *on the real robot* to grasp a soft object. To do this, use your interface to (i) place your interactive object marker approximately where the object appears in the robot's pointcloud stream, and (ii) send the robot to computed pre-grasp, grasp, and lift poses one at a time (close the gripper before lifting). See Lab 26 for additional tips on how to accomplish this part of the assignment. Make a video that shows both the use of your RViz interface and the successful grasping the object by the robot. Post this video with a brief description of what it demonstrates.

KURI PROJECTS:

1. (2 points) Post #1: Labs 19b-20b will introduce you to face detection on the Kuri and guide you through an exercise to make Kuri visually servo towards a detected face. The robot will use control wheels and its head pose to ensure that a detected face is roughly at the center of its camera image and it is of a particular size. Make a video of the servoing behavior demonstrating how (i) Kuri rotates, moves away, and moves forward to keep the detected person at a certain distance, and (ii) Kuri follows a person as they move continuously (but not too fast). Post this video with a brief description of what it demonstrates.
2. (3 points) Post #2: Labs 21b-26b will introduce you to coordinate frame transforms using TF and transformation matrices, and then guide you through making Kuri look at a 3D point using simplified inverse kinematics and navigate to a point relative to the pose of a detected face. It will also ask you to create emotional expressions for Kuri which can be triggered throughout the robot's execution. By the end of these labs you will make Kuri detect a face and localize it in 3D (expressing happiness when the face is present); and navigate in front of the person when commanded (responding to the command by nodding). Make a video of the full execution of this behavior and post it with a short description.

FETCH PROJECTS: Complete Labs 1-10 and use the browser-based teleoperation tool to control the simulated Fetch robot to do the following tasks:

• Drive from the starting pose to the table
• Raise the torso and lower the head to look at the table
• Move the arm (and possibly the base and torso) and open/close the gripper to pick up the blue cube on the table, without colliding with the table.

The tele-operator should be able to do these tasks without looking at the simulation (Gazebo) screen and without relying on their memory. In other words, the teleoperation interface should display sensory information to help guide the operator.

Make a screen capture video of the teleoperation process. At the end, open the Gazebo window to show that the object was successfully picked up by the Fetch robot.

KURI PROJECTS: Since our Kuri labs are currently under development, your task for this week is less structured and involves more exploration, but we will work more closely with you to help out as your efforts will contribute to future Kuri labs. The Fetch Labs 1-10 will serve as guidelines and you can still use the Fetch starter code as your templates. You will be able to skip some of the labs but be sure to read through them nonetheless for potentially useful general advise. Your goal is to develop a browser-based teleoperation interface that allows you to:

• Drive Kuri from the starting pose to another landmark
• Move the Kuri's head to look around
• Open and close Kuri's eyelids

The teleoperation interface should display Kuri's eye camera image to help guide the operator. Make a screen capture video of the teleoperation process demonstrating the three capabilities above.