Lab 11

1. The goal of this lab is to enable Cozmo to distinguish the following seven symbols (Figure 1) from each other, and from other random scenes the robot might see. As input, your algorithm will be given images previously taken by the robot, as shown in Figure 2. You must determine whether one of the seven symbols is in the image, and if so, which one.

We provide an image dataset containing grayscale images taken by the robot. The dataset contains 8 types of images, corresponding to either one for each of the seven symbols or the "none" category for pictures not containing any symbol (just a picture of an empty arena, cube or wall). Each of the symbols is flanked by vertical bars on the left and right, as can be seen in Figure 2. These can be used to locate the symbol in an image if needed.

We recommend using scikit-image and scikit-learn libraries. If you do not have prior experience with these tools, we recommend reading through:

• "Getting Started" and "A crash course on Numpy for images" section of the scikit-image user guide
• "An introduction to machine learning with scikit-learn" section of the scikit-learn user guide

For both scikit-image and scikit-learn, you are not restricted to only using methods covered in class; you can use the entire functionality of both libraries.

2. Pull the latest lab code to obtain the lab11/ directory. You are provided with the following files:

• train/ - directory containing images to be used for training your model. The correct label of each image is listed in its file name.
• test/ - directory containing images to be used for testing your model. The correct label of each image is listed in its file name.
• imgclassification.py – this is the main file where you will enter your solution. The code already contains functions for reading in the images from the train/ and test/ directories and for formatting the data into an array. Add code to convert pixel values of the images to features that can be used for training a classifier. Then train a classifier of your choice and test its predictive performance on the test set. You will need to fill in three functions in the ImageClassifier class, namely, extract_image_features, train_classifier, and predict_labels. To have your submission correctly graded, please do not modify the header and the return type for each function. You may add helper functions as needed, but make sure your code is self-contained.

We will evaluate the performance of your algorithm on 50 images. 25 images will come from a withheld test set containing images that were taken under the same conditions as the ones you have been provided, but that were not included in your dataset. The other 25 images will be a random selection of the images you have already seen in the test set. The training set will remain the same.

3. Next you will demonstrate the usage of your trained classifier in real time by the Cozmo robot. Develop a simple robot program that starts in an "idle" state and responds to seeing three different cards by triggering three different robot responses. You can print your own cards from this pdf.

Choose three of the cards that your classifier is trained to recognize. In response to seeing each card, the robot should use speech to announce the label name of the detected card and perform a physical movement. You can use one of the built-in Cozmo animations or create one yourself. You are not given any starter code for this part of the lab but you can easily reuse the skeleton of code provided in earlier labs. Be sure to add your new file to your lab11/ directory for your submission.

Make sure that your program is robust to occasional false alarms by looking at recognition results over a certain time window, rather than reacting to the detection at every time step.

To complete this part of the lab, make a short video that demonstrates the robot's reaction to three different cards.