Vision Processing

The vision system uses a Logitech c920 web-camera mounted 2 feet above the board. Because the game is set up in a grid like fashion, the vision system separates each region of the image to a card location.

The vision system is required to evaluate the elements on each card and determine the number, the color, and shape of the elements on each card. The vision system then returns a set of card properties to the robotic "agent." Prior to starting the game, cards are placed on the board. The system takes a picture of the cards and for each card in the grid, it finds the contours of the elements and asks a user to classify them. These training samples are stored and used for classification of each card. Once the system has enough training samples, the game can begin. For each card on the board, the system first finds all of the contours in the designated region. Because the contours can be split into different hierarchical levels:

"Just like nested figures. In this case, we call outer one as parent and inner one as child. This way, contours in an image has some relationship to each other. And we can specify how one contour is connected to each other, like, is it child of some other contour, or is it a parent etc. Representation of this relationship is called the Hierarchy." OpenCV contour hierarchy

[next on level, previous on level, first child, parent]

We use the relationships to determine if the contour is actually part of the element. The outer most contour should be the card. In order to confirm the contour is indeed the card, we can either compare against known card contours, or in a naive way assume the area of the contour is within a threshold. When a contour is determined to be on the outer most level and within the threshold, then the system evaluates the contours on the next level. For each contour on the next level, the system filters out contours not within a threshold. Once all the contours on the card are found. The system uses the k-nearest neighbor algorithm to classify the shape of the contours on the card. It then counts the number of contours on the card that match the same shape and use this as the number of elements.

Finally, the vision system detects the color of the elements. The system takes a naive approach by only looking at the rgb value of the center point of each contour. Based on some thresholds, the system classifies the color.