Stereo vision with webcams in OpenCV

I've been learning how to use the OpenCV libraries in C++ to perform stereo vision using multiple webcams. OpenCV is great, a life saver really, but the functions available in OpenCV require quite a lot of complicated coding on the part of someone who just wants to get a simple job done.

My goal in offering this Web resource is to take all of the lessons I've learned, and the materials and code that I've had to develop, and make it easier for others to go where I've gone, much as the people who have so far contributed to OpenCV have done for me.

Some of my code has been typed in almost verbatim from the very helpful textbook:

Bradski G and Kaehler A (2008) Learning OpenCV: Computer Vision with the OpenCV Library. O'Reilly Media. (link)

Their colophon at the beginning of the book permits free use of their code, but they encourage people to cite them. I am gratefully citing them here. Please buy a copy of their book, read it, and type in some of their code examples. This will help you immeasurably in your efforts to learn OpenCV.

For now, I'm going to assume that you know how to program in C/C++, that you have an appropriate compiler and integrated development environment for doing so, and that you know how to download and install the most recent distribution of OpenCV. Let me just say that you have to put the OpenCV libraries into your compiler paths, so the complier knows where to look for them. There are instructions here:

http://opencv.willowgarage.com/wiki/InstallGuide

Also for now, I'm going to say that this site is 'under construction'. As time passes, and I get better at this, I'll be adding sections, code, and maybe other resources like images and downloadable stuff. I'm also going to make myself available for questions. Please try your best to find the answers to these questions yourselves, before asking me. That's always good etiquette, in any area of study.

Some advice. Be awesome, and be fearless. All you have to do is learn, and practice. There is always an answer to your question...don't give up.

Jeff Colombe, Ph.D.

Unit 1: Detecting the number of cameras that are available

OpenCV, in past incarnations, used to allow you to directly find out, with a single function call, how many camera devices were available. No longer. Here is a routine that will allow you to retrieve this information.

If you don't detect how many cameras there are, and try to read from them, your program will exit without telling you why.

And, importantly, stereo vision is not just for two cameras anymore. Although OpenCV does not support more than two cameras for stereo calculations, the field is moving in that direction. So you might want to know how many cameras you can read from, not just one or two... Some applications today, like light-field imaging or 'lumigraph' imaging, may involve dozens or hundreds of cameras. So there.

I'm including all the stuff you'll need to make this work. (Why don't people do this more often?)

// Necessary headers (some may not be needed, but it doesn't hurt):

#include "stdafx.h"

#include "cv.h"

#include "highgui.h"

#include "cvaux.h"

// Function declaration:

int DetectNumberOfCameras( void );

// Function definition:

int DetectNumberOfCameras( void ) {

int num = 0;

int finished = 0;

while( !finished ) {

CvCapture* capture = cvCreateCameraCapture( num );

if( !capture )

finished = 1;

else {

num++;

cvReleaseCapture( &capture );

}

}

return num;

}

// Function call from your program:

int numcams = DetectNumberOfCameras( );

And there you are. Anything else you do downstream of this call will know how many cameras to evaluate or pull images from.

IMPORTANT: I've done experiments plugging in two or three webcams in various configurations using a powered USB 2.0 hub, and/or the two USB 2.0 'stacks' on my laptop. The two USB plugs in the same 'stack' are almost certainly on the same USB 'root' into the mainboard...the two 'stacks' are probably on different USB 'roots'. My webcam plugs are too thick to directly plug two of them into the same stack, thus the use of the hub. Two cameras plugged in anywhere work just fine for real-time (~32 fps) image acquisition. Three cameras on the same hub are all recognized by OpenCV, but I haven't been able to create camera capture objects for all three simultaneously...the third one won't work (yet). When I plug two cameras into the hub, and the third camera into the other 'stack', I can create camera capture objects for all three simultaneously. The unfortunate thing is that frame rates drop to either ~8 fps, or sometimes ~16 fps. I still don't know why.

USB 2.0 has a reported maximum data rate of 480 Mbits/sec. Two cameras at 640x480 pixels x 3 color channels (R,G,B), at 32 fps = 472 Mbits/sec. The good news is that USB 3.0 is coming very soon, with much higher advertised maximum data rates.

However, USB data rates on any given machine may not always be as available as one would like, perhaps because of events going on within the motherboard and OS, currently a bit mysterious to me.

IMPORTANT: USB connectors are not ideal under conditions of disturbance or vibration. They are designed for desktop use. If you bonk into the table, you might lose one or more cameras, and have to restart your program.

If needed, you can call this function periodically, when you don't need to gather images for your main analysis, to see if any cameras have disappeared or been added to the mix. This might be due to the vibration issues mentioned above, or because someone has removed or plugged in cameras on purpose.

Unit 2: Making a chessboard for camera calibration

Summary: Download the attached JPEG below, print it out on a B/W printer, and mount it on 9"x12" foam core from your local office supply store, or an 8"x10" piece of glass from a cheap picture frame, using spray glue.

Camera calibration is needed because of the distortions produced by poor lenses and misaligned imaging chips in cameras (some are affixed to the backplane using a drop of glue!). I love webcams because they are cheap, but they are cheap for a reason. You can correct for optical errors by waving a chessboard pattern of black and white squares in front of a camera, and have OpenCV analyze the positions of interior corners at a number of different spatial orientations of the chessboard (as you move it around), to determine an appropriate image warping algorithm for optical correction. The algorithm knows that the chessboard pattern is planar, and that assumption allows correction of optical errors.

I looked all over the Web, but found no chessboard pattern to download. So I made one in Adobe Illustrator, and I've saved it here as a JPEG file (see "Attachments" below). Feel free to download and print this pattern. For OpenCV's algorithms, this chessboard pattern is 6x8 intersections in dimension (these are internal intersections, not the number of squares).

After printing this on office paper, you will need to mount it securely on a very flat surface that will stay flat over the course of days, weeks, months, however long you'll need it. You can affix it to a flat surface using spray glue. Spray glue is needed, because anything that you would pour or rub onto a surface will be much less evenly distributed, and might cause wrinkles or other warping. I've used 3M Super 77 Multipurpose Adhesive, which you can get at a decent hardware store for about $8. Spray in a well-ventilated area, ideally outdoors with no wind. Spray it on the substrate, not the printout. Carefully align the printout on the substrate so it fits nicely. Pat it down, spreading from the center outward, very gently, making sure that all edges of the paper press down flat with no buckling or wrinkles. Take care not to get gooey glue on your fingers, lest you put glue onto the surface of the printout. Once positioned well and flattened, you might want to put the whole thing down on a reliable and clean flat surface, then stack some books on top, making sure that the books don't catch the glue on the sides. This will make the adhesive affix and dry for 10-20 mins without deviating from being very flat.

The surface that I mounted this chessboard pattern on was a 9"x12" piece of white foam core board from Staples ($4). Be careful! Look at the foam core to see if it is warped. Turn it so you look across its surface. If it bows, don't use it. This foam core had a peel-off adhesive side, which was horrible. Within 30 minutes, the paper was peeling off. Use spray glue instead. Worst case, you might need to buy an 8"x10" picture frame with glass, remove the glass, trim your printout, and spray-glue it to the glass. Discard the picture frame. Glass will be very flat. Not so difficult, and not so expensive. Just make sure that you take care of any sharp edges on the glass, and don't hurt your fingers while waving the thing around in front of your webcams.

Algorithms for evaluating the 'waving chessboard' are to follow...