Kinect

consOverview

YOU NEED TO USE A USB CABLE TO UPLOAD YOUR PROGRAM – YOU CANNOT USE XBEE TO UPLOAD PROGRAMS TO THE BOARD. YOU ALSO NEED TO DISCONNECT THE XBEE FROM THE ARDUINO BOARD WHILE YOU UPLOAD.

In this tutorial, you will learn how to control your Arduino through a Microsoft Kinect. The Kinect communicates over XBee with a Processing sketch running on your laptop. The tutorials will show you how to control two servos via the relative the hand movement movement. This tutorial assumes some basic knowledge of programming and using the Arduino sketch. The following software must be installed prior to trying the tutorials.Tested on Mac OSX.

Parts

Arduino microcontroller and carrier board

LiPo battery

Kinect controller (with external power supply)

XBee pair with USB dongle(available: ask your instructor)

Laptop

Software

Processing

Download at http://processing.org/download/. We will be using Processing as the link between the Kinect and the Arduino. Processing also includes graphical tools. Processing is a simple way to create drawings, animations and interactive graphics. Visualization of data coming from an Arduino is fairly straightforward. Code below tested with Processing 1.2.1.

Libraries

A simple Processing Wrapper for the OpenNI/Kinect library can be found here: Simple-openni. Follow the installation instructions for your operating system.

Hardware

Laptop, Arduino board, 2 continuous rotation servos. Instead of USB communication, the computer can talk to the Arduino using XBee radios. Doing so requires no additional software, however you must be sure the baud rate of the radios match in the Arduino and Processing software. 2 XBee radios and an USB dongle for wireless control of the Arduino (limited amount) are available in the supply locker.

Now that we have all of the hardware and software required to make the Kinect talk to the Arduino, let’s get started on a simple program. We will drive the mini-me using our hand movements. The processing code follows the SimpleOpenNi_NITE_Slider2d very closely and maps the tiles on the screen to the movement of your robot.

Upload Arduino code

Make sure your Arduino is plugged in via the USB cord and open up the Arduino software on your computer. The following code is the same Arduino code for the iPhone. Copy the following code into the empty Arduino window:

//----------------------------start Arduino code--------------------------------

#include <Servo.h> // include servo headers

// this would be whatever pin you want to use

#define SERVOR 8

#define SERVOL 9

// global variables

Servo servoL; // initialize left servo

Servo servoR; // init right servo

int message = 0; // This will hold one byte of the serial message

// played around with values that sets the servos to neutral position

// these values need to be set for each servo!!!

const int servoneutralLeft = 1515;

const int servoneutralRight = 1520;

// set neutral range for servos

const int minneutral = 1400;

const int maxneutral = 1600;

//--- Function: Setup ()

void setup()

pinMode (SERVOL, OUTPUT);

pinMode (SERVOR, OUTPUT);

servoL.attach(SERVOL);

servoR.attach(SERVOR);

servoL.writeMicroseconds(servoneutralLeft); // set servo to mid-point

servoR.writeMicroseconds(servoneutralRight); // set servo to mid-point

Serial.begin(9600); //set serial to 9600 baud rate

//--- Function: loop ()

void loop()

// Check if there is a new message

if (Serial.available() > 0)

message = Serial.read(); // Put the serial input into the message

int val=message; // val to match pwm delay in ms

int tempval=0; // temp storage

// we can send values from 0 to 255 to the arduino.

// both fadders are set up to go from 0 to 1.

// left servo: 0-127, right servo 128-255. should be enough resolution

int minpulse = 127*8/2; // max storage is 0-255.

// Begin LEFT servo code

if (val <= 127)

// scale everything from 1000 to 2000

tempval = val*8 + servoneutralLeft - minpulse;

if (tempval > minneutral && tempval < maxneutral)

    // Creates dead zone at midpoint of the

    servoL.writeMicroseconds(servoneutralLeft); // fader range (neutral) and trims input to neutral

else

    servoL.writeMicroseconds(tempval);

// End LEFT servo code

           // Begin RIGHT servo code

if (val > 128)

// scale everything from 1000 to 2000

tempval = val*8 + servoneutralRight - minpulse - 128*8;

if (tempval > minneutral && tempval < maxneutral)

    servoR.writeMicroseconds(servoneutralRight);

    // fader range (neutral) and trims input to neutral value

else

    servoR.writeMicroseconds(tempval);

} // End RIGHT servo code

//----------------------------end Arduino code--------------------------------

Run Processing code

Copy the code from below and paste it into an empty Processing window. Run the Processing sketch by hitting the play button, it looks very similar to the Compile button in Arduino. Whenever you want to control your Arduino with your Kinect, Processing has to run on your laptop. Wave your hand to "activate" until tile turns green. The relative hand movement will select tiles.

//----------------------------------start processing code------------------------------------

import SimpleOpenNI.*;

SimpleOpenNI          context;

import processing.serial.*;    //  Load serial library

Serial arduinoPort;        //  Set arduinoPort as serial connection

int val1 = 0;                        //  servo 1 value

int val2 = 0;                        //  servo 2 value

// NITE

XnVSessionManager     sessionManager;

XnVSelectableSlider2D trackPad;

const int gridX = 3;

const int gridY = 3;

Trackpad   trackPadViz;

//--- Function:

void setup()

  context = new SimpleOpenNI(this, SimpleOpenNI.RUN_MODE_MULTI_THREADED);

  context.setMirror(true);   // mirror is by default enabled

  context.enableDepth();    // enable depthMap generation

  context.enableGesture();   // enable the hands + gesture

  context.enableHands();

  // setup NITE

  sessionManager = context.createSessionManager("Click,Wave", "RaiseHand");

  trackPad = new XnVSelectableSlider2D(gridX, gridY);

  sessionManager.AddListener(trackPad);

  trackPad.RegisterItemHover(this);

  trackPad.RegisterValueChange(this);

  trackPad.RegisterItemSelect(this);

  trackPad.RegisterPrimaryPointCreate(this);

  trackPad.RegisterPrimaryPointDestroy(this);

  // create gui viz

  trackPadViz = new Trackpad(new PVector(context.depthWidth()/2, context.depthHeight()/2, 0),

  gridX, gridY, 100, 100, 15);

  size(context.depthWidth(), context.depthHeight());

  smooth();

  // info text

  println("-------------------------------");

  println("1. Wave till the tiles get green");

  println("2. The relative hand movement will select the tiles");

  println("-------------------------------");

  println("-------------------------------");

 // println(Serial.list());

  arduinoPort = new Serial(this, Serial.list()[0], 9600);    // Set arduino to 9600 baud

  println("-------------------------------");

  //set values to neutral for now

  val1 = (int) ( 127.0/2);  // left servo

  val2 = (int) ( 128 + 127.0/2);  //   RIGHT servi

//--- Function:

void draw()

  context.update(); // update the cam

  context.update(sessionManager);   // update nite

  image(context.depthImage(), 0, 0);  // draw depthImageMap

  trackPadViz.draw();

   arduinoPort.write(val1);     // Sends val1 to Arduino, left servo

   arduinoPort.write(val2);     // Sends val2 to Arduino, right servo

 //  println("Left servo value: " + val1 + " Right servo value: " + val2);

//--- Function:

void keyPressed()

  switch(key)

  case 'e':

    // end sessions

    sessionManager.EndSession();

    println("end session");

    break;

///--- session callbacks ----///

void onStartSession(PVector pos)

  println("onStartSession: " + pos);

//--- Function:

void onEndSession()

  println("onEndSession: ");

//--- Function:

void onFocusSession(String strFocus, PVector pos, float progress)

  println("onFocusSession: focus=" + strFocus + ",pos=" + pos + ",progress=" + progress);

///--- XnVSelectableSlider2D callbacks ----///

//--- Function:

void onItemHover(int nXIndex, int nYIndex)

//  println("onItemHover: nXIndex=" + nXIndex +" nYIndex=" + nYIndex);

  // we can send values from 0 to 255 to the arduino

  // left servo: 0-127, right servo 128-255.

  if ((nXIndex == 0) && (nYIndex == 0) ) // left back

    val1 = (int) ( 0);  // left servo

    val2 = (int) ( 192);  //   RIGHT servo

  else if ((nXIndex == 1) && (nYIndex == 0)) // full back

    val1 = (int) ( 0);  // left servo

    val2 = (int) (255);  //   RIGHT servo

  else if ((nXIndex == 2) && (nYIndex == 0))   // right back

    val1 = (int) ( 63);  // left servo

    val2 = (int) ( 255);  //   RIGHT servo

  else if ((nXIndex == 0) && (nYIndex == 1) ) // left

    val1 = (int) ( 63);  // left servo

    val2 = (int) ( 129);  //   RIGHT servo

  else if ((nXIndex == 1) && (nYIndex == 1))  // neutral

    val1 = (int) ( 127.0/2);  // left servo

    val2 = (int) ( 128 + 127.0/2);  //   RIGHT servo

  else if ((nXIndex == 2) && (nYIndex == 1))  // right

        val1 = (int) ( 127.0);  // left servo

    val2 = (int) ( 128 + 127.0/2);  //   RIGHT servo

  else if ((nXIndex == 0) && (nYIndex == 2)) // left forward

    val1 = (int) ( 63);  // left servo

    val2 = (int) ( 129);  //   RIGHT servo

  else if ((nXIndex == 1) && (nYIndex == 2) ) // full forward

    val1 = (int) (127);  // left servo

    val2 = (int) ( 129);  //   RIGHT servo

  else if ((nXIndex == 2) && (nYIndex == 2))   // right forward

    val1 = (int) ( 127.0);  // left servo

    val2 = (int) ( 192);  //   RIGHT servo

  else

    val1 = (int) ( 127.0/2);  // left servo

    val2 = (int) ( 128 + 127.0/2);  //   RIGHT servo

  trackPadViz.update(nXIndex, nYIndex);

//--- Function:

void onValueChange(float fXValue, float fYValue)

  // println("onValueChange: fXValue=" + fXValue +" fYValue=" + fYValue);

//--- Function:

void onItemSelect(int nXIndex, int nYIndex, int eDir)

  println("onItemSelect: nXIndex=" + nXIndex + " nYIndex=" + nYIndex + " eDir=" + eDir);

  trackPadViz.push(nXIndex, nYIndex, eDir);

//--- Function:

void onPrimaryPointCreate(XnVHandPointContext pContext, XnPoint3D ptFocus)

  println("onPrimaryPointCreate");

  trackPadViz.enable();

//--- Function:

void onPrimaryPointDestroy(int nID)

  println("onPrimaryPointDestroy");

  trackPadViz.disable();

///--- Trackpad class ----///

class Trackpad

  int     xRes;

  int     yRes;

  int     width;

  int     height;

  boolean active;

  PVector center;

  PVector offset;

  int      space;

  int      focusX;

  int      focusY;

  int      selX;

  int      selY;

  int      dir;

  Trackpad(PVector center, int xRes, int yRes, int width, int height, int space)

    this.xRes     = xRes;

    this.yRes     = yRes;

    this.width    = width;

    this.height   = height;

    active        = false;

    this.center = center.get();

    offset = new PVector();

    offset.set(-(float)(xRes * width + (xRes -1) * space) * .5f,

    -(float)(yRes * height + (yRes -1) * space) * .5f,

    0.0f);

    offset.add(this.center);

    this.space = space;

//--- Function:

  void enable()

    active = true;

    focusX = -1;

    focusY = -1;

    selX = -1;

    selY = -1;

//--- Function:

  void update(int indexX, int indexY)

    focusX = indexX;

    focusY = (yRes-1) - indexY;

//--- Function:

  void push(int indexX, int indexY, int dir)

    selX = indexX;

    selY =  (yRes-1) - indexY;

    this.dir = dir;

  void disable()

    active = false;

//--- Function:

  void draw()

    pushStyle();

    pushMatrix();

    translate(offset.x, offset.y);

    for (int y=0;y < yRes;y++)

      for (int x=0;x < xRes;x++)

        //          if(active && (selX == x) && (selY == y))

        //          { // selected object

        //            fill(0,0,255,190);

        //            strokeWeight(3);

        //            stroke(100,200,100,220);

        //          }

        //          else

        if (active && (focusX == x) && (focusY == y))

        { // focus object

          fill(100, 255, 100, 220);

          strokeWeight(3);

          stroke(100, 200, 100, 220);

        else if (active)

        {  // normal

          strokeWeight(3);

          stroke(100, 200, 100, 190);

          noFill();

        else

          strokeWeight(2);

          stroke(200, 100, 100, 60);

          noFill();

        rect(x * (width + space), y * (width + space), width, height);

    popMatrix();

    popStyle();

// end class trackpad

Other examples: