Client

We need two pieces of client software.

First, the one runs in a browser and sends user commands to the server.

Second, the one runs on the robot and controls the robot according to commands sent by the server.

Browser Side:

The mission of controlling a robot must be as fast as possible, so I decided to use WebSockets as our communication protocol. It doesn't include redundant http headers on each request and allows direct notification from the server.

Motion control:

/* It's simple to send motion commands to the robot. For example, we can agree that 0 means to stop and make a stop button on the web page. Whenever the user clicks that button, the browser side script sends 0 to the server and the server will redirect that to the robot. Of course, ideally we'll enable keyboard controls. */

Currently we implemented WASD controls on a desktop browser. The webpage for mobile devices is being developed. The mobile version will have a similar feeling to the desktop version but much more compact. Motion control on a mobile device will take the form of buttons. Using joystick on a phone is also considered, we'll try it if buttons are not flexible enough.

JS Joystick:

http://jeromeetienne.github.io/virtualjoystick.js/examples/basic.html

Video streaming:

/* The problem is live streaming. As far as I know, html5 video element doesn't support live streaming. One possible solution I came up with is to send the frames from the server to the browser over WebSockets. Then, the browser side script needs to decode that video frame and draw it on the screen. Since this is done in JavaScript, we may face severe performance issue in the decoding and rendering stage. More searching should be done before we try this solution. */

Video streaming is implemented by transferring jpeg-encoded frames and rendering them in an <img> element. This approach required more bandwidth than actually video streaming, but it reduced the delay in video encoding.

Beautiful UI:

WIP

Robot side:

This part serves as a general driver of the robot. It receives commands from the server and controls the hardware accordingly. Since we are out of a browser, TCP is chosen as the communication protocol for the best performance.

Motion control:

/* The plan is to make an abstract class that defines possible motions of the robot. This design separates networking and hardware control code. For example, we define an abstract class called DriverBase and define the abstract method Forward() in it. The job of the client side code is to receive commands from the server, once it get the "forward" command, it will call Forward() on a DriverBase object. Hardware developers are in charge of actually define the behavior of Forward(), ideally it makes the robot move forward. */

The robot client is written in C++. We made a header file that defines the possible motions of the robot, such as forwarding and turning. Each action is packaged into a C++ function that is called. People who know hardware is in charge of implementing this header, the rest of the code base acts like a "host". It manages all the working threads, establishes connection to the server and calls the header functions according to the instructions from the server.

Video streaming:

We are using OpenCV to capture frames from the camera, encode them to jpeg images and send to the server. This approach leaves us the space to do more stuff with OpenCV since only one process can capture the camera at a time.