11-783 Rich Interaction in Virtual Worlds

This course can also be taken as ECE 18-980, please contact Joy (see below) for details. The course is held in a virtual world, shared between CMU's Pittsburgh and SV campuses. 11-783 is a 6-unit lab course, typically held in the spring semester. The lab is often held together with 11-753 Advanced Speech Lab. Students can register for one lab, or both labs at the same time. This lab also shares infrastructure with 15-291 "Computational Intelligence Lab", see http://www.speech.cs.cmu.edu/coinlab/. Time and place will be agreed upon between the instructor and the students.


Massively Multi-player Online Role-Playing Games have evolved into Virtual Worlds (VWs), and are creating ever richer environments for experimentation on all aspects of human to human, or human to machine communication, as well as for information discovery and access. So far, interaction has been constrained by the limited capabilities of keyboards, joysticks, or computer mice. This creates an exciting opportunity for explorative research on speech input and output, speech-to-speech translation, or any aspect of language technology. Of particular interest will be a combination with other novel "real world" (RW) input, or output devices, such as mobile phones or portable games consoles, because they can be used to control the VW, or make it accessible everywhere in RW. Language technologies in particular profit from "context awareness", because domain adaptation can be performed. For scientific experimentation in that area, Virtual Worlds offer the opportunity to concentrate on algorithms, because context sensors can be written with a few lines of code, without the need for extra hardware sensors. Algorithms can also run "continuously", without the need for specific data collection times or places, because the VW is "always on". 

In this lab, we will enhance existing clients to virtual worlds so that they can connect to various speech and language related research systems developed at LTI and CMU's Silicon Valley campus. The lab will be held jointly at the CMU's Pittsburgh and Silicon Valley Campuses. We will "eat our own dog food", so the goal will be to hold the last session entirely in a virtual class room, which will by that time include speech control of virtual equipment, speech-to-speech translation, and some devices that can be controlled using non-PC type equipment, like mobile phones. Eventually, this virtual room or the know-how gathered in creating it will also benefit other educational activities. 

Requirements & Milestones

Literature survey and proposal

Each student must pick a topic and write a survey of the state of the art by the end of the second week. The student should identify and read relevant papers in the area he choses to work on. The survey should lead up to the planned work, which should include a statement of the objectives, the description of the approach, planned evaluation, the estimated significance of the research contribution and a set of milestones with planned deadlines. 


Proposed method(s) and algorithm(s) must be implemented and a comparison between methods must be achieved, if applicable. The lab requires exercising and demonstrating programming skills in conjunction with analytic skills. 

Preliminary results

Six weeks into the class, a detailed work plan for the rest of the term is expected from every student, submitted as part of a two-page progress summary, which should include an informative and concise description of the selected research problem, an analysis of suitable evaluation criteria end algorithms, and an outline of the remaining work. Preliminary results are encouraged, as is the demonstration of a functional prototype at this time. 

Documentation and presentation

  • write and submit a final report on your system and experiments;
  • submit the final version of your documented code with a short user’s manual and system documentation (so that the code can be shared and used by other students); 
  • present your work to the instructor (and possibly other students); arrange your time with the instructor through email. 

Grading Criteria

  • The originality of the idea, the rigor of the evaluation, and the significance of the findings to those reported by other students or in the literature. 
  • The quality and quantity of the programming part, and appropriateness and clarity of the documentation. 
  • How well the approach/ results are presented, both in written and oral form.