Homepage - Neuro Fly Virtual Reality

Background:

Being able to monitor a stationary fruit fly as it navigates a virtual environment is of immense importance in the ongoing study of the neural networks responsible for a fruit fly's vision. Discovering and mapping this neural network is an unfinished project 50 years in the making, and this virtual arena will be the first of its kind which has the potential to lead to breakthroughs in this field. In the past, the fruit fly was able to fly freely in a cylinder, which images were being projected onto, however it lacked the ability to analyze the fly’s neural process because of the dynamic nature. In this proposed project however, with the use of genetically engineered fruit flies, the static nature of the fly and the spherical design, the fruit flies will be in a more natural environment and thus reproducing neural behavior similar to their natural environment.

Objective:

For this ground breaking 156B project, the focus is to observe the neural network of a fly in a virtual reality which will be accomplished by building a arena for a statically secured fruit fly, develop a image processing algorithm to measure the amplitude of the wings, and developing a feedback loop based on the amplitude of the fly wing to feed into the virtual reality to determine the trajectory of the fly.

• Build a virtual reality arena for a statically secured fruit fly

  • Using commercially available parts

  • Minimal fabrication

  • Ease of production for other labs

• Develop code to measure and record a fruit fly wing motion

  • Image Subtraction

  • Wing Edge Detection

  • Ellipse Fitting upon Wing Edge

  • Angle Detection of Wing

  • Feedback to Virtual Reality Environment

• Develop a means of projecting onto the spherical display structure

  • Determine a suitable material for the projection display sphere

  • Decide on a suitable means of projecting onto the display

  • Determine geometric conversions within the projector software for projecting onto a sphere

The purpose of this project is to gain an in depth knowledge of the brain of the fruit fly. In particular, the neural circuitry that governs innate visual behaviors in the fly such as navigation, predator escape response, and obstacle avoidance. Our focus is not limited to studying "reflex" behaviors and we're also interested in decision making and how flies choose one stimulus from a barrage sensory stimuli they receive while flying around the natural environment. This has applications to the study of behavioral selection and attention. An example of this application for attention would be when a human being experiences tunnel vision, and a example of behavioral selection would be when a choice must be made.

Proposed Design:

The main components of the proposed design will include the following: two projectors, a acrylic sphere, high speed camera zoom lens, a 3D Micro-manipulator, a IR backlight, and a tether rod. This design was mainly considered for its rigid body setup, stability, and low cost while also allowing fine precision movements for placement of the components. This virtual reality environment will consist of a fruit fly secured to the tether rod, via UV activated adhesive, and inserted into a hollow plastic sphere. Using a high speed camera, the fly wing beats will be measured and used as a feedback signal for the projected image being displayed on the sphere. This feedback system will be implemented using C programming language.

Image Processing:

In simulating a virtual reality the hyper-realism is maintained by having changing environments due to the behavior of the subject undergoing this VR. To mimic these expected changes in the virtual environment, the indications given by the subject are analyzed to identify what they expect to see.This means that the fruit fly will believe it is controlling the environment by traversing through it. Images are taken by a live-feed camera of fruit fly, and because fly behavior is due to changes in their wings, processing images of their wings are of the utmost importance. The process of maintaining an updating virtual reality is through a continuous loop of analyzing fly wing motion adjusting the projections accordingly.

Real-Time Background Subtraction Demo FitEllipse and Background Subtraction Demo

Tethered Flies:

Tethering the Fruit Flies example video

(click the tab for tethering flies, which begins at 1:27, this will give you an overview on how the process is conducted to tether a fly and get it ready for the experiment)

http://www.jove.com/video/1063/a-magnetic-tether-system-to-investigate-visual-olfactory-mediated