Home - Team 04

UCSD MAE Flyception3D

Team members:

Mingcong Tang

Zizhao Gong

Keren Shao

Background:

    In the study of animals’ neural mechanism, people are interested in understanding how animals react towards environmental stimuli. The current method is generally done in the setting in which the animals are tethered in a controlled environment. Such an environment will cause the animals to respond to artificial stimuli  instead of their natural behavior. Even though the limit of the restraint environment has been overcome for the research of some specific animals, like mice and elegants, research on insects still has high demand in constrained areas. In order to prevent such a situation and produce more convincing results, Dr. Dhruv Grover at the Kavli Institute for Brain and Mind proposed a new system called “Flyception”. In this system, Dr. Dhruv introduced the combination of a galvo system and multiple cameras shown in Figure 1 to the study of flies in order to avoid the problem of insects moving out of the observation area. Because the galvo system consisted of two mirrors that can rotate to reflect flies into the view of cameras, the system can collect data while allowing flies to move freely in an elliptical acrylic plate. However, in order to further improve the quality of the system the area should be extended to a rectangular 3D space. If this improvement is successful, the flies will be allowed to fly freely in the arena, and thus produce more comprehensive and versatile data for neural research.

Project Overview:

    The Flyception 3D project focuses on real-time tracking of moving objects, such as flies, in a 20x20x20 cm 3D arena with two low-speed cameras, one high speed camera and a galvo system. Low-speed cameras are used to generate a relatively accurate position estimate of the object. Then a high-speed camera and a galvo mirror system is mounted above the arena to perform real-time tracking of the flies. The workstation will acquire images from the low speed cameras, identify the objects in those images, compute the 3D cartesian coordinates of the object and finally convert it to control signals for the galvo systems. The object will then stay in the center of the high speed camera’s view due to mirror rotations. This approach is successful and we can successfully track moving objects in a constrained 3D space as a result.

Objectives:

• • Mount two low-speed cameras so that they can see the entire arena.

  • Mount one high-speed camera and Galvo system assembly such that they can see the entire arena within the Galvo’s rotation range

  • Construct algorithms for low-speed cameras and Galvo calibration and real-time tracking of small objects.

Photo of final setup:

Figure 1: Overall hardware Setup

Cameras-Galvo system Design

• Two low-speed cameras working at 40 Hz mounted on top two corners of framework symmetrically.  

• One high speed cameras working at 1000Hz mounted on linear stages moving on x and y direction. 

• Galvo mounted on a linear stage moving on z direction. Mirrors can see the arena though a hole on the breadboard.

Figure 2: Final CAD assembly

Coding Design

• The calibration part calibrates the two low-speed cameras. 

• The tracking part is responsible for acquiring streams of images from the low-speed cameras and, using triangulation algorithms to compute the object’s 3D location.

• The conversion part converts the coordinates of the 3D locations and sends the corresponding signals to the galvo and the High-speed cameras

LED Test Design

• Use the LED light as the objective being tracked. 

• Move the LED light manually in expected area and use low-speed cameras to calculate coordinates of the LED.

• Use the view of high-speed camera to check if the galvo is following the light. 

Figure 3: LED Testing Setup and High-speed Camera view

   Performance

• The galvo always follow the LED as long as the height of LED is within our constrained space.

• The deviation from center of high speed cameras is approximately 0.3 millimeter.

• The latency at 33-50 Hz is negligible. However, due to hardware limitation, frequency higher than this range will start to create significant latency. 

Figure 4: Time v.s Dis between Center and LED (pixels)

Table 1: Statistics of LED Testing

Future Improvement

Table 2: Potential Future Improvements 

Impact on Society

The ultimate objective of Flyception 3D  is associated with the neural recording  of body movement of small animals, such as Drosophila. Thus, this project will allow researchers to better understand the neural movement of freely flying flies that evoke behaviors that would be observed in realistic environments.

These brain imaging techniques could further be applied to other studies of neural mechanisms of a vertebrate’s brain with sophisticated behaviors. It will help advance biological studies by improving techniques to conduct research on the neural basis of complex behaviors.