Project: Tracking Fruitflies

Get this HDF5 project file related to fruitfly video from the wonderful work of Kristin Branson, Janelia Research Campus. Extract the files from the tar file. Next import the flyBowlMovie200.h5 file into ImageJ-FIJI's ilastik plugin. That is, invoke ImageJ-Fiji and go to Plugin > ilastik > Import HDF5 > Select flyBowlMovie200.h5 file. Accept the default axis order, and click OK.

Next, save the video frames in ImageJ-Fiji as image sequences, i.e., 1.jpg, 2.jpg, etc. Suppose the directory containing the frames is named as flyseq.

Tracking Strategy

In order to track objects you must be able to first detect them in each frame. By detection we mean localization. In each frame we shall first locate the body-centers of the fruitflies. In the second stage we shall need to associate the detected locations in each frame with those in the subsequent one.

Hence in summary, we can view tracking as two stage process: in the first one we are detecting the locations of each object. In the second stage the association problem amounts to stitching the detected locations together along time to compute the trajectories.

Detection: Background Subtraction

We are going to solve the detection problem by zeroing on the region of interest (ROI) first. Once the ROI is estimated, anything that moves in that ROI becomes easier to detect. We read the first frame of the video as follows.

clc, clear all, close all;

folderpath = 'flyseq';

im = imread(fullfile(folderpath, [num2str(1), '.jpg']));

figure(1), imagesc(im), colormap(gray), axis image, axis on;

Next we smooth the filter with heavy Gaussian blur to smooth out the small bodies of the fruitflies.

sgm = 25;

h = fspecial('gaussian', [sgm, sgm], sgm);

im_filt = imfilter(im, h);

figure(2), imagesc(im_filt), colormap(gray), axis image, axis on;

This is followed by a thresholding operation that essentially gives us the ROI.

mask_im = im_filt > 125;

figure(3), imagesc(mask_im), colormap(gray), axis image, axis on;

Let us save this ROI mask as:

save('region_mask_data.mat', 'mask_im');

Next, we load the ROI mask and shrink the disk shape inward by 10 pixels (to avoid false detections along the boundary regions).

clc, clear all, close all;

load('region_mask_data.mat', 'mask_im');

se = strel('disk',10);

mask_im = imerode(mask_im,se);

Next we invert the image as follows.

inverted_im = 255-im;

This is followed by heavy blurring -

sigma_val = 15;

blur_kernel = fspecial('gaussian', [sigma_val, sigma_val], sigma_val);

im_filtered = imfilter(inverted_im, h);

In the next step we apply the useful MATLAB routine FastPeakFind() of Adi Natan to detect the local peaks in 2D image signal.

maskedimage = im_filtered.*uint8(mask_im); % we keep only the ROI part

p=FastPeakFind(maskedimage, 80); % the value 80 is a local peak threshold

% the rest is visualization of the output

figure(5), imagesc(maskedimage), colormap(gray), axis image, axis on; hold on;

plot(p(1:2:end),p(2:2:end),'r+', 'MarkerSize', 12);

So, the detection of foreground essentially becomes a background estimation task.

It is time now to write the full code for detecting the fruitflies in each frame of the entire video in a loop.

for now_n = 1 : 200

    im = imread(fullfile(folderpath, [num2str(now_n), '.jpg']));

    im_filtered = imfilter(255-im, h);

    p=FastPeakFind(im_filtered.*uint8(mask_im), 80);

    figure(10), imagesc(im), colormap(gray), axis image, axis on; hold on;

    plot(p(1:2:end),p(2:2:end),'r+');

end

What is wrong in this video result? Can somebody identify?

Hint. Have I been able to detect all the flies correctly in each frame? If not, what is the probable way to improve this result? Please pay particular attention to the threshold value 80 in the following peak detection call:

p=FastPeakFind(im_filtered.*uint8(mask_im), 80);

If you try two particular values, namely, 60 and 100, you will see the effect of false detections (aka. false alarms) and missed detections respectively.

Tracking: Data Association

This is given as assignment. Once you have the detection can you assign each detection with its nearest neighbor in the immediate next frame? What happens when one or few of the fruitflies go(es) undetected? In such case can you come up a good strategy for handling missed detection?

Another area of consideration would be to lower the threshold value to a reasonably low value such that you start receiving spurious detection. What is your thought in such case? Can you come up with a strategy to handle the false positive cases?

Final Assignment

Convert the entire project into OpenCV-Python and reproduce the results.

Page updated

Google Sites

Report abuse