Interactive Border Detection

Introduction

Border detection, a fundamental concept in digital signal processing (DSP), particularly in the realm of image processing, involves identifying significant changes in pixel values within an image. These dramatic changes, often referred to as edges or borders, typically indicate the boundaries of objects or regions within the image. 

For our project, we completed a real time border detector using DSP skills only, without any machine learning techniques. 

Our Environment and Dataset

We want our dataset to have obvious ground and windows information. We collaborated with NYU Self-Drive for their image maze dataset. We want to navigate through maze and identify borders. Below is the maze from Self-Drive.

Algorithms Used

Convolutions

Edges are regions with rapid intensity changes. Calculating the gradient of the image highlights areas of high intensity change. We use a kernel that extracts gradients in pixels, and performs a 2D convolution through the image. We implemented this using python cv2’s canny() function.

Filters

Since we want only large and strong edges, there is a lot of information that is considered noises in an image. Therefore, before edge detection, we need to blur the image to get rid of the noises. We use a gaussian filter. It acts like a low pass filter, removing high frequency noises. We implement it using cv2.GaussianBlur().

HSV and RGB Masking

We also use multiple masking algorithms, which are all 2D convolution based. For example, we use a dark color mask that will mask out all light colors that fall below HSV range.

We also use uniform masking, which calculates the color deviation from the mean of a patch. This can be used to identify windows, since they have large color deviation features. We hard coded the RGB filter. Here is the mean and uniform determination.

Ground Detection:

Here is the code flow chart for ground detection. Since the lighting of the ground will starkly contrast that of the wall in definition, we use two color mask, one light and one dark, to filter the strong border. Then we apply the Hough transform to original image to obtain strong borders. Finally, the intersection between Hough transform and color masking will produce strongest borders, which is ground borders.

Window Detection:

Below is the window detection flow chart. It uses the fact that the feature inside a window has rich color information compared to its neighbor pixels. Therefore, we use a uniform filter to black out the window information. After filling some gaps, we again apply the Hough transform to trace out the border.

Program in use

Here is the GitHub link of the program: https://github.com/hg2622/Interactive-Border-Detection.

The main code is camera.py, which calls multiple imported functions. When the code start to run, a window with four video outputs will be shown. Below are the two screenshots of the stream at two different stamp points.

The top left image is the grayscale output. The top right is the Sobel border detection. The bottom left is a color masked result. The bottom right is the high pass filter result using frequency domain. 

The user needs to actively adjust the parameters of HSV threshold, gaussian filter factor and other parameters until the border detection and color mask video shows clear results, and the Fourier transform video shows less noise and only the strong parts. This is the interactive part. Users have to first find sets of parameters that specify their environment in which they perform border detection. 

As the camera rolls, the four videos will simultaneously stream results, and the ground and window detection results are saved in the backend as shown.

User can choose to run the camera.py and saved the frame for post-processing of window and ground detection, or they can consider the multithread_camera.py for parallel computing, that way, the result of the detection will be saved simultaneously as the stream.

Here is a video that covers the whole project.

Detection Result 

Window Detection Result:

Below shows some selected window detection result from frames saved during streaming.

As we can see, most of the window outlines are placed with green markers. We also did an experiment, out of 20 images, 19 of the windows were fully detected, achieving 95% detection rate. Some ground detections are missing in the above image, but most of the windows are drawn.

Ground Detection Result:

Below is the selected ground detection results.

The ground is placed with green dots. However, we see that some windows are filled with green dots; this is because there are strong, short border features. However, a patch of green dots are very distinguishable from a line of dots.

More results are saved in folders named ‘ground’ and ‘window.’

Applications

The DSP methods, once fit to a certain environment setting, produce very accurate results. Compared to machine learning methods, it saves computing resources since it does not need training. Every algorithm we use is single layer computation. However, the accuracy of DSP might not be as good as border detection by machine learning, and they both need a certain level of parallel computing.

This real time interactive border detector shares similar applications with machine learning methods. After we outline the border of windows and ground, we can do the following things:

Visual Odometry:

By comparing 2 frames, we can tell how much our camera has moved. Using reference borders to accomplish this is much simpler than using the original image.

SLAM:

We can make a simple SLAM with the detection result. We can draw a map that traces the ground and locates features windows.