Teaser Figure

Dataset and Model training

Out of the entire Woodiccv dataset, we only used 250 photos because of the high processing expense. The dataset contains total of 8234 images. The 250 images dataset was divided into a holdout, testing, and train set at random. 150 photos were used for training, 80 images were used to tune the hyper-parameter (learning rate), and 20 images served as a holdout set for various metric calculations. We employed the random cropping data augmentation strategy to avoid the model from overfitting.

A pretrained unet model is initially downloaded from the torch hub in order to train the model. Researchers can obtain a variety of pretrained models simply at Torch Hub, a model library. Since it is simple to use through python api, DL researchers can immediately download and test the models to suit their needs. The downloaded model was then changed to ensure that its input and output shapes were appropriate for our use. After creating the training loop, the model was trained using our dataset. Binary cross entropy was employed as the loss function. We used a simple data augmentation strategy during the model training procedure to prevent overfitting i.e.  Our training loop randomly selects a portion of the image for each epoch and uses that image for training. Because of the random cropping strategy used throughout the training procedure, our model's input shape was (3x640x640) when working on unrectified images. We tried to do 1024x1024 random cropping for unrectified images but the height of the image was less than 1024. Hence we chose to use 640x640 size. For testing and holding datasets, no augmentation was used. To update the parameters of the model, we utilized the RMSProp optimizer with a learning rate of 1e-05. We utilized RMSProp optimizer over Adams optimizer because it has been found that training model with RMSProp optimizer makes convergence faster. After each epoch, the dice coefficient for the test set was determined as we trained the models for significant number of epochs. In order to calculate metrics for the holdout set, the best dice score and the model based on it were both recorded and saved using pytorch function.

Training loss and validation dice score during training for unrectified images

Training loss and validation dice score during training for rectified images

Video 1 : What is UNET architecture? Video 2: How to Implement UNET?

The UNet architecture is symmetric and consists of two major parts — the left part is called the contracting path, which is constituted by the general convolutional process; the right part is an expansive path, which is constituted by transposed 2d convolutional layers. The above two videos provide a general description of UNET model and how to implement any UNET model.

As is with the implementation of any segmentation methodology the two most common challenges are:
High GPU memory requirement - Deep learning models are memory intensive. A single pass consumes a lot of memory if the model is big and the image has a higher resolution. Techniques such as Random cropping to some smaller size and choosing less batch size are often used to counter this problem.

False positive detection - A false positive detection is when the model falsely identifies a pixel value as a true pixel. Figure 14 (a), (b), and (c) depicts this issue where a false positive identification is done on the building.

FALSE negative detection - A false negative detection is when the model fails to identify a true pixel in reference to ground truth. Figure 15 (a), (b), and(c) depicts this issue where a false negative identification is done on the truck.

Perspective Transformation

Transformation of fisheye to rectangular images is done in two steps

Step 1: Conversion of fisheye to the spherical coordinate system

P’(x’,y’) = P(cos φ sin θ, cos φ cos θ, sin φ)

where,

P ′ − Fisheye coordinates

P − Spherical coordinates

φ = (r∗aperture)/2

θ = atan2(y′, x′)

Step 2: Converting spherical coordinates to rectangular coordinates

longitude = asin(z/R)

latitude = atan2(y,x)

This presented many challenges, some of whom are listed below

• Dimension of rectified images - As is evident from the formulae, while unwrapping a fisheye image the information available is depended upon the intrinsic properties of the lens i.e Radius of the fisheye lens (R), vertical field of view (vfov) and horizontal field of view (hfov). Also while mapping the output image is of the size (R,2 π R) ideally.

Simplifying assumptions were made in order to limit the scope of the program development, in our case we developed a program for isometric aperture lenses (those having equal vfov and hfov), also the fov of the lens is assumed to be 180deg. The programs work well for other fov as well but need to be modified for center point definition. Upscaling needs to be done to deliver an output image of the same size as the input. Figure 8 depicts an unrectified and unscaled output from the transformation.

• Aperture identification - Figure 10 depicts the transformation with the wrong aperture value.

Our [8]dataset included images taken from 4 cameras namely FV (Front View),RV (Rear View),MVL (Mirror View Left) and MVR (Mirror View Right). Out of these four, only images taken from FV cames were chosen as it will be the primary camera for oncoming vehicle detection. By specifying camera intrinsic properties for all the images remained constant, thus aperture is determined for one image and hardcoded into the program.

• Forward mapping information loss - Forward mapping is when we loop over all the points of the input image and find the corresponding point in the output image. The values of the corresponding points can then be cross-matched but this method leaves gaps in the output image as evident from Figure 9.

This can be overcome by inverse mapping i.e looping over points of the output image to find corresponding points in the input image. Due to time constraints, we were unable to develop a working algorithm using inverse mapping of points. Such an algorithm would solve loss issue as is shown by the work of [6](Scott, Fish eye lens dewarping and Panorama Stiching) , [5](Bourke, Converting a fisheye image into a panoramic, spherical or perspective projection) and [7](Ha, Github/Py-Fisheye-dewarp: Fisheye image dewarp / equirectangular rotation)

Additionally, it was observed that these losses are concentrated more toward the periphery of the output image, also since our ground truth data contains a mask these small losses toward the periphery can be glossed over by using a mask.

• Mirror Inversion - Output images from the transformation are flipped (mirror image) along the vertical axis.

This can be easily remedied by flipping the output image matrix.

Experiments and Results

The performance of both approaches was compared and measured using predefined metrics such as Dice score, IOU score, Accuracy, Precision, Recall, and F1 score.

Dice Score - The Dice score is the metric used to assess model performance. The score ranges from 0 to 1. A score of 1 corresponds to a pixel perfect match between the deep learning model output and ground truth annotation. The formula for the Dice score is given below.

Here, TP is true positive, FP is false positive, TN is true negative and FN is false negative.

IOU Score - IOU score is also the metric commonly used in object detection. It is the ratio of area of overlap and area of union.

Accuracy, Precision, Recall, and F1 score are the most commonly used metrics for classification tasks. These scores represent metrics for pixel level classification. Out of all these, the recall score is the most significant because it represents how many percentages of pixels of vehicles are accurately being predicted.

Qualitative Results

Conclusion

The deep learning model performs better when trained and tested on the unrectified fish eye images. Although the accuracy metrics seems to be marginally higher, this doesn't reflect the actual model performance because there are fewer vehicle pixels than background pixels.

Moving forward a number of steps can be taken to further improve the detection rate and metric scores namely

• Inverse transformation mapping - With inverse mapping we would get better transformation images as there will be no forward mapping information loss. This method would also avoid upscaling of output images and the size of the output image will be similar to the input image.

• Training with a bigger dataset - Training with bigger data would allow for generating a better model. Also with an improved GPU, we can feed a bigger randomly sampled image into the UNET model.

• Experimenting with other models such as deeplabv3, lenet etc to see whether these models perform better than general UNet model.

• Making rectified output image the same size as the unrectified ones.