Parallel Parking Detection using Computer Vision and Convolutional Neural Nets

Project Goals:

The main objective of the proposed system is to find parallel parking spaces along with regular parking spots which is a major challenge that most people and autonomous vehicles are facing today.

Major Challenges in real-world

The more people on the road increases and drivers become aggressive. Some of the major problems that drivers face today in real-world are :

1. Circling the parking lot for parking the vehicle: Research shows that 30% of traffic is made up of people looking for a place to park and results in congestion and traffic accidents.

2. Lack of timely information: Drivers don't have real-time parking information and are confused about where to go and park.

3. Greater demand for Parking: During the holiday season, people tend to move to other cities or go shopping and lack of information on parking spaces is quite a big problem nowadays.

Source: Greatest Challenges Parking Facilities Managers Face by All traffic Solutions.

Existing Parking Systems

1. A counter-based system is a system that relies on sensors at the entrance and exit of the parking lot. These systems only provide information on the total number of vacant spaces, which doesn't guide the driver to the exact location of the parking space and such systems cannot be applied to on-street parking bays and residential parking spaces.

2. Wired sensor-based and wireless magnetic-sensor-based systems rely on ultrasonic, infrared light, or wireless magnetic-based sensors installed on each parking space.

The main disadvantage of both the systems is it requires the installation of costly sensors and also require transceivers.

References: http://ceur-ws.org/Vol-2087/paper5.pdf

Literature Survey

1. Funck et al use an algorithm to compare the reference image and input datasets to calculate the vehicle to parking space pixel area using PCA (principal component analysis).

Funck, S., Mohler, N., and Oertel, W. (2004). Determining car-park occupancy from single images. In IEEE Intelligent Vehicles Symposium, 2004, pages 325–328.

2. Tsai et al train a Bayesian classifier to verify the detections of vehicles using corners, edges, and wavelet features.

Tsai, L. W., Hsieh, J. W., and Fan, K. C. (2007). Vehicle detection using normalized color and edge map. IEEE Transactions on Image Processing, 16(3):850–864.

3. True adopts a combination of vehicle feature point detection and color histogram classification.

True, N. (2007). Vacant parking space detection in static images. University of California, San Diego, 17

4. The COINS (Car park Occupancy Information Systems) integrates advanced image processing techniques including seeding, boundary search, object detection, and edge detection together for reliable parking space detection.

Bong, D., Ting, K., and Lai, K. (2008). Integrated approach in the design of car park occupancy information system (coins). IAENG International Journal of Computer Science, 35(1):7–14.

5. Amato et al. (2016) develop a decentralized solution for visual parking space occupancy detection using deep CNN and smart cameras.

Amato, G., Carrara, F., Falchi, F., Gennaro, C., and Vairo, C. (2016). Car parking occupancy detection using smart camera networks and deep learning. In 2016 IEEE Symposium on Computers and Communication (ISCC), pages 1212–1217

Presentation 1

Phase 2 is about the actual implementation of the parking detection model.

The objective of Phase 2 is to detect bounding boxes and masks of the parking lot images using object detection and Object segmentation models in Deep Learning.

Let us understand both terms in detail.

Object Detection and Object Segmentation

1. Object Detection

The basic understanding behind object detection is to concentrate on precisely estimating the details and exact location of objects contained in each image alongside considering to classify images. Object detection usually is in use in the real world and the use-cases are face detection, pedestrian detection, and skeleton detection. Object detection helps to provide valuable information for semantic understanding of videos and is useful in many applications including image classification, human behavior analysis, face recognition, and autonomous driving by Tesla.

Results related to the Object detection application obtained using the COCO dataset for all the categories.

2. Object/Image Segmentation

Image segmentation is a very important topic in image processing and is helpful to understand the content of the image. It allows exploring many applications such as image compression, scene understanding, locating objects in satellite images ..etc. With the advent of deep learning, More CNN and FasterRCNN models are used for image segmentation along with traditional segmentation models.

Results related to the Object segmentation application for COCO dataset.

Object Detection and Image Segmentation using Mask-RCNN.

Mask R-CNN an object detection and instance segmentation model developed on top of Faster R-CNN. Faster R-CNN is generally a region-based network consists of convolutional neural layers, The output of the Faster RCNN model returns bounding boxes for each class and its class label with a confidence score/detection score.

The architecture of Faster R-CNN works in two stages:

Stage1: It consists of two networks which are the backbone (ResNet, VGG, etc..) and region proposal network. These networks run once per image to give a set of region proposals which are regions in the feature map that contain objects.

Stage2: The network predicts bounding boxes and object class for each of the proposed regions obtained in stage1.

Faster R-CNN predicts object class and bounding boxes. Mask R-CNN is an extension of Faster R-CNN with an additional branch for predicting segmentation masks on each Region of Interest (RoI).

Understanding of Mask R-CNN

Instance segmentation is usually a challenging task that requires detecting all objects in an image and also segmenting each instance at a time which is regarded as two independent processes. It might exhibit systematic errors in some instances due to the multi-task scheme. To solve this, we utilize an additional branch that runs parallel to the existing branches in Faster R-CNN for classification and bounding box regression. The objective of Mask R-CNN is to predict segmentation masks in a pixel-to-pixel manner. This new additional segmentation branch encodes an m*m mask to maintain object spatial layout.

For more information on MaskRCNN: https://arxiv.org/pdf/1807.05511.pdf

To clone the Mask RCNN model to the working project: https://github.com/matterport/Mask_RCNN.git

Data preprocessing:

1. Create a Custom Dataset Class to train the Mask R-CNN model

This custom dataset needs to have three functions

1. load_data(): This will add our specific classes to the data loader class.

2. load_masks(): Load instance masks for the given image. This function converts the different mask formats to one format in the form of a bitmap [height, width, instances].

3. image_reference(): Return a link to the image on the COCO Website.

First, the model must be defined via an instance MaskR-CNN class. This class requires a configuration object as a parameter. The configuration object defines how the model might be used during training or inference. In this case, the configuration will only specify the number of images per batch, which will be one, and the number of classes to predict. It is recommended to use a stronger GPU to train the Mask RCNN model.

2. Prepare Data

Prepare the Data for training the MaskRCNN using customDatasetBuilder class which creates Tensorflow Records.

Model Training

1. Initialize Mask R-CNN model for “training” using the Config instance that we created.

2. Load the pre-trained weights for the Mask R-CNN from COCO data set excluding the last few layers .

We exclude the last few layers from training for ResNet101. Excluding the last layers is to match the number of classes in the new data set. if it is a Custom dataset use pre-trained coco weights to load or if it is the weights and training on imagenet model then use model imagenet weights and finally use the final weights that are previously trained on layers saved in the model path.

3. Train the heads and all layers with higher learning rate to speed up the learning

We can increase the speed of learning for head layers by increasing the learning rate

Also we can increase the epochs to anywhere from 100–500 and see the difference in the accuracy of the object detection. I have used only 20 epochs as I trained it on a GPU hosted in Google Colab pro.

4. Save the trained weights for the custom data set and make inference/predictions for object detection and image segmentation.

5. The output of the model contains a dictionary for each image processed when we passed an image to detect() function. It has keys for the bound boxes/coordinates/regions-of-interest (ROI), masks, mask_ids, and confidence scores. The keys of the dictionary of note are as follows:

1. ‘rois‘: bound boxes/coordinates/regions-of-interest (ROI) for detected objects.

2. ‘masks‘: masks for the detected objects.

3. ‘class_ids‘: integers for the detected objects.

4. ‘scores‘: confidence score

We can draw each box detected in the image by first getting the dictionary for the first image (e.g. results[0]), and then retrieving the list of bounding boxes (e.g. [‘rois’]).

Predictions from the Mask R-CNN model to detect objects in the parking image.

1. Detecting cars, trucks, and boats with bounding boxes as those are the only objects that can be seen inside the parking space.

2. Create an image mask using the weights trained on the Mask RCNN model.

Presentation 2

Future Scope

1. The limitation that this project had is the view of the video angle. If the view is a satellite view, then we would be able to apply OpenCV image transformations to find out parking lines and parking components.

2. We can use Canny Edge Image Detection to detect edges in the image frames so that we can consider the only masked region as our parking regions.

3. Use OpenCV Hough Line transform on the edge images in order to detect white lines/ yellow lines.

4. We have used an object detection model to detect objects in the image. Instead, create a CNN model that takes empty parking spaces as input and trains on them and try to detect empty spaces using the model.

5. The backbone model that we have used is ResNet, VGG, Inception models are heavy models with many hidden CNN layers in them. It would be good if we produce lighter models.

Presentation 3

Source Code:

Planning to use Tensorflow and Python with OpenCV module for the Computer Vision task to extract frames from the input video. The code for the same will be updated on GitHub on a Phase basis.

*Complete Code is posted on Github and i have used Google Drive as the working repository for the project.

https://github.com/akusuma1/DATA606_Capstone_Project

References

1. Mining, Data. “The Survey Says: Car Parking.” The National, The National, 18 Jan. 2017, www.thenationalnews.com/opinion/the-survey-says-car-parking-1.69243?videoId=5754807360001.

2. “Common Objects in Context.” COCO, cocodataset.org/#home.

3. Bill Yang Cai, et al. “Deep Learning Based Video System for Accurate and Real-Time Parking Measurement.” Arxiv, 20 Feb. 2019.

4. Zhang, Lin, et al. “Vision-Based Parking-Slot Detection: A DCNN-Based Approach and a Large-Scale Benchmark Dataset.” IEEE TRANSACTIONS ON IMAGE PROCESSING, 11 Nov. 2018.

5. Ibrahim, Hossam El-Din, Car Parking Problem in Urban Areas, Causes and Solutions (November 25, 2017). 1st International Conference on Towards a Better Quality of Life, 2017, Available at SSRN: https://ssrn.com/abstract=3163473 or http://dx.doi.org/10.2139/ssrn.3163473

6. Geitgey, Adam. “Snagging Parking Spaces with Mask R-CNN and Python.” Medium, Medium, 21 Jan. 2019, medium.com/@ageitgey/snagging-parking-spaces-with-mask-r-cnn-and-python-955f2231c400.

7. Khandelwal, Renu. “Object Detection Using Mask R-CNN on a Custom Dataset.” Medium, Towards Data Science, 9 Feb. 2020, towardsdatascience.com/object-detection-using-mask-r-cnn-on-a-custom-dataset-4f79ab692f6d.

8. Brownlee, Jason. “How to Use Mask R-CNN in Keras for Object Detection in Photographs.” Machine Learning Mastery, 1 Sept. 2020, machinelearningmastery.com/how-to-perform-object-detection-in-photographs-with-mask-r-cnn-in-keras/.

9. Acharya, Debaditya, et al. “Real-Time Image-Based Parking Occupancy Detection Using Deep Learning.” Ceur-Ws, ceur-ws.org/Vol-2087/paper5.pdf.

10. Cao, Yuanzhouhan, et al. “Exploiting Depth From Single Monocular Images for Object Detection and Semantic Segmentation.” IEEE Transactions on Image Processing, vol. 26, no. 2, Feb. 2017, pp. 836–46. DOI.org (Crossref), doi:10.1109/TIP.2016.2621673.

11. Mask R-CNN for Object Detection and Segmentation https://github.com/matterport/Mask_RCNN

12. “Image Processing in OpenCV¶.” OpenCV, opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_table_of_contents_imgproc/py_table_of_contents_imgproc.html.

13. “Greatest Challenges Parking Facilities Managers Face.” All Traffic Solutions, 9 Aug. 2019, www.alltrafficsolutions.com/blog/greatest-challenges-parking-facilities-managers/.