• Choosing the model:

Choosing the model for training the vision-based detection was really tough.

The first choice was YOLO v2 as its faster than other networks like CNN, Fast RCNN, Faster RCNN, etc.

• Training the YOLO v2 model:

An open-source pre-trained pothole detection YOLO model was found from an internet resource, the pre-trained model, however, had many errors. Debugging was very time consuming, but it was then able to do predictions. 

• Retraining the YOLO v2 model:

However, the model was now able to do predictions but the results, but the accuracy was 61%, so the model had to be retrained using our dataset containing the 953 images. As a result, the accuracy was improved.

• Problems with YOLO v2:

The retrained YOLO v2 predicted very well, but issues came when trying to convert the TensorFlow model to the TensorFlow Lite model. Also, a lot of memory was needed to store the tflite model on the mobile phone.

• Solution: Mobile Net SSD v2 quantized coco:

Mobile Net SSD v2 quantized coco is specially designed for mobile applications and Google coral devices. The dataset we used contained 260 images with 321 pothole labels.

• How training was done?

Training of Mobile Net SSD v2 quantized coco was done on a custom-built Computer using NVIDIA GTX 1070 with 8GB VRAM with 1920 CUDA cores. The training took 6 days with 134887 number of iterations. No more iterations took place as the loss was constant.

• Converting the model to tflite model:

To use the Deep Learning model for pothole detection we used an open source TensorFlow pre trained model that was trained with the YoloV2 architecture to detect the potholes. The model was predicting well but problem occurred while integrating it to the android application. A model trained on YoloV2 cannot be converted easily to TensorFlow lite format as some layers of this model doesn’t support TensorFlow lite.

For deploying a machine/deep learning model to android phone the TensorFlow provide TensorFlow lite format for the machine/deep learning model. TensorFlow lite model is then used in android for inferencing.

For using a machine learning model in an android device, the TensorFlow recommends to use their pre trained ssd models like mobilenet and its quantized version.

Our dataset was collected from streets of Lahore like college road, Valencia road etc. After labeling the images training was started on ssd mobilenet v2 quantized version model. After the training we got excellent results from our model. 

Now it was the time to convert that model to TensorFlow lite format, it wasn’t the easy task as we have to build TensorFlow from source and had to create separate environment for converting a TensorFlow model to TensorFlow lite format.

After the training process we exported the TFlite graph (.pb and .pbtxt) using the last saved training checkpoint because for converting a TensorFlow model to TensorFlow lite format you have to export a tflite graph of that trained model. The tflite graph had to run through the TensorFlow lite Optimizing Converter i.e. TOCO as it helps us to convert our tflite graph to .tflite format and for using TOCO we have to build TensorFlow from source. 

While converting the TensorFlow model to TensorFlow lite format we had to resolve many dependencies.  For more information on converting TensorFlow to TensorFlow lite you can go through this link:

github.com/EdjeElectronics/TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi#step-1-train-quantized-ssd-mobilenet-model-and-export-frozen-tensorflow-lite-graph

• Choosing the sensor-based training algorithm:

It was difficult to choose between Support Vector Machine (SVM) classifier and the Logistic Regression (LR). However, after testing both the algorithms SVM proved to give better accuracy 0.63 whereas LR gave 0.55 accuracy.