Embedded Files
SSD-MobileNet
YOLO v3-tiny
1. Demonstration
1. Demonstration
The videos on the left hand side are the demos showing the object detection we developed on the Android app. The top one is the application with SSD-MobileNet model. The bottom one is the application with YOLO v3-tiny model.
There is a small square image on the bottom right of the application screen. It's where we display the latest result that the model has predicted. The application then reshape the predicted bounding box to fit the screen view size and draw it on the screen. As a result, the bounding box simultaneously display on the screen and the bottom-right image, but the frame on the screen and the bottom-right image is inconsistent.
The box on the bottom left of the application screen indicates the information the model do the inference.
2. Performance evaluation
2. Performance evaluation
2.1 Data
2.1 Data
Since we pretrained our model by COCO, the testing data set we use is COCO 2017 Val. It includes 5,000 images with 80 categories. In its annotation file, each bounding box is denoted as image id, xmin, ymin, width, height, and category id. Thus we can do the evaluation by comparing the ground-truth bounding box in annotation file and the predicted bounding box we detected by our models.
2.2 IoU (Intersection over Union)
2.2 IoU (Intersection over Union)
IoU = (the area of overlap between the predicted bounding box and the ground-truth bounding box) / (the area encompassed by both the predicted bounding box and the ground-truth bounding box)
2.3 mAP (mean Average Precision)
2.3 mAP (mean Average Precision)
mAP (mean Average Precision) is the average of AP (Average Precision) over all categories.
We use mAP@.5, which means the mAP with IoU=0.5, to evaluate the quality of the model. The evaluation code is developed on top of the COCO API.
2.4 Inference Time
2.4 Inference Time
Since we aim to do real-time object detection on Android app. Speed is a very important factor to evaluate. We see the inference time that a model in the Android app detect new bounding boxes on a frame as the metrics to evaluate speed. The inference time is shown in the left bottom information box on application screen.
2.5 Evaluation Results
2.5 Evaluation Results
For the speed evaluation, it's easy to see that SSD-MobileNet is faster by the inference time we provide on the right sheet and also by the demo.
For the quality evaluation, we can see that the mAP@.5 is higher on the SSD than YOLO.
We can see the further discussion in the next section.
Report abuse