Editor’s note: This is the third in a series of three posts outlining the findings of research our in-house computer vision team conducted regarding the accuracy of popular open-source object detection models for detecting vehicles, as measured by pixel level accuracy. Before diving in, be sure to check out part one and part two to understand the scope of our experiment.
Localization accuracy and object size
To determine if the localization accuracy is correlated with the size of an object in the image, our team computed the box size (√(w*h)) of the true positives (TP) over Intersection over Union (IoU) and pixel deviation.
Following is the curve for Mask R-CNN which shows that larger objects tend to have higher IoU values than smaller objects. At a box size of 130 pixels, the average IoU is 0.9, at 75 pixels it is 0.8, and at 50 pixels it is 0.7.
Figure: Relationship between object size and IoU
We see the opposite correlation when using pixel deviation to measure localization accuracy. Smaller objects tend to have a smaller pixel deviation than larger objects. At a box size of 80 pixels, the average pixel deviation was 3 pixels, at 110 pixels it was 5 pixels, and at 150 pixels it was 13 pixels. Intuitively this makes sense, since Faster R-CNN and Mask R-CNN rescale the features inside every box generated by the region proposal network to a fixed size prior to classification.
Figure: Relationship between object size and pixel deviation
Performance on gray images and noisy images
In our final set of experiments, we evaluated the models on degraded image data. We generated three new test datasets by converting images to grayscale, adding 5% of salt and pepper noise, and adding Gaussian noise with a variance of 0.05. Sample images from all four test datasets follow.
Original Image
Gaussian additive noise with a variance of 0.05
Grayscale
Salt and pepper noise, 5% of pixels
The precision-recall (PR) curves for the top two performing models follow. At 90% precision, Faster R-CNN NAS achieves 75% recall on the original image set, 72% recall on the gray images, 65% recall on the images with salt and pepper noise, and 55% recall on the images with added Gaussian noise (var = 0.05).
Figure: Figure: PR curves for Faster R-CNN NAS IoU thresholds of 0.5
Mask R-CNN does better, achieving 75% recall in original color images, 72% recall on the gray images, 67% recall on the images with salt and pepper noise, and 62% recall on the images with Gaussian noise.
Figure: PR curves for Mask R-CNN IoU thresholds of 0.5
Following are some examples of Mask R-CNN on images with added Gaussian noise.
Mask R-CNN detection on original image
Mask R-CNN detection on image with Gaussian additive noise with variance of 0.05
Mask R-CNN detection on original image
Mask R-CNN detection on image with Gaussian additive noise with variance of 0.05
Summary
Across our experiments, we uncovered several interesting findings.
The winners: The top performing models were Mask R-CNN ResNet v2 and Faster R-CNN NAS. For Mask R-CNN, 50% of the TPs had an IoU > 0.9, 80% had an IoU > 0.8, and 90% had an IoU > 0.7. When using pixel deviation to measure localization accuracy, Faster R-CNN NAS performed best with 25% of TPs having a deviation of < 3 pixels, 50% of TPs having a deviation of < 5 pixels, and 90% of TPs having a deviation of < 13 pixels.
Object size and localization accuracy: We found that the IoU of the detections increases with the size of the vehicles in the image, while the pixel error favors smaller objects, i.e., the pixel deviation decreases with object size.
Image quality: Surprisingly, removing color from images only slightly decreased the models’ accuracy. Adding heavy Gaussian image noise lead to a significant drop in recall.
Humans are still superior: Human annotations were more accurate—as much as three times as demonstrated by the pixel error: 24% of TPs were within 1 pixel deviation, 73% of TPs were within < 3 pixels deviation, 86% of TPs were within < 5 pixels deviation, and 94% of TPs were within < 10 pixels of deviation.
We hope you enjoyed following along with our experiments. To learn more about how Mighty AI can help your team curate high-quality training datasets for your computer vision models, get in touch today.
Leave a Reply