Fine-Grained Image Classification

Fine-grained image classification is a cutting-edge challenge in computer vision that involves distinguishing between very similar categories within a broader class. Unlike general image classification, where the goal is to categorize images into high-level categories (e.g., dogs vs. cats), fine-grained classification requires identifying subtle differences between subcategories (e.g., different breeds of birds).

We implemented, trained, and compared two state-of-the-art models, modifying them for efficiency and compatibility.

We tracked model performance using Weights & Biases (WandB) and conducted comparative experiments:

Model 1

Training Phase:

We have monitored Model 1 throughout its training steps. The training was done on the competition challenge dataset. The graph in 4.1a shows on the y-axis the accuracy percentage from 0 % to 100 % and on the x-axis represents the training steps. Initially, the training accuracy starts from around 30 % and reaches a stable 95+ % after about 15 epochs. With the drop-out method used the accuracy was not increasing as steadily as without drop-out but achieved the same accuracy early and a higher one overall, as can be seen in 4.1b. The high training accuracy suggests that the model could be overftted.

The graph in 4.2a shows that the training loss is steadily decreasing, close to zero. This is expected as the model learns. 

The graph in 4.3a shows a rapid increase in validation accuracy over the training process. The early steps show noticeable fluctuations in 4.3b as the model fine-tunes its parameters and copes with the eliminated neurons. Both graphs show a very high, close-to-perfect validation accuracy, which is an indicator that the model is overfitted. This can be confirmed in Figure 4.2. 

The gap between training and validation accuracy, along with the plateauing validation accuracy, strongly suggests that the model is not generalizing well to unseen data. The model achieves a nearly perfect level of accuracy and minimal loss, which suggests that the model has learned the training data very well. Fluctuations in validation accuracy suggest instability in generalizing with new unseen data

Testing Phase

After sending our model to the provided server to test its performance, we achieved a test accuracy of 71 %. The accuracy is not on par with current state-of-the-art models, but it is still good compared to the other teams. The ”low” accuracy on the testing set and the high accuracy in training and validation suggest that Model 1 is overfitted. This issue should be tackled in future work

Model 2

Training Phase:

When testing Model 2, we can observe that, overall, it exhibits a lower accuracy on test data compared to Model 1's accuracy rate, which suggests that it presents a better generalization. Decreasing loss values indicate an improved performance on test data as training progresses. From the graph in 4.5b, we have a hint at how badly Model 2 generalises unseen data. Accuracy on test data tends to fluctuate but generally trends upwards, indicating that the model is improving its performance on every training step. The test accuracy is lower compared to Model 1, since it should perform better on the training data. Similar to the local loss, the total average loss on the test dataset starts high and then gradually decreases close to a plateau with some fluctuation with the increase of the training steps.

Testing Phase

Because of a downtime of the test server, we couldn’t test the performance in its current state. We reckon that it won’t perform better, because of results in the early stages that only achieved values around 66 % after epoch number 14.

Comparison Between the Models

Model 1 shows a better learning curve. Accuracy and loss during training are not rapidly changing but steadily increasing or decreasing, respectively. While probably being overfitted, it still accomplished a good accuracy on the test set. Although we haven’t tracked the time per epoch, this model has shown better performance regarding computational load. Beating the time per epoch by almost half regarding Model 2.

Model 2 shows a more rugged response to the testing of patterns on the CUB-200-2011 dataset. Reasons for that could be a small batch size. It also shows that the MMAL-Net has issues adapting to new images. Perhaps this originated in the transfer learning process because the model was pre-trained on the FGVC-Aircraft dataset beforehand. But on the other side, the same model performed more smoothly during training on the competition challenge dataset.

All in all, we would prefer Model 1 because of better results and performance benefits. Also, it is a bit easier to understand and implement into other CNNs.

During the training phase, both models exhibited promising performance on benchmark datasets, such as CUB-200-2011 and FGVC-Aircraft. Model 1 incorporated modules to enhance crucial features, suppress background noise and diversify part-specifc representations. Model 2 adopted a region-based approach, generating multi-scale proposal windows and employing attention modules to locate objects and identify informative parts without manual annotations. Both models demonstrated promising performance on benchmark datasets but Model 1 would be recommended. 

However, limitations were identifed, including increased model complexity, overftting and generalization issues and computational overhead. Future work could focus on exploring new techniques to improve efcient architectures and explore alternative loss functions or regularization methods to enhance generalization capabilities.

References

• Jianwei Song and Ruoyu Yang. Feature Boosting, Suppression, and Diversification for Fine-Grained Visual Classification. 2021. arXiv: 2103.02782 [cs.CV].

• Fan Zhang et al. Multi-branch and Multi-scale Attention Learning for Fine-Grained Visual Categorization. 2020. arXiv: 2003.09150 [cs.CV].