Using Vision-Language Model (CLIP) for Automotive Parts Classification

The primary objective of this project is to develop a scalable vehicle classification system capable of identifying a car's make, model, year, and specific exterior components from a single image. Unlike traditional approaches that rely on standard Convolutional Neural Networks (CNNs) with fixed classification heads, this project explores an unorthodox application of Contrastive Language-Image Pre-training (CLIP) By leveraging CLIP's vision-language capabilities, we aim to create a flexible system that can interpret complex visual features and map them to natural language descriptions, allowing for a more dynamic and scalable classification process.

Over a period of five weeks, this pipeline allowed us to collect approximately 14,000 images. Following collection, the data underwent a rigorous manual cleaning process to ensure that labels for year, make, model, and part type were accurate. The total dataset, including augmented samples (4 augmentations per original image), comprises approximately 70,000 images.

CLIP has trained on hundreds of millions of image-text pairs to understand the relationship between visual concepts and natural language. The model learns by maximizing the similarity between the image embedding and its correct text embedding while minimizing similarity with incorrect text descriptions. It is important to note that CLIP is not a generative model; it does not create new images or generate text descriptions from scratch. Instead, it functions as a similarity-based matching engine. The model then calculates the cosine similarity between the input image and every provided text label. The label with the highest similarity score is selected as the correct prediction.