Computer Vision for Quality Inspection of Manufacturing Product
Department of Computer Science, UW-Madison
Stephen Ling, Feijun Chen, Yizheng Shi, Zhucheng Li
Content
1. Introduction
1.1 Motivation
Defects in products are undesirable in the industry and need to be inspected and found during the massive manufacturing processes, which can be highly inefficient when dealing with human resources. Quality inspection, to eliminate undesirable imperfections, has the fundamental issue: the examination procedure is manual. Due to human error, this procedure is time-consuming and is not entirely accurate. Failure in accuracy can lead to a significant loss for corporations and public safety issues. Under this circumstance, better approaches with the involvement of computer vision technologies can be used. Therefore, developing analysis algorithms and models to process product image data is found to be effective, that has improvements in cost reduction and accuracy enhancement.
1.2 Problem Statement
In this project, we are planning to solve the problem: how to apply computer vision in classifying qualified impellers and defective impellers in order to better manage the process of quality inspection and control. The major goal is to find the image processing methods and classification algorithms that could result in higher accuracy and are able to detect damaged/defective areas of the manufacturing product.
[Click Image to Compare Qualified and Defected Impellers]
1.3 Data Overview
The data is provided by Ravirajsinh Dabhi on the Kaggle platform. The dataset we would use contains 1,300 images of impellers with the size of 512*512 pixels. Specifically, images could be separated into two categories based on the purpose of this project: 519 images for qualified impellers, and 781 images for defective impellers. When reading images, we assign label = 1 for images of qualified impellers, and label = 0 for images of defective impellers.
Link: https://www.kaggle.com/datasets/ravirajsinh45/real-life-industrial-dataset-of-casting-product
1.4 State-of-Art
An overview of computer vision methodologies for quality inspection (a classification problem)
Computer vision is widely used for quality inspection of different objects like glass defects, food/agricultural products, textiles, packages, etc. (Neethu & Anoop, 2015). Moreover, the current process for quality inspection through computer vision involves 5 main steps: image acquisition, pre-processing, segmentation, feature extraction, and classification (Rajan, Jayakrishna, Vignesh, Chandradass, & Kannan, 2021).
For image pre-processing, the main tasks performed are correction of geometric distortions, removal of noise, gray level correction, and correction for blurring (Shirai, 1987). In specific, averaging and Gaussian filters are often used for noise reduction (Brosnan & Sun, 2004). Moreover, due to the cost of collecting defective objects, a limited number of images are collected. As a result, data augmentation is also usually performed to enhance learning performance without materially undermining the knowledge included in the original data (Ha & Jeong, 2021).
Feature extraction strategies are determined based on the feature type. For defective industrial product quality inspection, the feature type researchers are interested in is the texture feature. For texture feature extraction, commonly used models are the Gaussian Markov Random Field (GMRF) model, Homogeneous Texture Descriptor (HTD), LoG method, Difference of Gaussian (DoG) method, and Support Vector Machine (SVM) (Salau & Jain, 2019).
Through feature extraction, the feature vector is obtained for classification models. Based on the literature, both neural network models and non-neural network models like SVM, kNN, etc. are used for image classification. From the quality inspection perspective, the widely used model is the Convolutional Neural Network (CNN) due to its self-learning and compression capabilities, and recognition of hidden patterns: in the ImageNet Challenge, the CNN model obtained higher accuracy than humans perceived (Ha & Jeong, 2021). Also, other deep learning methodologies like Restricted Boltzmann Machine (RBM), Deep Belief Networks (DBNs), etc. are also widely used (Voulodimos, Doulamis, Doulamis, & Protopapadakis, 2018).
Finally, since the quality inspection problem is a type of supervised pattern classification problem, the evaluation methodologies like confusion matrix, ROC curves, etc. could be used for the evaluation of model performance.
2. Image Pre-Processing
Data Augmentation
The augmentation functions are meant to add more invariances into the model with different image properties but the same label, which thus increases the size of the dataset for better training outcomes. The following methodologies are used for data augmenting:
Stretch: choose a ratio, ranging from 1 to 1.15, and stretch the image horizontally and vertically at such random ratios.
Brightness: adjust the brightness of the image with a ratio from 0.7 to 1.2.
Flipping: randomly flip the image vertically, horizontally, or diagonally.
Crop & Zoom and Channel Shift are not chosen because such methods would "destroy" certain features.
Original
Stretch
Brightness
Flipping
3. Feature Extraction
3.1 Canny/Sobel Edge Detector
Since the surface of the impellers is very smooth, the defective areas could be considered as noise pixels on a smooth graph. The Sobel and Canny edge detection operator is sensitive to noise pixels, we believe they would be good feature extraction methods for our problem.
In a comparison of these two methods, the Sobel detector cannot produce smooth and thin edges like the Canny detector, but the Sobel would be simpler and more time efficient.
Output of Edge Detector
Visualization of Feature Vectors obtained through GLCM
(PCA used to reduce dimension to 2 for visualization purpose)
3.2 Gray-Level Co-occurrence Matrix (GLCM)
The GLCM functions characterize the texture of an image by calculating how often pairs of pixels with specific values and in a specified spatial relationship occur in an image, creating a GLCM, and then extracting statistical measures from this matrix (MATLAB, n.d.). GLCM is commonly used to extract texture features from images.
In this project, we select the Contrast measurement method, which measures the local variations in the gray-level co-occurrence matrix (a measure of the intensity contrast between a pixel and its neighbor over the whole image), as extracted feature vectors. The resulting feature vector for each image is in shape(1, 4).
4. Models
4.1 K-Nearest Neighbors (kNN): Fastest Training
K-Nearest Neighbors algorithm classifies an object by assigning it to the class of most common among its k nearest neighbors. We choose kNN because it does not require a training period (instance-based learning), which could greatly reduce the running time of our model.
However, kNN does not work well with high-dimension data and is sensitive to outliers that with feature vectors extracted through GLCM, the highest accuracy is only around 87%.
The sklearn package is used to develop and train the model.
Demonstration of KNN
4.2 Convolutional Neural Network (CNN): Highest Accuracy
A Convolutional Neural Network (CNN) is a Deep Learning algorithm that usually takes images as input and assigns learnable weights and biases to different items in the image so that the model could be able to differentiate each image or image class (Saha, 2018).
Image: A CNN sequence to classify handwritten digits (Saha, 2018)
The plot below is the structure of the CNN model we used for classification. Balancing between computation time and accuracy, the CNN model is designed with batch = 16, and epoch = 10.
The tensorflow package is used to develop and train the model.
CNN Model Summary
CNN Model Structure
4.3 SHapley Additive exPlanations (SHAP): Finding Defective Area
In addition to classifying images of defective and qualified impellers, we are also interested in identifying defective areas for defective impellers. First, we wish to solve this problem as an object detection problem. However, there are different types of impellers that we are unable to go over 781 images to crop images of defective areas for object detection.
However, Lundberg & Lee purposed a methodology named SHAP (SHapley Additive exPlanations), which can be used to figure out how the features of given data are related to the outputs (2017). In general, SHAP values are commonly used to increase transparency and interpretability of machine learning models (Trevisan, 2022).
Since qualified and defective impellers are identical other than the defective areas, we could detect defective areas by visualizing SHAP values to identify what parts of images were regarded as important features for classification.
5. Result & Analysis
5.1 K-Nearest Neighbors (kNN)
Two plots below indicate the training and testing accuracy curve for kNN model with feature vectors obtained through Sobel (Left) and GLCM (Right).
Accuracy Curve (Sobel)
Accuracy Curve (GLCM)
Confusion Matrix
In the problem of image inspection of impellers, it is considered that the following effects will occur due to misclassification.
False Negative : Since it is a "mis-detection" case which means the model cannot classify defect images as 'defect', there is a risk that it may adversely affect the subsequent process or lead to defects in the finished products.
False Positive : Since it is a "over-detection" case which means the model misclassify normal images as 'defect', there is a risk that the line downtime will be unnecessarily increased or the cost for reproduction will be unnecessarily increased.
GLCM
According to confusion matrix, out of 650 images, 45 was misclassified to be defective (False Positive), 41 was misclassified to be normal (False Negative).
Canny
According to confusion matrix, out of 650 images, 147 was misclassified to be defective (False Positive), 60 was misclassified to be normal (False Negative).
Sobel
According to confusion matrix, out of 650 images, 108 was misclassified to be defective (False Positive), 30 was misclassified to be normal (False Negative).
Conclusion:
The GLCM model shows the most accuracy with 86.77% in general. Canny shows the least accuracy with 68.15% in general. For correctly classifying the defective impellers, Sobel is the best model with an accuracy of 92.31%. For correctly classifying the qualified impellers, GLCM is the best model with an accuracy of 82.63%.
5.2 Convolutional Neural Network (CNN)
The two plots below indicate the trend of accuracy and value loss as the number of epochs increases. It is obvious that both training and testing accuracy increase as the number of epochs increases. Eventually, the CNN model stops training at epoch = 7 due to value loss no longer improving, and the overall training time is around 7 minutes. The test accuracy is 97.76% with a loss of 0.5405, which suggests the CNN model performs significantly better than the previous kNN model.
To further understand the model performance, we calculated several model evaluation statistics including precision, recall, f1-score, and accuracy. When reading images, we assign label = 1 for images of qualified impellers and label = 0 for images of defective impellers.
Precision = True Positive / Actual Results. Precision suggests the percentage of relevant results. In comparison, Recall = True Positive / Predicted Results. Recall refers to the percentage of total relevant results correctly classified by the algorithm. Finally, the F1-score, is simply the harmonic mean of precision and recall.
The evaluation statistics suggest the model performs better in predicting defective impellers with f1-score = 0.985 > 0.976. In general, the model reaches 98.2% accuracy for the task of quality inspection.
5.3 SHapley Additive exPlanations (SHAP)
The absolute SHAP value shows us how much a single feature affected the prediction. The positive sharp value indicates how likely a feature contributes to the prediction of label = 1 (Qualified), and the negative sharp value indicates how likely a feature contributes to the prediction of label = 0 (Defective).
To interpret the plots below, we can say the blue dots indicate the features that "push" prediction toward defective impellers, while red dots indicate features that "push" prediction toward qualified impellers.
Also, using SHAP to interpret misclassified images, the SHAP visualization suggests the shadows have a major contribution to misclassification problems. This suggests the manufacturing product images should be obtained under the strong light condition to avoid the influence of shadow during the prediction.
Example 1
Example 2
Example 3
5.4 Comparison
We randomly select 5 highly rated implementation of CNN model under the data and run each model. All models take more than 20 minutes training time, and the resulting accuracy is between 92% and 99%. In comparison, our model takes less than 7 minutes to train and reach 98% accuracy. In addition, through using SHAP values, we are also able to identify defective areas on impellers.
6. Problem & Future Implementation
6.1 Encountered Problems
This is our first time working on a project in the Computer Vision field so we encountered several difficulties during our exploration and implementation processes.
It is a bit difficult when handling labels for traditional classifiers: for neural network models, we can directly use image-generating functions to iterate images, but for traditional classifiers, we need to pack all images with labels into a data frame without assigning a wrong label to the image during augmentation and train/test splitting.
The lack of computing power makes our initial neural network models unable to run. So, we had to adjust some training factors, such as the batch size and the number of layers, to avoid failure of maximum space of memory allocation and long computation time.
For identifying defective areas, we started with the idea of object tracking in class. However, after implementing the base algorithm, we realize that there are different types of defective impellers so we have to crop each type of defective area for object tracking. Then, after reading a few papers, we come up with the GRAD-CAM heat-map, which is a technique used by some researchers. However, we still failed in imposing a heat-map on the original images. Finally, we realize our goal through SHAP values.
6.2 Future Implementation
The original dataset contains 1300 images, including both defective and qualified images. The augmentation methods can be improved by using combinations of augmentation functions to transform images, which allows the images to be more different from the original image and thus provides a more varied dataset for training. In addition, a policy other than random to automate the process of calling functions could be chosen to gain a more solid dataset. Research on what a good policy is to augment images is an extensive area of research
Further, defective areas could be more clearly displayed (e.g., draw a box around the area of defection). In this project, even though the SHAP value could identify some defective areas, it is still not obvious to viewers.
It is also worth-thinking that how to improve the prediction accuracy by removing the shadow on the impellers since we conclude in part 5.3 that SHAP visualization suggests the shadows have a major contribution to misclassification problems.
Finally, we could also record a video of impellers on the conveyor belt, and try to tag defective impellers in the video.
7. Materials
Github Repository: https://github.com/Garrred/CVProject
8. Reference
Brosnan, T., & Sun, D. W. (2004). Improving quality inspection of food products by computer vision––a review. Journal of food engineering, 61(1), 3-16.
Dhingra, S., Bansal, P. (2020). Experimental analogy of different texture feature extraction techniques in image retrieval systems. Multimedia Tools and Applications, 79, 2739127406. https://doi.org/10.1007/s11042-020-09317-3
Ha, H., & Jeong, J. (2021). CNN-based defect inspection for injection molding using edge computing and industrial IOT Systems. Applied Sciences, 11(14), 6378. https://doi.org/10.3390/app11146378
Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.
MATLAB. (n.d.). Texture Analysis Using the Gray-Level Co-Occurrence Matrix (GLCM). MathWorks. Retrieved December 3, 2022, from https://www.mathworks.com/help/images/texture-analysis-using-the-gray-level-co-occurrence-matrix-glcm.html
Neethu, N. J., & Anoop, B. K. (2015). Role of computer vision in automatic inspection systems. International Journal of Computer Applications, 123(13).
Rajan, A. J., Jayakrishna, K., Vignesh, T., Chandradass, J., & Kannan, T. T. M. (2021). Development of computer vision for inspection of bolt using convolutional neural network. Materials Today: Proceedings, 45, 6931-6935.
Ray. (2022, February 19). Color, shape and texture: Feature extraction using opencv. Medium. Retrieved November 8, 2022, from https://medium.com/mlearning-ai/color-shape-andtexture-feature-extraction-using-opencv-cb1feb2dbd73
Salau, A. O., & Jain, S. (2019, March). Feature extraction: a survey of the types, techniques, applications. In 2019 International Conference on Signal Processing and Communication (ICSC) (pp. 158-164). IEEE.
Saha, S. (2018, December 18). A Comprehensive Guide to Convolutional Neural Networks — the ELI5 way. Toward Data Science. Retrieved December 3, 2022, from https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53
Shirai, Y. (2012). Three-dimensional computer vision. Springer Science & Business Media. Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018). Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018.
Trevisan, V. (2018, December 18). Using SHAP Values to Explain How Your Machine Learning Model Works. Toward Data Science. Retrieved December 3, 2022, from https://towardsdatascience.com/using-shap-values-to-explain-how-your-machine-learning-model-works-732b3f40e137