About the safety and security of food, human concern has been increased manifold in the last few decades. Although regarding the usage of pesticides, herbicides and fungicides, stricter and accurate rules have been applied worldwide to make food and environment, safe and beneficial for humans and animals. Another form of pesticides are herbicides (also known as weed-killers). Weeds can be avoided by pulling the weeds by hand, but it can be a difficult and tiring task. To avoid this, herbicides are used by gardeners and farmers to kill unwanted weeds/plants in crops, parks, grocery plots, gardens, etc. The herbicides which are used to kill unwanted plants, have both pros and cons for wanted plants, humans,   animals, and also for environment [1]. These unwanted plants cause many problems in the cultivation of lands like, poison the livestock due to herbicides, increase harvesting and labor cost of production, choke the irrigation canals, decrease average production of any crop, poor air quality and air pollution [2].

World population is increasing continuously with an average growth rate of 1.05% in the last year 2020 [3], thus needs more high nutritional food to fulfil the needs of more people. For the continuity of human life on the Earth we must increase the food quantity as well in the next coming years [4]. To accomplish food requirements for humans, we need a proper mechanism to save food and to increase the productivity of lands. Weeds in crops are one of the major causes in decreasing the productivity of the lands. At the meantime, it is estimated that we lose about 40% of the global yield due to these unwanted plants in crops and it could increase in the next coming years [5]. Weeds are the unwanted plants in crops and compete with crop plants for gaining resources like water, fertilizers, light, space, nutrients and many other resources which damages the land’s productivity to a great extent. Additionally, the quality of farm products reduces, difficulties for harvesting machines in harvesting process and overall the value of cultivated areas reduce due to these unwanted plants [6]. We can increase the farm’s productivity for food to fulfill the human needs for different food products by destroying the weeds from the crops using herbicides. But these herbicides cause bad impact on environment and crops also increases the overall cost of production and labor.

There exist some common species of crop and weed, found in Pakistan’s different agricultural areas, mostly in two provinces, Punjab and Sindh. The main crops in these two provinces are rice, maize, cotton, wheat, sunflower, soybean, carrot, and sugarcane. Some of the basics and common weed species found in these areas are charlock, fat-hen, velvetleaf, black-nightshade and some other broadleaf weeds. Pakistan’s economy is heavily dependent on agriculture sector, these weeds (unwanted plants) are one of the greatest challenges in the growth of agricultural productivity. Deep learning based solutions could be employed for the detection of several weeds and crops using standard RGB images. But the problem is the lack of standard RGB datasets in Pakistan’s agricultural areas for weed/crop detection and classification.

Taking out weeds by hand is still the primary option to maintain the quality and quantity of the farm products. Secondary option is the use of herbicides to control weeds. To control weeds, herbicides are applied uniformly on all areas of field, so areas containing no weeds are also sprayed unpremeditatedly. To secure agricultural productivity and to reduce the usage of herbicide in agriculture field, accurate and automated weed detection and identification is the basic step. Efficient mechanism for weed identification helps in visualizing the threats of weeds in agriculture. Literature review reported a lot of work in weed identification to control weeds and to reduce the competition between crops and weeds for several nourishment resources. Kamilaris et al. surveyed the related studies and described them briefly [8]. These studies are basically image oriented techniques for capturing images like near-in-frared (NIR) images, standard RGB images, multi-spectral images, etc. for weed identification. Survey accomplished by A. Kamilaris et al. shows that majority of work done for weed identification used standard RGB images, due to their ease of availability and scalability. To gain the benefits of advanced standard image capturing techniques, artificial intelligence (AI) based techniques could be hired for weed identification. There are two possible ways to enroll AI techniques in weed identification, one is using handcrafted features techniques, which are classified as machine learning (ML) techniques in which features from the input data are extracted manually and used for weed identification [8] [9] [11]. Second way is to extract the features automatically through convolutional neural networks, in which features from the input data are extracted automatically. In machine learning, features are extracted manually by the experts, so that only dominant features should be used for weed identification system to make the system effective and efficient. But the problem in manual feature engineering is that, it is not estimated that the number of features extracted through handcrafted features engineering are good enough to classify the data into appropriate classes. It may be possible that features extracted for one class may overlap with the features of another class, thus results an inefficient weed identification system. To solve such issues, automatic features engineering techniques are being used by most of the researchers to build an accurate weed identification system [12] [13] [14] [15]. Due to the advancements for pattern recognition, convolutional neural networks (CNN) have become very popular in research areas of AI. Developments in hardware technologies (GPUs) and the availability of large datasets has made it easy for us to investigate the performance of deep neural networks (DNN). The scalability to different data and the growth in CNN’s performance are the main causes for CNN’s popularity.

In recent years, tremendous advancements have been made in computer vision (CV), machine learning and deep learning areas [16] [17]. The most dominant and significant benefit of the deep learning based mechanism is its automatic complex and dominant features extraction from the raw input data [17]. In deep learning, there exists several approaches for features extraction, among all of these approaches, convolution neural networks have made tremendous advancements in terms of pattern recognition in large scale images and videos. The CNN models are based on the working idea of human brain neurons and following same structure to convolve features from the raw input data [18]. With the success of deep learning techniques, the use of the CNN’s is quite common approach for image classification and object detection problems. A lot of work has been done using deep learning CNN’s for the detection of weeds and crops through standard RGB images [12] [13] [15] [19]. Computer vision and deep learning proved very beneficial in this domain and improved the overall results manifolds as compared with previous handcrafted feature extraction methods.

The deep learning based solutions for weed control, could be classified into two main categories. The first one is simply classifying the input images into corresponding weed or crop classes and in this category an input image is required for the model. The model then simply classifies the input image into corresponding predefined classes and therefore ultimately no need to draw bounding boxes or perform segmentation. The second category of deep learning methods works by detecting the objects in the image and then draw bounding boxes around the detected objects. In this category we need to annotate the images and then provide them for model training, validation and testing. These annotations could be bounding boxes, segmented polygons or individual keypoints (landmarks) around the targeted objects. This section first discusses the methods proposed by researchers for classification and then the methods for localization of objects.

For the purpose of weed detection and identification, a dataset was published recently named DeepWeeds [20]. DeepWeeds dataset containing 17509 images belonging to eight weed species found in northern Australia, is a public and large image dataset. The study also proposed a methodology which used two benchmark deep learning models, Inception-v3 and ResNet-50 for accurate weed and crop classification. The trained models were evaluated on test data and the benchmark results on the published dataset reported an overall accuracy of 95.1% and 95.7% for Inception-v3 and ResNet-50 respectively. Hu et al. [24] proposed a deep learning based graph weeds net (GWN) for weeds recognition in standard high resolution RGB images. In this study the input images are divided into three levels (multi-scale), at first level whole image is taken, at second level image is divided into 4 patches (2x2), and at last stage image is divided into 16 equal patches (4x4). By dividing the image into three scales, GWN tried to find the best regional areas with highest probability of having weeds instead of other unrelated plants and background areas. Graph weeds net (GWN) model is evaluated on an open access dataset, published recently in 2019 named DeepWeeds [17]. GWN performed well on weeds classification problem and achieved an overall accuracy of 98.1%. Chavan et al. [24] proposed a deep learning based convolutional neural network for the classification of different weed and crop species at early growth stage. They named their proposed model as AgroAVNET which is a hybrid model developed by combining AlexNet and VGGNET. The concept of normalization in AgroAVNET is taken from the AlexNet and depth of filters in fully connected layers are inspired by VGGNET architecture to gain the benefit of both models in the classification of weeds and crops to control weeds. The performance of AgroAVNET is compared with the two base models, namely AlexNet and VGGNET. The proposed AgroAVNET model architecture outperforms both base models. They also proved the ability of incremental learning of AgroAVNET model to use the previously trained model to add new species for classification by using hybrid VGGNET model [25]. The proposed model AgroAVNET is evaluated on ‘Plant Seedlings Dataset’ [21], a publically available dataset having large number of standard RGB images corresponding 12 weed and crop species. AgroAVNET performs well and achieved an average accuracy of 98.21%, while AlexNet achieved 94.99% and VGGNET achieved an average accuracy of 92.81%. There are also several methods proposed by researchers for crop specific weeds identification and classification to control these unwanted plants (weeds). Ferreira et al. [22] proposed a convolutional neural network for weed detection in soybean crop using standard RGB images and classifies these images into soil, grass, broadleaf weed and soybean crop classes. A convolutional neural network proposed, namely ConvNets for the training of neural network CaffeNet architecture is used, available in a software Caffe [23] as the state of the art method (replication of AlexNet which won the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)) for weeds classification in soybean crop. First of all a robust dataset is created by using super-pixel segmentation algorithm. This dataset contains thousands of images for each class, namely grass, broadleaf, soil, and soybean. The performance of ConvNets is then compared with other conventional handcrafted feature based methods like support vector machines (SVM), AdaBoost and Random Forests. ConvNets achieved an accuracy of 98%, when used for the detection of grass and broadleaf in relation with soil and soybean. However an overall average accuracy of 99% is achieved using ConvNets. Yu et al. [24] in a study proposed a deep learning based convolutional neural network for weed detection in perennial ryegrass. Several deep convolutional neural network (DCNN) models are employed to detect weeds in perennial ryegrass images to control weeds automatically through automatic spot spraying system. Some common DCNN’s like AlexNet [25], VGGNET [26], DetectNET [27], and GoogleNet [28] are used for the detection of Euphorbia maculate, Glechoma hederacea, and Taraxacum officinale weeds, found in perennial ryegrass. These three DCNN models are evaluated on three weed species through single-specie neural network and multi-specie neural network. In case of single-specie neural network AlexNet and VGGNET outperform when compared with GoogleNet and DetectNET models performance. While in case of multi-specie neural network VGGNET performs well as compared with other methods and achieves 0.93 F1 score. DetectNET performs well with 0.98 F1 score for the detection of Taraxacum officinale weed specifically.

To promote the usage of automatic devices in smart agricultural farming Trong et al. [29] proposed an intelligent voting system for weeds only classification. The proposed voting method works by fusing the features of multimodal deep neural networks. The voting is done with the help of score vector which is calculated for each DNN by using priority weights and Bayesian conditional probability-based methods. The DNN having more accurate classification, is given high priority as compared with other DNNs. The proposed method is evaluated with five DNN architectures VGG, Inception-Resnet, Resnet, NASNet and Mobilenet. Training and validation is done on two standard datasets ‘Plant Seedlings Dataset’ [21] dataset and ‘Chonnam National University’ (CNU) [29] weeds dataset. The proposed model achieved an accuracy of 98.77% on CNU weeds dataset and 97.31% on Plant Seedlings dataset with the ability of making predictions in real time. Multiple convolutional neural networks are used by Peteinatos et al. [30] for weed identification in Potatoes, Sunflower and Maize crops. A huge collection (93000) of standard RGB images is collected for 12 weed species found in Potatoes, Sunflower and Maize crops. To construct an accurate and efficient weed classification system three common CNNs VGG16, Exception, and ResNet-50 were used and trained on the collected 93000 images. About 70% images from the dataset were used in model training and remaining 30% images were used for validation (15%) and testing (15%) respectively. F1-score, precision, recall and Top-1 accuracy were the evaluation metrics used. The proposed deep learning based neural network achieved an average Top-1 accuracy of 77% to 98% for weed identification in targeted crops. In another method proposed by Jin et al. [12] for weed identification in vegetable plantation, image processing and deep learning methods were combined to achieve a less complex weed identification system. To downgrade the complexity of the proposed CenterNet model, they mainly focused on detecting vegetables only area from the image and a bounding box was drawn around the detected vegetables. The green objects retained in the image after the detection of vegetable plants were considered as weed species. To determine the color index in the image, Genetic Algorithms (GA) are used. The proposed model CenterNet achieved a recall of 95.0%, a precision of 95.6%, and a F1-score of 95.3% respectively [12]. On the other hand to avoid overfitting, a bunch of deep learning based models (Exception, Inception-Resnet, VGNets, Mobilenet, and Densenet) are used as base-models by implementing the concept of transfer learning for weed identification using robotic AI based sprayer [19]. On the top of these base-models, some popular machine learning models (Support Vector Machines, XGBoost, and Logistic Regression) are used to classify the extracted features into corresponding classes. The proposed method by Garcia et al. [19] is evaluated on an open access dataset named early-crop-weed [31] and successfully achieved the state of the art 99.29% F1-score using Densenet and Support Vector Machines models. There exist a number of challenges like different growth stages, illumination changes, variance in plant appearance and foliage occlusions, in the detection of convolvulus sepium (hedge bindweed) in sugar beet fields. Gao et al. [32] proposed an image based deep convolutional neural network (DCNN) for the detection of Convolvulus sepium  in sugar beet fields to tackle all the challenges described above for C. sepium detection. In this study a DCNN was proposed which is YOLOv3 like architecture for weed detection and localization, YOLOv3 is a variation of you only look once (YOLO) commonly used lightweight model for object detection [33]. To avoid the labor intensive task of taking and labeling images manually, they generated synthetic images from the formerly taken images to increase the volume of dataset for training. Instead of randomly initializing the weights they used a pre-trained model weights darknet53, (trained on ImageNet dataset) to take the advantage of transfer learning. The results of proposed model are quite good when compared with the results of YOLOv3 and YOLOv3-tiny network architectures. The proposed YOLOv3 like architecture achieved the average precisions of 76.1% and 89.7% for C. sepium and sugar beet respectively. For an efficient, accurate and automatic identification system for weeds, many studies have been carried out in the past by researchers. The majority of these studies have been carried out only for classification directly from the input image by using simple handcrafted features as well as automatic features engineering through deep learning CNNs [20] [22] [24] [29] [30] [34] [35]. Mostly studies used support vector machines (SVM), Random Forests, AdaBoost and other machine learning classifiers for final classification of weeds and crops. All these machine learning classifiers are dependent on good feature extraction methods. If the features (color, texture, pattern, shape) extracted from the input images are inappropriate than the performance of these classifiers will not be as expected. A variety of challenges adds up with these limitations in the classification of image like background analysis, illumination conditions, viewpoint, intensity variations, image occlusion and image deformation.

Furthermore, several studies have also been carried out for the detection (bounding boxes, localization) of crop/weed species from standard RGB images through automatic deep learning based feature extraction methods [12] [19] [32]. All these studies have proposed deep learning based solutions for automatic weed/crop identification and localization with datasets having small number of images and thus lack of generalization. Moreover the datasets used to train the models don’t have much variations in images for same class, however in real life environment there are a lot factors to be considered which cause variations in images like weather conditions, background variations, image acquisition time (morning, noon, afternoon, evening night) etc.

Several machine learning based solutions like support vector machine (SVM) [19], random forest , AdaBoost, neural networks (NN) [22], etc. were proposed by the researchers to control unwanted plants (weeds) in farms. But due to the chances of misleading features extraction and selection in manual features engineering, the researchers moved themselves towards automatic features engineering through deep convolutional neural networks (DCNN). Some common deep learning based solutions proposed by researchers in last decade to control weeds were YOLO [32], DenseNet [19], ResNet [30], VGGNet, AlexNet, GoogleNet [24], etc. In most studies the researchers just use a bunch of deep convolutional neural networks and selected the best one without concentrating on which model architecture should outperform on the selected dataset. It is also studied that no common standard dataset used by the searchers, instead a number of datasets used for training and testing, to test the performance of the trained models. Here we proposed a standard pipeline for the classification of weeds and crops into predefined classes, using deep convolutional neural networks with the concept of transfer learning as shown in Figure 1. In proposed pipeline first of all, a standard RGB image is taken as input image and then some preprocessing techniques like color correction, enhancing contrast, resizing image, etc. applied to enhance the performance of the proposed segmentation model. The task of the trained model for segmentation is to extract the region of interest (ROI).