Maturity analysis of date fruits for robotic harvesting

Introduction

Date fruits can be harvested and marketed at three stages of their development—Khalal, Rutab, and Tamar. Dates in Khalal are the first in the harvesting season, where dates are considered ready for sale as fresh ripe fruit. Khalal dates have a crisp texture, hard surface, and bright yellow or red color. The Rutab stage is the second in the harvesting season. Rutab starts with the appearance of a partially browned color at the date’s tip and then spreads gradually to the whole date. Rutab is delicate, highly perishable, and has short development time. Tamar is the final development stage of dates. Tamar dates have texture from soft pliable to firm to hard, and color from amber to dark brown. In the Tamar stage, dates are non-perishable, so they can be stored for a long period of time and can be consumed throughout the year. The choice for harvesting at one or another stage depends on several factors, including climatic conditions, date variety, and market demand [1].

Samples of Barhi date fruit in immature and three mature stages (Khalal, Rutab, and Tamar). 

Challenges

Maturity classification of date fruits is a challenging because dates grow in clusters (bunches) and can be harvested in different maturity stages that overlap because the dates in one bunch do not mature at the same time. This fact makes them hard to classify or even label by an expert. Furthermore, many external effects make classification based on maturity difficult. This includes images of date bunches covered by bags and images with poor illumination.

Most fruits have one maturity index, while the date fruit has three maturity indices. However, dates are usually harvested as bunches, and individual dates in one bunch do not mature uniformly (i.e. a date bunch usually has individual dates at different maturity stages).

Proposed solution:

For harvesting objective, the date bunches are categorized into seven classes : Immature-1, Immature-2, pre-Khalal, Khalal, Khalal-with-Rutab, pre-Tamar, and Tamar. These maturity classes are based on the color and texture of dates as well as the harvesting decisions and methods described by experts and farmers.

The proposed date maturity classes for harvesting objective

Sample images of Sullaj dates labeled into the seven maturity classes based on the threshold values of individual dates.

Demonstration of the labeling process into the maturity classes. The process depended on the visual estimation of the number of individual dates in date bunches that belonged to the four maturity stages. 

Deep convolution neural networks CNNs are utilized with transfer learning and fine-tuning on pre-trained models, e.g. VGG-16, ResNet-50, ResNet-100, inception v3. To build a robust machine vision system, we used a rich image dataset of five date types for all maturity stages. The dataset was designed with a large degree of variation that represents the challenges in natural environments and date fruit orchards. 

Example results:

The video shows the results of maturity classification of date fruits using Convolutional Neural Networks CNNs with transfer learning based on AlexNet, VGG-16, and ResNet-50 models. 

Accumulated accuracy is used in the video, which calculates the accuracy of all frames from the beginning of the video to the current frame, and hence the final average accuracy is at the end of the video.

Training process of the CNN classification system using transfer learning.

Deep learning architectures of the proposed date fruit classification system based on VGG-16 pre-trained model. 

Referred paper:

H. Altaheri, M. Alsulaiman and G. Muhammad, "Date Fruit Classification for Robotic Harvesting in a Natural Environment Using Deep Learning", in IEEE Access, vol. 7, pp. 117115-117133, 2019. (10.1109/ACCESS.2019.2936536)


Dataset download links: (You have to login with an IEEE Account to download the files. IEEE Account is FREE)

High resolution images (8079 images, 42 GB).

A preview of the 8079 images. size: 224 X 224. (108 MB). 

The annotation (labeling) files (104 KB) 


Code download link:

https://studentksuedu-my.sharepoint.com/:f:/g/personal/435108376_student_ksu_edu_sa/En8jKZ_wAWFOqyoga-PeQjAB4EDmzyFpQXiMOMAqZdFUMg?e=WFd7hU

[1]  Z. Abdelouahhab and E. Arias-Jimenez, Date palm cultivation., 1st ed. Food and Agriculture Organization (FAO), 2002.