Abstract

The identification of rare fruit varieties facilitates sustainable fruit production systems and plays a crucial role in preserving biodiversity, contributing valuable expertise to various aspects of fruit cultivation and conservation. We present the results of interdisciplinary research to develop an intelligent system for the identification and description of fruit varieties. The project employs a multimodal approach, integrating multisensory visual inputs from cameras (2D images) and LiDAR (3D scans) with advanced machine learning models to identify historical apple varieties. The unique dataset comprises images and 3D scans from the Swiss apple core collection, harvested from 2021 to 2023, with approximately 24 samples per variety each year. Our primary focus is on integrating diverse sensory inputs and evaluating various machine learning models, including traditional feature-engineering based methods, convolutional neural networks (CNNs), and emerging geometric deep learning techniques. This approach aims to enhance the accuracy and efficiency of identifying subtle varietal differences. Our initial results indicate that the use of 2D imaging could surpass the performance of human experts in identifying apple varieties. We utilized a pretrained 2D CNN on segmented apple images to achieve above 90% accuracy. Additionally, similar effectiveness was observed with a multi-input 1D convolutional network that processes apple contours and extracted color information. These advanced machine learning approaches demonstrate superior performance compared to traditional methods based on hand-engineered features, offering promising prospects for more nuanced varietal recognition. The ongoing analysis of 3D data offers insights into crucial aspects of the apple that are not discernible in 2D images, providing a richer understanding of varietal characteristics. In summary, our innovative approach of integrating machine learning with multisensory data, incorporating the expertise of pomologists, represents a significant improvement for the conservation of apple biodiversity.