Image to image translation is the problem of transferring an image from a source domain to a target domain. We present a new method to transfer the underlying semantics of an image even when there are geometric changes across the two domains. Specifically, we present a Generative Adversarial Network (GAN) that can transfer semantic information presented as segmentation masks. Our main technical contribution is an encoder-decoder based generator architecture that jointly encodes the image and its underlying semantics and translates both simultaneously to the target domain. Additionally, we propose object transfiguration and cross-domain semantic consistency losses that preserve the underlying semantic labels maps. We demonstrate the effectiveness of our approach in multiple object transfiguration and domain transfer tasks through qualitative and quantitative experiments. The results show that our method is better at transferring image semantics than state of the art image to image translation methods.

We consider the problem of building accurate three dimensional (3D) reconstructions of orchard rows. This problem arises in many applications including yield mapping and measuring traits (e.g. trunk diameters) for phenotyping. While 3D reconstructions of side views can be obtained using standard methods, merging the two side-views is difficult due to the lack of overlap between the two partial reconstructions. We present a novel method that utilizes global features to constrain the solution. Specifically, we use information from the silhouettes and the ground plane for alignment.

Estimating accurate and reliable fruit and vegetable counts from images in real-world settings such as orchards is a challenging problem that has received significant recent attention. In practice, fruits are often clustered together. Therefore, methods that only detect apples fail to offer general solutions to estimate accurate fruit counts. In this work, we formulate fruit counting from images as a multi-class classification problem and solve it by training a Convolutional Neural Network. We first evaluate the per-image accuracy of our method and compare it with a state of the art method based on Gaussian Mixture Models (GMMs) over four test datasets. Our network outperforms it in three out of four datasets with a maximum of 94% accuracy. Next, we use the method to estimate yield for two datasets for which we have ground truth. It is 96 - 97% accurate.

We consider an agricultural automation scenario where a robot, equipped with a camera mounted on a manipulator, is charged with counting the number of apples in an orchard. We focus on the subtask of planning views so as to accurately estimate the number of apples in an apple cluster. We present a method to efficiently enumerate combinatorially distinct world models and to compute the most likely model from one or more views. These are incorporated into single and multi-step planners. We evaluate these planners in simulation as well as with experiments on a real robot.

We present a fruit counting algorithm which takes segmented and registered images of apple clusters as input. It outputs number and location of individual apples in each cluster. Our primary technical contributions are a representation based on a mixture of Gaussians, and a novel selection criterion to choose the number of components in the mixture. The method is experimentally verified on four different datasets using images acquired by a vision platform mounted on an aerial robot, a ground vehicle and a hand-held device. The accuracy of the counting algorithm itself is 91% . It achieves 81–85% accuracy coupled with segmentation and registration which is significantly higher than existing image based methods.

We present computer vision algorithms to collect yield related information in an apple orchard using images collected from a single camera. The goal of our system is to give farmers the capability to use their phones or digital cameras to record images and obtain yield related parameters. There are two challenges in this setup which necessitate novel methods: (i) It is very difficult to generate dense matches using standard image features. (ii) The constrained geometry of the setup causes existing structure from motion algorithms to fail. We present a novel piece-wise incremental structure from motion technique to register and reconstruct the apples which is used for extracting count and diameter information. We validate our approach by presenting results from multiple field trials.

We present a method to construct a semantic map of an apple orchard using a LIDAR and a camera rigidly attached to each other. The system is able to capture the map as a standalone sensor which is light-weight and can be mounted on a variety of platforms. At the geometry level, we present a new method to associate image features captured by the camera with 3D points captured by the LIDAR. We then use this method to register 3D point-clouds onto a common frame. We show that our association method yields superior registration performance compared to common methods which work in indoor or urban settings. At the semantic level, the apples are identified as distinct objects. Their locations and diameters are extracted as relevant attributes. As an example, a semantic map of an orchard row is constructed.

We present a novel system for surveying apple orchards by counting apples and estimating apple diameters. Existing surveying systems resort to active sensors, or high-resolution close-up images under controlled lighting conditions. The main novelty of our system is the use of a traditional low resolution stereo-system mounted on a small aerial vehicle. Vision processing in this set up is challenging because apples occupy a small number of pixels and are often occluded by either leaves or other apples. After presenting the system setup and our view-planning methodology, we present a method to match and combine multiple views of each apple to circumvent these challenges and report results from field trials. We conclude the paper with an experimental analysis of the diameter estimation error.