Development of Planted Seedling Area Recognition Algorithm for Vision-based Autonomous Driving Rice Transplanter Development

1. Motivation

  • As the agricultural population is gradually decreasing, it is required to develop autonomous driving rice transplanters to cope with the aging problem.

  • The existing GPS based self-driving rice transplanters sometimes invades the area of seedlings that has already been planted.

  • Therefore, it is necessary to develop an algorithm that recognizes the area of seedlings from images and controls the heading direction of the rice transplanter.

2. Goal

  • Develop a deep learning algorithm that recognizes the area of planted seedlings by attaching an RGB camera to a rice transplanter.

  • Find the boundary between the planted area and the non-planted area.

  • With the detected boundary line, the autonomous driving rice transplanter is controlled so that it does not invade the planted area.g

(a)

(b)

(c)

Figure 1. (a) Showing a planted area, (b) an unplanted area, (c) a boundary line between the two areas

3. Method

3-1. Dataset

  • When planting with a rice transplanter, we acquired the planting images using an RGB camera. Figure 2 below is the dataset we used to train the deep neural network.

Figure 2. Dataset used for algorithm development

3-2. Data Augmentation

  • The amount of acquired images is not enough to train the network because the experiment environment is limited.

  • Therefore, we used image augmentation to increase the amount of training data.

  • A python augmentation library called Albumentation is used for our research.

  • Figure 3 below is an example of image augmentation.

Figure 3. Image augmentation example

3-3. Seedling Area Recognition Network Using U-Net

  • We decided to use U-Net, which is convenient to use and well known for high performance in the image segmentation field.

  • After receiving the image and passing through the U-Net, the network parameters were learned through binary cross entropy loss.

  • Figure 4 shows the flow chart for learning the seedling area recognition network.

Figure 4. Flowchart for training seedling area recognition network and structure of U-Net

  • The existing U-Net uses batch normalization (BN) for all convolution layers for efficient batch training.

  • However, we used domain normalization (DN) to reduce the domain gap that occurs because the environment is different for each rice paddy field.

Batch Normalization : A method of normalizing data by Equation (1) using the mean

and standard deviation of all pixel values of N data in one batch


(1)

Figure 5. Area to which Batch Normalization is applied

Domain Normalization : A method of normalizing data by Equation (2) using the mean

and standard deviation of all pixel values in one image



(2)

Figure 6. Area to which Domain Normalization is applied

3-4. Result

Figure 7. Loss curve when using BN

Figure 8. Loss curve when using DN

Figure 9. Area prediction results using BN

Figure 10. Area prediction results using DN

  • When we used DN, we were able to predict the area of the seedling better than BN.

  • Through these results, we were able to prove the assumption that the neural network can recognize the area of the seedling.

4. Implementation

  • As shown in Figure 11, after directly acquiring data, we proceeded with deep neural network training.

Figure 11. Image obtained by attaching a camera to a rice transplanter

  • After obtaining the result of the trained network as shown in Figure 12, we found the boundary line between the planted area and the non-planted area through image processing.

Figure 12. Result of recognizing the area of seedlings

  • You can check the loss curve that occurred when training the deep learning algorithm in Figure 13 below.

Figure 13. Train loss and validation loss curve generated when training a deep learning algorithm

※ Terms for validating the performance of the model

  • TP : If the actual label is True, but the prediction is True

  • FN : If the actual label is True, but the prediction is False

  • TN : If the actual label is False, but the prediction is False

  • FP : If the actual label is False, but the prediction is True

  • In order to verify the performance of the algorithm, we obtained accuracy, recall, and precision.

Proportion of values that the algorithm correctly predicted labels

among real data

Proportion of values predicted by the algorithm to be True

among values whose actual label is True

Proportion of values whose actual label is True

among the values predicted by the algorithm to be True




Accuracy = 0.974




Recall = 0.946




Precision = 0.985

5. Future work

  • The task of finding a boundary line between the planted seedling area and the non-planted area was completed.

  • Therefore, we plan to configure the controller so that the rice transplanter does not invade the planted seedling area while autonomously driving along the boundary line.

# Photo at work