ETHAN NG

Background information of the project

This project aims to investigate the application of Artificial intelligence (AI) in improving the design of cleaning robots and robot inclusive environments. We worked on training two separate AI Models to be implemented into the two robots we were working on, Panthera and hTetro. Panthera is a reconfigurable pavement sweeping robot while hTetro is a floor cleaning robot. During the project, our mentor worked together with his coworkers working on the two robots to guide us and give us more insights about their working experience.

When we joined the project, the robots' prototype have already been assembled. The task we were engaged in was to use machine learning to train the robot to identify common trash. We approached this by firstly collecting an image dataset. We then used the dataset to train a neural network to develop an AI Model. This AI Model will be able to detect common trash found on pavements and hawker centers, and classify the items to be picked up or not to be picked up by the robots. We were tasked to publish two separate journals, one on each robot and the research we conducted.

Elaboration of activities done

Firstly, we had to collect an image dataset large enough to train the robot. We did this by going out to the campus during different times of the day and taking images of natural and artificial trash, those that we collected and scattered on the pavement. We repeated this process but in hawker centers and selected the trash accordingly, to train hTetro.

After collecting the images, we used a python program, labelImg.py, to label the images into xml files. These xml files were converted to csv files to be inputted into the neural network for training. During the training process, the neural network learns to identify the images and the objects it contains, then checks if it is right or wrong. This repeated process using hundreds of different images gradually increases the accuracy of the model. Training would usually take about 8 hours, and we used the time we had to work on the journal. Sometimes the computer would run into an issue halfway through the training and we would have to restart the process.

At the beginning, the accuracy of our model was low and it incorrectly classified many objects, with a low confidence level. We realized that our mistake was not standardizing the labels for each object. Furthermore, the quality of the images taken were not ideal, this also affected the training of the neural network and lowered the accuracy. We worked on clearly setting the labels for each type of object, and went out to collect even more images. We picked the images that were of higher quality from the first time and reused them together with the new images. After retraining the model, the accuracy was much higher than before and we were satisfied.

Our mentor's coworkers helped us in writing parts of the journals, such as the Robot Architecture section which we were unfamiliar to since we worked mainly on the software. We worked on writing the introduction and experimental results. To accomplish this, we had to read up on many scientific papers and journals to conduct literature reviews. This was new to us as we were not familiar with writing scientific journals. We used Overleaf, a LaTeX editor to organise and write our journals. Learning to use Overleaf was difficult, as it was something completely new to us.

Work done and reflections

As of writing this, we are still working on publishing the first journal. We aim to complete it within 2 weeks, then take another 2 weeks to complete the second journal. We were unable to complete the journal due to time constraints. The 2 weeks will be used to conduct real-time experiments with the robots to get experimental results that can be used in the journals.

During this project, we managed to train 2 separate AI models. We went through the whole process of collecting an image dataset, labelling them, then using the labelled images to train an AI model.

Throughout the project, we learnt a lot more about machine learning and its many uses in for cleaning applications. We learnt many things that could not have been done in a classroom. The experience of going through the whole process was really refreshing and fulfilling. I felt that I gained many insights about the machine learning industry and this motivates me to learn more about robotics and its many branches.

content knowledge / skills learnt

Firstly, this project exposed me to machine learning, a topic I was unfamiliar with before. Our mentor was friendly and guided us throughout the whole journey, from the start to the end, clearing any doubts we had. He also gave us more insights about his everyday job. Although I was interested in machine learning previously, I did not really get the hang of it as it was confusing. However, this programme allowed me to learn a lot more about the topic. I learnt how computers use large image datasets fed to a neural network to train an AI model.

Secondly, I learnt how to write a scientific journal, from the structure to the contents. The process of reading other scientific journals and papers, learning how they write, what they write, opened my eyes to this new way of writing. We had to go through the process of planning the journal, conducting literature reviews for the introduction, then documenting our process of training the AI model. After that, we had to report our experimental results. All these were done with Overleaf and I learnt a lot more about Overleaf after this programme.

Lastly, I realised that the process of machine learning was not as easy as I expected it to be. After being exposed to the process where we start from scratch and collect data ourselves, I realised that the process of data collection was tedious and we had to collect data over and over again because of our mistakes. However, we learnt from our mistakes and eventually gathered a dataset of good quality which could be used in the training process. The training process was tedious as well as a single run could take 8 hours and if an issue cropped up in the midst of the training process, we had to restart the whole process, taking an even longer time. Despite all the problems we faced, we managed to produce an AI model with high accuracy that we were satisfied with.

To conclude, I am really grateful to have been given the opportunity to participate in this month long attachment programme.