SHI QILUN

Background Information of our project

We were assigned the task to work on the software of two different floor cleaning robots — Panthera and hTetro, by working on the object classification of their cameras. After that, we need to record our experimental results in the form of a conference paper (it was later changed to a paper for a scientific journal publication).

Panthera is a self-reconfiguring pavement sweeping robot that is able to reconfigure its width in response to pedestrian density and pavement width. Currently, it is able to respond to pedestrians via a separate machine learning algorithm and hence contract to allow them to pass. Specific to Panthera, we needed to clearly define the cleanliness threshold of the pavement. In other words, we needed to define trash items commonly found on pavements into 2 classes: “Pick Up” and “No Pick Up” This was to ensure that the robot is able to recognise small garbage to clean and ignore other things like road signs and cracks.

hTetro is a floor cleaning robot in 4 separate square modules that can reconfigure itself into shapes that resembles Tetrominos in the game of Tetris. It is currently able to reconfigure; manoeuvre and calculate the size of the area it needs to clean and the appropriate shape for it to shift into. Our task is to train the algorithm to detect trash items that cannot be cleaned by hTetro so that the robot can avoid them, preventing damage to the robot and time waste. We defined 2 classes named “Can Clean” and “Cannot Clean” for this purpose.

I helped in the photo taking of pictures for both classes of pictures for the training of Panthera’s AI as well as pictures for both classes of pictures for the training of hTetro’s AI.

Elaboration of Activities Done and Deliverables

To complete both tasks, we took photos of trash and non-trash items and used LabelImg software to label them for the two separate machine learning algorithms to train on. In addition to that, we assisted in writing scientific papers on https://www.overleaf.com. In total, we worked on 2 scientific journal publications with assistance from mentors Mr Balakrishnan (we address him Mr Bala) and Dr Mohan.

Since there is hardly any trash items found in SUTD, sometimes we needed to bring trash items from home and lay them on the ground to mimic trash items that might be found in real environments.

The task of labelling was split among us. I labelled over 500 photos in total. During labelling, I drew a box that needed to circumscribe the item as much as possible while ensuring that there was no overlap of boxes (to avoid this, each trash item was placed slightly further apart from one another, or we took pictures containing a single trash item only.)

In addition to the work mentioned above, I also assisted in writing the literature review for both papers. Below are some of the links I used to do literature review.

The Panthera paper:

• https://ieeexplore.ieee.org/abstract/document/8793975
• https://deepai.org/publication/robotic-detection-of-marine-litter-using-deep-visual-detection-models

The hTetro paper:

• https://www.hindawi.com/journals/js/2018/3035128/

After doing the literature reviews, I found out that most forms of the current algorithms used in the cleaning robots only identify the trash items found on the floor but not classify them, which leads to inefficiency because the robot may spend a long time trying to remove a piece of garbage that is too big to be picked. That was the significance of our work on object classification.

3 Content Knowledge/Skills Learnt

Problem-solving skills

During the attachment, we were given 2 tasks to resolve. As mentioned, we need to gather datasets (photos of trash), process the data (label trash items in each image as “Pick Up” and “No Pick Up”) and eventually sent them to training for the robots’ AI to recognise them.

But problems were encountered as we progressed. Since we did not know any requirements of image labelling, our labelling of trash items were not standardised. For example, one of us labelled a few items close together in one box, while another drew one box for each item. Furthermore, we did not see the importance of taking pictures of one object from different angles and backgrounds. Hence, when we sent the processed images to training, the recognising accuracy of the robots’ AI was very low. The AI even failed to recognise the items in certain images occasionally. To tackle such problems, we standardised our image labelling by circumscribing the item as much as possible when drawing the box. Meanwhile, we began to take pictures of the same objects from multiple angles. As a result, the recognising accuracy of the AI improved with the new datasets.

From this experience, we learnt the steps of problem-solving. We identified the problem first, then we thought of ways to resolve it. In this case, we made changes to the data collection and processing, which improved the outcome (i.e. recognising accuracy of the robots’ AI).

Writing skills for scientific reports - formal writing style

After training the robots’ AI to recognise trash, the next step was to document this process and contribute our results to other researchers who may benefit from our research. However, we needed to learn the style of writing a scientific report, which is a very formal setting. We were told to do a literature review to find out previous work done about the use of AI in the cleaning sector, in which we learnt that robots were used in a variety of settings, from marine cleaning to grass patch cleaning in parks.

We also learnt to include flowcharts and diagrams to show the process of trash detection of the algorithm of the robots. Fortunately, we had a tool named https://www.overleaf.com to help us. We gained insights on the way of writing a scientific report from this experience.

Communication skills

The process of data collection (i.e. photo taking) might sound simple, but it was very intimidating because we were aiming for gathering, numbering and labelling about 1900 images within 10 days. Any miscommunication during this period could affect the data we sent to the mentor and eventually, the training results. Hence we must communicate clearly on the standardisation of labels, the classification of items in the photos as “Pick Up” and “No Pick Up” or “Can Clean” and “Cannot Clean”, as well as the numbering of images. For example, “Pick Up” and “Pick up” had the same spelling but they were different classes because of the capitalisation. I realised that clear and effective communications were crucial to the completion such a daunting task.

2 Interesting aspects of learning and 1 Takeaway for Life

One interesting aspect of my learning was that I learnt that it is impossible to achieve an AI model of 100% accuracy. Although we trained the model with hundreds of images, the model did not achieve the accuracy we had expected. However, after repeated trial and error, collecting a larger dataset, we managed to improve the accuracy of our model and we were satisfied with the results.

This was quite different from my expectation as I thought that AIs are able to complete many complex tasks with superhuman accuracy from the reports we read. However, we did not know that the training process was a whole different story. We had to feed the AI with the large datasets of well-taken photos in order to improve its accuracy of detection when placed in a real environment. Furthermore, the labelling must circumscribe the target items as much as possible to avoid false detection from the AI. Overall, the training process was way more meticulous than what I assumed to be

Another interesting aspect was that we learnt that was the degree of variation required in our photos. Mr Bala told us to take the same items from 10 different angles and vary the brightness of the background. Doing so would improve the detection accuracy in the training of the robots’ AI. Interestingly, if we take duplicates (taking multiple snapshots without changing anything), the detection accuracy will not improve. This showed us the importance of variation.

My takeaway from the attachment to SUTD was that I should persevere in face of difficulties. Our mentors gave us sites that showed us advanced algorithms used in object detection trainings for AIs and told us to read and understand as much as possible. I found them difficult because I only possessed basic programming skills before going for this attachment. The papers that we used for literature reviews also contained advanced algorithms and mathematical equations used by the researchers to develop the AI. Even though I could not comprehend them in the beginning, I tried my best to read on and figure out. Eventually, I managed to understand better. Had I chosen to give up earlier on, I would not have learnt anything. This is like situations in life where you must overcome the obstacle. We should have the mental strength to push on and try our best to overcome them instead of trying to escape and hide from it.