Raspberry Pi Neural Network

Project Ideas

Raspberry Pi Music Sensor: set to play a randomized song each time you enter a room. https://www.makeuseof.com/tag/play-theme-tune-enter-room-raspberry-pi/

Worries:

• not a very complex design and would not take a significant amount of time/effort

• likely would only be used for a few weeks until put on a shelf

Most challenging/exciting part:

• Someone else walking into a room for the first time and hearing music welcome without seeing the sensor or knowing where the music is coming from

Raspberry Pi Magic Mirror: uses an old laptop screen to create a "mirror" that can display information such as the weather that day. https://www.makeuseof.com/tag/old-laptop-screen-magic-mirror/

Worries:

• would need a old laptop screen

• slightly outdated project

• expensive (~$100)

• less electronics work and more woodworking

Most challenging/exciting part:

• Placing the laptop screen into the wooden frame I built for the screen and turning the project on for the first time. Seeing all my hard work fit together perfectly and accomplishing what it was meant to do by displaying the information I had coded it to present.

Raspberry Pi Neural Network: AI image recognition box that uses a Pi Camera to take a picture of an object and then a neural network to identify the object. https://hackaday.com/2017/06/14/diy-raspberry-neural-network-sees-all-recognizes-some/

Worries:

• slightly outdated project (on Raspberry Pi 3)

• I have had issues in the past downloading from the internet onto a Pi Board (see 11th Grade, Food Computer)

• expensive if parts cannot be gotten from the FabLab

Most challenging/exciting part:

• The first time my project successfully recognizes an everyday object and announces its success to the world.

Project Choice

I chose to build the Raspberry Pi Neural Network (https://hackaday.com/2017/06/14/diy-raspberry-neural-network-sees-all-recognizes-some/). A successful project will fit neatly together in a box with the electronic components, including a portal charger, inside allowing for a mobile image recognition machine. Upon pressing a momentary switch, the Pi Camera will take a picture and the neural network will work to recognize the image, which will then be announced by the speaker. I chose this project because it is challenging and will push and develop my electronics skill. Neural networks have always interested me as I believe AI already is and will continue to be a core component of the modern world. Until now, I have not been able to pursue AI and the personal project feels like a perfect opportunity to explore this fascinating concept while simultaneously working with a component I'm familiar with in the Raspberry Pi.

My project will be different from the source project due to its design and components. Visually, I hope to have my project more appealing by not placing it in a plastic tupperware. I plan to use our woodworking machines to create a box that will perfect fit my components and position the camera in a stable place instead of allowing it to flop around. I will also be using a Raspberry Pi 4 board instead of a 3 so ideally my project will work even faster than the ~10 seconds it took for the original to recognize an image.

Tools

I will use the CNC machine at the FabLab to create a wooden box for my components. Then I will either hand-sand or use the sanding machine to give the box a polished look and feel. Once I have completed by project and can accurately measure the volume needed to contain my parts, I will create a file to cut on the CNC machine.

Components

Final Materials List

Daily Journal

Project Management

While I did not create a concrete plan for myself, I believe I did a good job staying on schedule. Each Monday I mapped out for the week what I wanted to accomplish. The majority of my work at home was on the neural network software on the Pi at my PC. My work in the Fablab centered on assembling the box and making sure the parts would fit together safely.

It took me longer than I wanted initially to get Google's sample network up and running. Soon after I got it running we began fully in-person school which further limited the time at home I had to work on training my own network. Once I realized that using my AMD card would be unsafe I did not have enough time to order or find a cheap Nvidia card to train the network. With more time I would've certainly found a old Nvidia card to train the network because I already did the grunt work of taking the pictures.

Tools

I used the table saw to create my box. Mr. Dubick and Dr. Taylor assisted me with using the clamps to ensure consistent cuts for the different sides of my box. I had been planning to make a tab box but Dr. Taylor showed me rabbit joints, which would be much stronger. I looked at examples of these joints around the lab and made them for my box. I have not had any concerns about the structural integrity of my box as a result. I used wood glue to hold the box together. I then used the sanding machine as well as individual pieces of sand paper to smooth all the edges of the box to make it easier to hold and carry around. I used a 3D-printed enclosure built for a PiCamera to hold the camera stable and safe inside my box. I taped this enclosure to the inside of the front panel of the box. Finally, I used the hand drills to create the holes in my box to allow the cables to come through the box seamlessly. I wish I was more precise on this last part because the holes do look rough and significantly less uniform than the rest of the box. Nevertheless, they serve their intended purpose. Lastly, I used the drill to create a hole in the front of my box for the camera.

Video of using table saw to make initial cut

Instructions

Much of these instructions follow https://github.com/EdjeElectronics/TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi/blob/master/Raspberry_Pi_Guide.md.

Deviations will be starred. Example: 7* means this step is not included or is changed from tutorial

1*. Format your SD card using https://www.sdcard.org/downloads/formatter_4/index.html

2*. Plug in ethernet, HDMI, SD card, keyboard, mouse, and PiCamera to Raspberry Pi. Then connect power source to an outlet and HDMI to a monitor. Turn on the Pi.

3. Update the Pi in the terminal by using:

sudo apt-get update

sudo apt-get dist-upgrade

Then ensure the camera is enabled in the Raspberry Pi configuration menu.

4. Download GitHub repository with TensorFlow Lite scripts from https://github.com/EdjeElectronics/TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi.git and, for easier use, rename the folder it creates to tflite1 with commands:

mv TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi tflite1

cd tflite1

5. Install virtual environment for TensorFlow using:

sudo pip3 install virtualenv

and

python3 -m venv tflite1-env

6. Turn on the environment using:

source tflite1-env/bin/activate

6a*. Any time you want to turn on the neural network you will first need to issue:

cd tflite1

to be inside the folder containing the neural network followed by (from step 6):

source tflite1-env/bin/activate

to make sure you are inside the virtual environment where TensorFlow is situated.

7. Install TensorFlow, OpenCV, and the dependencies needed for both.

The creator of the github repository created a script that downloads the packages and dependencies to make it much easier:

bash get_pi_requirements.sh

8. Set up TensorFlow Lite detection model

8a*. Here is where I used Google's neural network initially and then later went back and began the process of training my own network. These instructions will detail Google's network, but both options are possible at this point.

To download Google's network, issue:

wget https://storage.googleapis.com/download.tensorflow.org/models/tflite/coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip

and unzip the file with

unzip coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip -d Sample_TFLite_model

9. Finally, ensure your camera is plugged in and in the terminal issue:

python3 TFLite_detection_webcam.py --modeldir=Sample_TFLite_model

This runs the webcam script and should make a screen pop up with the neural network successfully running.

Note: All the steps that follow are no longer part of the Github guide

10*. To create the box first place the Pi on a plank of wood and trace around the Pi the dimensions you want for the box. I recommend leaving some space inside the box for the camera and wires instead of a tight fit.

11*. Use a table saw to cut out the base for the box.

12*. To create the rabbit joints draw a line down the 2 longer sides and the back of the base about 1/2 of an inch from the outside. Lower the saw down to roughly 3/8 of an inch and cut from the line drawn to the edge of the base. You are not trying to cut through the wood, only to reduce the thickness in this precise area. This will likely take more than one cut. Repeat for both sides on the same side of the wood.

13*. Mark and cut out two identical sides for the box. Their length should be equal to the length of the base. The height of the box is a personal preference, just ensure enough space will be left for the camera in the front.

14*. For the sliding top, raise the saw to 1/2 of an inch and mark out a rectangle at the top of each side. The rectangle should be the length of the box and about the width of the saw wide. Draw the top of the rectangle starting about half an inch below the top of the sides. Then cut out the rectangle.

15*. For the back of the box the dimensions should already be determined. The height is the height of the sides and the width is the width of the base. Cut out the back.

16*. Measure the width of the sides. On the back of the box draw a line along the sides and bottom this width away from the edge. Lower the saw to 3/8 of an inch again and cut the same way as in step 12 for both sides and the bottom of the back.

17*. At this point you should have 4 pieces of wood cut out (base, back, and two sides). Put together these pieces to get a sense of how they will fit together and align the rabbit joints to maximize points of contact. I recommend first attaching the back of the box to the base, then moving to the sides. Place wood glue along one side of each contact point then hold the pieces in place together for roughly 10 minutes or until the pieces are stuck together and you cannot wiggle them without excess force.

18*. For the top of the box try to find a thicker piece of wood. Measure the length of the base of the box and width of the box and cut this piece of wood to fit those dimensions. Note: it's better to start out too big and then cut smaller than having to find a new piece of wood because you cut too much initially.

19*. To make the top slidable, measure the width of the rectangle cut out on the sides of the box. On the longer sides of the top, mark slightly more than this width down from the top. For the depth of the cut, I recommend starting with 1/4 inch and then cutting more if necessary. This should leave a significant area that needs to be cut out. Make this cut.

20*. The top of the box should now slide in and out of the rectangles cut in the sides. To create a level surface on top, either sand or use a saw to cut out any excess. Do not wood glue the top onto the rest of the box because it will no longer slide and allow access to the electronics inside.

21*. For the front of the box use a thinner piece of wood. Measure the remaining width of the base for the width of the front. For the height make sure some space is left to allow the user to pull the sliding top on and off. Cut out the front on the table saw and wood glue it to the front of the box.

22.* Place the Raspberry Pi inside of the box and orient it so the PiCamera will face the front of the box. On the side of the box where the power source and HDMI cable will be plugged in drill out two oval holes wide enough for the cables to fit through. You do not need to drill holes for the keyboard, mouse, or ethernet. These can simply be attached through the open top and then unplugged once they are used to turn the network on.

23*. Optional: 3D-print out a case for the PiCamera. Place the Camera inside the case

24*. On the front of the box, drill a hole the radius of the PiCamera regardless of if you are using the case. Then place the PiCamera inside the box up to the front with the camera inside of the hole. Run the webcam script from step 8 to ensure the camera's visibility is unimpeded. Take the camera/case to the inside of the front of the box and ensure it does not wiggle around as this will block the camera's view.

25*. Plug in all components except the ethernet. Run:

cd tflite1

source tflite1-env/bin/activate

python3 TFLite_detection_webcam.py --modeldir=Sample_TFLite_model

Unplug the keyboard and mouse.

Congratulations! You have created a portable image recognition neural network.

Finished Project Working

Challenges

Installing the Pi Camera was a challenge as I assumed I knew how to physically put the camera into the board. However, I was not lifting up the flaps on the sides which made it impossible. I found a video on Youtube that described the process and walked through the code of installing the camera onto the board. I then did so on my PC. This experience was frustrating as this was not a complicated step, but I learned to be patient and look for help if necessary.

The button I have from the lab cannot currently connect to one of the Pi's GPIO pins as intended. I will either use a breadboard to connect the two for now, or look for an alternate switch or button that will function better.

However, the main issue I ran into was with training my own neural network. The safest and proven way to train a neural network involves taking thousands of pictures and using an Nvidia graphics card to train the network. I took over 2,000 pictures of cards, but I have an AMD graphics card on my PC at home. There is an experimental version that allows an AMD card to train a network but I do not trust the software at this time. Therefore, I stuck with Google's network after about 5 hours of research. My project showed that a complete, stable, independent box can be used for image recognition utilizing a neural network.

Final Summary

I chose to create an image recognition neural network because the concept of using AI seemed interesting and the end result would be a project that took complex work, but was easy to understand. This was my first experience working with AI, and I really enjoyed the process. Similarly to my previous engineering projects, the lesson that the last 10% of the project takes half the time was reinforced.

This project, combined with the hybrid nature of school this year, taught me the importance of setting goals throughout a project. Having a class fly by with minimal progress was much easier this year compared to prior projects. By planning out on Monday what I wanted to have done by the next Monday, I stayed on track to complete my project. Similarly, before I had decided on the physical portion of my project, many in-person days passed with me only researching and waiting for a remote day to continue working on my network. I believe my wooden box could have been more polished if I had better taken advantage of the in-person days earlier in the year.

As this project was my first experience with AI, I learned a lot about both TensorFlow Lite and neural networks in general. The process of creating a class list, taking and labeling thousands of pictures, and then allowing my GPU to train the network was much more involved than I had believed it would be. I was under the assumption that it might only take between 10-20 pictures per object to create a passable neural network. I could not have been more incorrect. The roughly two thousand pictures I took were time-consuming but taught me about the effort required to create a valuable piece of technology. Vincent confirmed the time-consuming nature when he told me that it took him nearly 20,000 pictures just to distinguish between cats and dogs. While I did not fully train my own network, I completed much of the process towards doing so. With more time the very first thing I plan on doing is finding a Nvidia graphics card as cheap as I can because my AMD can only train a network on experimental software that I do not trust on my PC. With many of the necessary files created and software downloaded the final steps towards training my own network would not be that complex. While my network would not be able to recognize the nearly 100 objects that Google's can, it would provide a more realistic example for what a PC and single individual can create.

While downloading the neural network was largely following an already defined process with only a couple deviations, creating the wooden box with its slidable top was the unique part of my project. I'm glad I did not end up making a simple wooden tab box as I had planned on making for a couple days. The rabbit joints took several days to make because I had never made them before and just went off other examples in the lab, rather than a guide. The most challenging part was getting the height of the saw in a constant, yet precise height to make sure the box was uniform and sturdy. Adding the slidable top to allow access to the electronics was also difficult, but makes the final project much better and easier to use compared to an open or permanently closed top. I figured that creating the box would be the very end of my project and not take very long, but the weeks I spent creating the box proved otherwise.

My senior project, with Raspberry Pi software, woodworking tools in the lab, and coding combined several of the tools I have spent my last 5 years at Latin learning how to use. I'm glad to have created a successful, complex project to cap off my time engineering at Latin. One of my favorite parts of this project is that it is never "complete" because the network can only be improved by adding more pictures and allowing it to "learn" more. Going forward I plan to complete training my own network with a Nvidia graphics card and then continuously adding more pictures of cards to create the best network I can. Although not a noble purpose, I hope to give my network the capability to count cards and keep a tally in blackjack to allow for optimal play. While this may be far out, the technology is rapidly improving and only getting smaller so that one day it may fit in a pair of glasses and be accessible for everyone.