Electrical Capstone Project

For my electrical capstone project, I decided to make a sensor glove that converts American Sign Language into written text. I have always been interested in sign language so I immediately wanted to complete this project, even though it seemed daunting. The website I gathered my inspiration, starting code, materials, and process outline was this site, created by Ayooluwa Olorunmola on December 15, 2022. My goals for this project were to obviously have it function, clean it up, and have updated code and completion guidelines. My swim coach, when I told him I had to miss practice to head to the lab after school to work on this project, asked me what the application of this project would be. I told him that my mom works in the emergency room as a doctor and sometimes helps patients who use ASL. They have a touchpad that the patient can type out their sentences and responses onto, however the process is long and inefficient. In a perfect world with a flawlessly working project, my glove could be used in situations like these to break this language barrier to create an easier process for both the doctor and the patient. A successful project would have individual signs that are converted into "if" statements built with ranges of bend for each finger. The microcontroller would measure the bend of each flex sensor and by fitting into the ranges on a certain sign, would output the word onto an LCD or serial monitor. 

Project Presentation

Below is the final presentation of my project. Skip past for the details of completion. 

Daily Journal

Wednesday, November 6 : Discovered project, created a rough-draft list of materials, reached out to creator for application details. I ended up creating the project without the phone application because I did not have an Android phone. Below is the picture of the project the original creator made. My goal for this project is to make it cleaner and further develope the code.

Friday, November 15: I created my bill of materials with costs and links to each component in this Google Sheet:

Monday, November 18: I created my Gantt planning chart that can be found here:

Tuesday, November 19: I continued work on understanding Gantt chart and start to develop and understand code. This is the initial code I will be developing and changing that was found on my site of inspiration from above. This day I read through the code to understand what would actually happen and how the signs were being read. I also added the ring and pinkie fingers to the initialization because in the inspiration I used, he did not use a ring or pinkie finger on his glove.

Wednesday, November 20: I started my glove hardware in Tinkercad and a site called Wikwo. These two sites will allow me to simulate my hardware before creating it in real life to limit any problems with creation. I have to use both sites for my project because all of the components I will be using are available on either site, but sometimes not both. Below is a picture of the Tinkercad portion of my project. Both Tinkercad and Wikwo are missing components of my project, so on Tinkercad I focused only on the flex sensors. The intial Tinkercad I worked on can be found here and is pictured below. 

Thursday, November 21: I looked through the lab to update my bill of materials with what materials of my project I already had. Below are pictures of the materials that I found throughout the lab for my project. 

Friday, November 22: I worked on the portion of hardware planning in Tinkercad. Below is a picture of the original creator's digital hardware planning for one of his sensors.

Monday, December 2: Two pieces of my project were delivered on this day. I ended up only needing to order the IMU, microcontroller, LCD, and glove. The IMU and microcontroller (a Seeed Studio XIAO esp32s3) were delivered on this day and pictures are attached below. I worked on planning a timeline and further developed my code. I was having a problem on Tinkercad. Whenever I would move the flex sensor, the degree of bend the flex sensor was being bent was not displayed exactly on the serial monitor. When at 0°, the valued showed 33 and when at 180°, the value showed 6. 

• Tuesday, December 3: I kept working on the problem with the display of the degrees. I moved to working with just one flex sensor, whose Tinkercad circuit can be found here, to simplify what I needed to change. I figured out how to change it so the degrees were displayed. I needed to add a mapping function that converted the values in the 33 to 6 range into values in the 180° to 0° range. However, the value that was displayed was only exact when the bend was 0° and 180°. Otherwise, the values were very far off. Below is the setup for just one flex sensor.

Wednesday, December 4: I was given permission by my teacher to ask ChatGPT to help with this problem. It ended up being that the map function that converts the measured bend resistance in my code was a linear function. It needed to be an exponential function so the values were closer to the actual bend degrees. It is still not perfect, but it is much closer and the thresholds are useable. 

Thursday, December 5: A requirement for honors students is milling a double sided board. This way, no jumper cables would be in the way and making the project look messy. I started working on creating my board for milling in KiCad. On this day I finalized what board I would be using. I knew I needed 8 analog pins - 5 for each flex sensor and 3 for the x, y, and z planes - and I also needed the SCA and SCL pins which usually double as analog pins. Because the SCA and SCL pins took up two analog pins, I struggled to find one that had enough analog pins. I landed on the Seeed Studio XIAO esp32S3. Because of its attachment, there were two extra analog pins that would make up for the loss of due because of the SCA and SCL pins. Below is its pinout. 

Friday, December 6: I worked mainly on organizing my digital portfolio. I also worked further on my KiCad board. I chose surface mount resistors for this project as they fit in best for this project. Below is my progress after these two days. I had not finalized on whether or not this was the correct footprint for the LCD, so I had to wait until that part was delivered to finalize and start connecting them. 

Monday, December 9: Today I tested to make sure my microcontroller was working by blinking the built-in LED on the microcontroller. Because I am using a seeed studio, specifically a XIAO esp32s3, I had to first download the library for the microcontroller onto the computer because it was the first time someone had used this microcontroller on that specific computer. I simply plugged the microcontroller into my computer and pressed "yes" when Arduino asked me to set it up. It took a while to load, but when it uploaded I selected my specific type of seeed and uploaded a standard blink code, that can be found in Tinkercad's presets, onto it. It uploaded and the LED on the board blinked on and off. To make sure it responded if the code was changed, as sometimes a blink code is already added to these board, I made the LED blink faster in the code and uploaded this to the microcontroller. It responded and the speed changed so I knew it would work when assembling my project. The videos below show the changes. My LCD and gloves also arrived today, pictured below, so I could check that the footprint was correct and continue on creating my board on KiCad.

Tuesday, December 10: With all of my parts in, I was able to add the correct footprints to my KiCad so I could start to connect everything together. I could not find footprints for the flex sensors, LCD, or the IMU. I realized that I would only be plugging those things in anyways, not permanently adding them into the board. I simply added surface mount connectors where I would add connectors in real life for the real components. I started connection of the pieces in the footprint editor. The connection of the pieces will allow for connections to be formed in the PCB editor later on. These are not the real connections that are on the board, it simply gives the capability for these parts to connect. 

Wednesday, December 11: I had connected all the footprints in the way the original schematic had shown, so I transferred this work over to the PCB editor of KiCad. When it loaded in, I had to add connections of the actual PCB boards. This is the wiring that is engraved into the board. As I was adding multiple connections that had crossed over each other without a problem in the footprint editor, KiCad would not let me place my connection. It turned out, crossing of wires and connections on the PCB board was a no-go. I completely panicked because as shown in the picture of my original layout before, there was a lot of crossing. I had to completely reconstruct my layout and figure out where to put the lines and how to get around the other connections. I was able to put the line that connected all of the flex sensors to power running through the flex sensor pad as this would not interfere with any connections and would not interfere with the flex sensors themselves. I ended up having to build resistor "bridges". I will use zero ohm resistors to form one pathway while another could run through the pad of that resistor bridge. Though I do actually have resistors with (10k ohm) charge in this project (horizontal on the left side of the microcontroller), I used resistors with zero ohms (vertical on the left side and all directions on the right side) to continue the flow of the board without interrupting it. The picture to the right shows this extremely confusing and painstaking work-around. Below that is the view of the PCB editor with this very fun process. This is part of what the board will look like in real-life once it is milled. 

Thursday, December 12: On this day, I realized that time was running out. My teacher changed the requirement for the honors students so we no longer had to mill a board, but still had to design it. I knew that a major goal of my project was to clean up the inspiration site that I had found, so I still wanted to mill a board regardless of how difficult mine would be. I made a deal with my teacher that if I designed the board, which would still be exceptionally challenging even by itself with the problem I encountered the day before, he would mill it for me. I continued to work on adding the resistor "bridges" for the flex sensor side and had it working, until I realized I would have to repeat this process for the other side with my LCD and IMU. I, unhappily, started working on that challenge and luckily finished it later that day. Below is a picture of the final footprint view. 

Friday, December 13: Though I had finished the footprint editor the day before, I still needed to send the updates I made on the footprints to the PCB editor. When I did this and figured out the resistor bridges in the PCB editor, I believed I was done. I was wrong. When I went to the rules checker to check that my board would be able to be milled if I had no errors, several errors appeared. A common one was that my connections that were running through other components, such as the resistor bridges, flex sensors, the LCD, or IMU, were too close and were actually touching the pads that the actual parts would touch. I simply moved them away and the errors were resolved. The most difficult error turned out to be a connection issue. The error was not detailed and said there was simply an issue with one of my resistors. I tried turning the resistor around to make sure in the footprint editor I had not connected it the opposite way. I tried reconnecting the other components to the resistor, yet nothing was working. I went back to the footprint editor and it turned out I had added a connection in between the two pads of the resistors and because I was not connecting the pads in the PCB editor, it was unfinished. I simply deleted this connection in the footprint editor and the error was resolved. I added the edges of the board by drawing a square and changing its properties to an edge cut. My board was now ready to be milled by my teacher. 

Sunday, December 15: I worked a notable amount on my digital portfolio and Gantt chart. I also worked on the code I would use for all of the flex sensors together. I created a Tinkercad circuit with the 5 flex sensors that can be found here. I ended up copying the code and adding each statement for each flex sensor finger. Once I had them all in, I pressed start simulation and all of the fingers' degree values were displayed on the serial monitor, however when there was no bend, they were each displaying 180° and when they were fully bent, they were showing 0°. I fixed this by in the mapping function, instead of having it as: 

thumbDegrees = map(thumb, minValue, maxValue, 0, 180);

I changed it to:

thumbDegrees = map(thumb, 33, 6, 0, 180);

This way, the threshold of 6 to 33 that I had found earlier in this project was translated directly into degrees. I then started crafting the "if" statements I would be using to make each sign's word or phrase appear on the monitor as opposed to just degree values. Below is a picture highlighting the issue I had and another picture of the display of the values in Tinkercad.

Monday, December 16: Today was a very busy day (as you may be able to tell from this wall of text). The board I had designed was milled by my teacher so I was able to start adding my components directly onto the board. Normally, I would test before permanently soldering the components on by assembling a breadboard. However, I was running out of time and believed it would be safest and smartest to go straight into assembly. I figured, if I'm going to short the components, I might as well do it on the real board and not the breadboard. Joking, hopefully. Before I added any components, I noticed some white spots on the copper of my board. I was worried that some of my connections were not made. I used a multimeter to conduct a continuity test to make sure they were and the sound from the multimeter ensured they were. I started to add my components by taking soldering paste and adding a dot to each pad of the board where the microcontroller would connect. This way, when the components were placed, the soldering paste would help them stick in place and would melt into solder when heat was added, connecting the two without having to angle the soldering iron and solder to get the solder into tight places. I then added all of the zero ohm resistor "bridges" using solder paste. The solder paste was especially helpful for these extremely tiny surface-mount resistors. I at first had trouble with making sure my soldering paste wasn't running off any forming connections with the copper trails with the tracks that were really close to the resistor. I had to remove a resistor and clean up the area around one of the pads. After that was fixed, it got easier to place them with experience. Every time after I placed a resistor, I used the multimeter to check the continuity of the "bridge" itself, the wires running underneath the "bridge", and the wires and the "bridge". This made sure that connections were made, connections were not broken or interrupted, and they were not formed accidentally, respectively. After the "bridges" were placed, I soldered on the 10k ohm resistors with the same method. I then added on the connectors for all of the flex sensors, the LCD, and the IMU. I used soldering paste to help them stick as well. I also took male-male jumper wires, cut off one of the male ends, stripped the wire of the end I just cut off, twisted the wires that were hanging out, soldered these hanging wires together, and then soldered the cut end of the wires to the flex sensor with a purple wire on the left and a white on the right. I repeated this process for all of the flex sensors. These male ends will allow for the flex sensors to connect into the connectors I had added onto the board. Below are photos and videos of the day's work:

Tuesday, December 17: I added male-female jumper wires to the LCD to have it connect in with the connectors on the board. I added headers onto the IMU so I could also slide this into the connector. I only soldered the top 4 pins to the header because those are the only pins I will use. I added shrink wrap to the flex sensors to prevent shorts if the two wires happened to touch. To do this, I put the tube onto the place I wanted and used a heat gun to make it flush against the wires. I added the updated code from the weekend onto the board without any components plugged in to prevent any shorts. I connected all of the electronics into the board. I ran the code and the LCD lit up, but no writing appeared on the board. I at first thought the brightness was too high and used a wrench to twist the blue box on the back of the LCD to adjust. No writing appeared then either. I continued to adjust the code and I had the esp pinout on my screen to help. As I was working, I noticed that the pinout had two pins, A4 and A5, that were labeled SDA and SCL. I realized that these were the only two pins on the esp that had I2C capabilities. The SDA pin has a built-in connection to the pinkie finger on my board and the SCL pin is not connected to anything. I had connected the SDA and SCL to pins on the opposite side of the board to pins A8 and A9 as I had thought I would be able to declare them as the SDA and SCL later in my code. This was not the case. Because the pins I had selected had no I2C capabilities, they could not communicate with my LCD even if I declared them in the code. I was too late to re-mill and re-solder a new board with these pins reconnected. In order to try and go around this problem and get some part of the project functioning, I tried forming connections with jumper wires between the designated SDA  and SCL pins and the one I had the SDA and SCL of my LCD connected to. I also had to use a knife to cut the copper connection between the pinkie and the SDA pin in order to even try to have the other fingers and the LCD functioning. I retried the code with these connections and without the pinkie finger and the LCD now showed two rows on black boxes. I ran out of time on this day and I decided to switch my focus to building my project on a breadboard to figure out the code that would get the LCD functioning. 

Wednesday, December 18: Today was the final day of assembly for our projects. I focused on creating my electronics into a breadboard to try and at least get one flex sensor to output values onto the LCD. I soldered headers onto the esp 32s3 so that female jumper wires could be added. I then set up the LCD, microcontroller, IMU, and only one flex sensor onto a bread board. I then uploaded my code for one flex sensor onto the board. The LCD would light up, but no words or values would show. From here, I took out the flex sensor and the wiring for it. I just wanted to focus on finding the code that would send information to the LCD. I found sites that had information on the LCD I was using, yet all of them used an arduino microcontroller. When I used these codes, they would not work. I then found this site that used an esp 32s3. I deleted all the code and first used this set of code to find what I2C address my specific LCD was. The serial monitor followed this code and told me that its address was (0x27). The serial monitor also told me that there were 2 I2C devices found and the other's was (0x68). I was confused, until I realized that my IMU is and I2C device. I disconnected the IMU to figure out what the address of the LCD was and to not complicate the process. I ran the code again and it showed that the one device's address was (0x27). I then followed the rest of the website and used this next set of code with this specific address that would ideally make the LCD output "Hello World! LCD Tutorial". I uploaded this and I could see through the camera screen that the words were being added, but the black behind the letters was making it hard to see. I adjusted the brightness like I did before. I had found how to connect the LCD to the microcontroller. I used elements from this code and added them into the one flex sensor code. At first, nothing was changing and the LCD did not respond to the change. I kept searching for answers online and found one piece of information to add the line "Wire.begin(D4, D5)". D4 and D5 are the SDA and SCL pins, respectively. I thought that adding this would help to initialize which pins were the I2C pins. Becuase they were written as digital pins, I changed the initialization of my flex sensor pin to a digital as well. I uploaded this code and the LCD outputted: "T: 0". This was a huge step. Although it did not respond or change the value when I bent the flex sensor, I still was able to figure out how to get the LCD to respond to code and a change in code. This is the final code that I used to make this happen. 

Thursday, December 19: I presented my project to the class. Below is the slides I used to present what I had of the project. 

Final Outcomes

Originally, I had not expected this project to be easy. I was correct. There was a multitude of things to accomplish: understand the original code, simulate it, develop the code for one sensor, all sensors, develop a board, mill the board, connect electronics, adjust code, add the entire sign language! Ambitious is an understatement. I unded up not being able to output all the values at once. I was able to have one value posted on the LCD. I did not end up with a glove that could translate sign language at all. Although I did not produce what I had wanted to, I talked with my teacher about possibly finishing and upgrading this project as my capstone for next semester. So, stay tuned! I believe that now that I understand the ins and outs of this project, I understand what has to be completed to have my project working fully, I have a KiCad board that I can edit with the pin change, and I know how to connect to the LCD, I can definitely perfect this project if given more time. I know this was an ambitious project to take on in this little time, but I found it extremely interesting and I still do not want to give up on it.