Augmentative & Alternative Communication Device
By The Hickories
Final Documentation
Team Hickories: Aidan, Ella and Gemma
Client: Maggie
Project 3, or “a useful implement for a person living with a disability” had our group of three working with a client from the organization CLASS (Community Living and Support Services) in order to make an assistive device that was uniquely tailored to them and their needs. This project spanned about a quarter’s worth of weeks and took many different iterations and ideas to reach where it ended up. We had an initial interview consultation with our client Maggie, a second interview follow up during out ideation phase, a prototype critique and a final critique - all of which helped us flesh out what we were making and how we were making it. For this project, Team Hickories made an Augmentative & Alternative Communication Device.
For more information on our initial interview process, click here.
Team Hickories and our client Maggie at the prototype critique session!
What We Built
The transmitter box (left) and receiver box (right) next to each other.
For this project, we built an Augmentative & Alternative Communication Device for Maggie. The function of this device is relatively simple. The polygonal transmitter box has one button on each side, and each button transmits a different message to the receiver box. The message that each button sends is indicated by a laser etched message printed on each respective side. The receiver box receives the signal and displays the message on an E-Ink display, also lighting up a light strip on top to green to indicate a new message. When the person reading the message has read it, they click a button on the top and the message and light strip clears.
Receiver Box
Transmitter Box
A front view of the final receiver box.
Click through to see all five sides and the top/bottom of the transmitter box.
Devices in Action
A full "cycle" of use for the devices - a button is pressed on the transmitter and received/read on the receiver.
All of the different buttons and their corresponding messages in action.
Narrative Description
Maggie is sitting in the computer lab at CLASS and is notified that its time to move to a different space; she is almost done with the art she is working on, and leaving the space right now feels overwhelming, but communicating this feels hard. She pulled out the device and sets the receiver near her on the desk so the CLASS individual can see the screen, and presses the "Can I have more time?" button. They tell her she can have 5 more minutes and she spends that time preparing to move to the next space, feeling comfortable and calm having communicated her needs.
How We Got Here (Prototype and Process)
Prototype
This prototype was built to primarily answer the following design question:
How does the user respond to the size, shape, and input types?
The prototype we built for the critique was firmly focused on giving Maggie as many options on tactile inputs, shape and form as possible while still being manageable. We built three boxes out of two kinds of material (a cardboard-like material and clear acrylic). These three boxes were varied in size, shape and number of sides. We also embedded a number of different kinds of buttons, switches and sliders. Additionally, we brought a number of index cards to the critique along with an LED hooked up to a wireless transceiver which communicated with a button in order to model what the function of the box would look like. After discussing with Maggie, we put the most important phrases to her on the cards.
All of our different transmitter enclosure options we presented to Maggie in order to find the best size and shape.
A close up of one of our boxes with the different types of inputs (buttons, sliders, etc) - we went through all of these inputs with Maggie to see what her preference was.
Flash cards we wrote during the critique session after discussing different message ideas with Maggie to simulate what pressing a button would do in our final device.
A video demo of the LED/button combination we used to demonstrate what the practical functionality of the boxes would be.
Prototype Ideation and Fabrication Process
Original Ideation Sketches (for Prototype)
The original sketches of our first ideation of the design of the device - in this version there was only one box with a message.
Prototype Brainstorming and Fabrication
A photo of the brainstorming we did on a whiteboard table during one of our prototyping sessions.
A "failed" laser cut where the settings we used didn't quite cut through the scrap we used. We ended up salvaging this cut for one of the prototype boxes.
The first laser cut we did of the polygon shaped transmitter prototype.
Initial troubleshooting and working to understand with the wireless transceiver.
Prototype Findings
Our guiding question for our prototype, "How does the user respond to the size, shape, and input types?" was answered through our prototype critique with Maggie. We identified which buttons, switches, and sliders she preferred (she liked the feel of the momentary switches) and which side and shape of the boxes she liked. She preferred the cube and the pentagon, and when it came down to deciding between the two, she liked the idea of the pentagon fitting more of her favorite characters and more messages and buttons than the cube would. Having the physical prototype there for her to touch and feel and hold was very helpful in this process, and once we had answered our guiding question, we still had a bit of time left. With this extra time, we talked about her favorite TV show and game characters, and how we could incorporate these characters as well as different colors into the final project design.
Feedback that we integrated into our final design included using the buttons that Maggie liked the best, shaping the receiver and the transmitter boxes in the shapes that she preferred, and putting her favorite colors and characters into the aesthetic design of the project. We also spent a lot of time during the prototype critique discussing the most important messages that should go on the box. We made a list of messages with Maggie and then narrowed it down with her to the most important 5 messages, which were the ones that ended up on the project.
The only feedback from the critique that we received that we didn't incorporate into the final design was a plethora of favorite characters and messages that simply didn't fit on our design, since an important aspect of the design was it being portable. We worked through the long list of characters and messages and narrowed it down with Maggie to just 5.
Surprises that we encountered during the prototyping process included having a hard time getting firm answers from Maggie on some topics. We worked through different ways of framing our discussions to get the most helpful feedback out of our critique, but we are glad that we had a long period of time for the critique to ensure we got the information we needed from her.
Overall Process
Code and Wiring
The first button rig made to test the status of the code and circuit consisting of the wireless transceivers and the e-ink display with the buttons.
The E-ink display in a particularly chaotic moment of update problems.
Wiring the soldered components of the receiver box right before final assembly.
Final Enclosure Fabrication
The laser cutter cutting and engraving one of the sides and the top of the transmitter box enclosure.
All of the sides of the receiver box laser cut out.
All of our transmitter box side panels post-first acrylic paint wipe.
Soldering and Final Assembly
The beginning of soldering of the the proto-board used for the buttons and adjacent components in the transmitter box.
All of the wires and electronic components for the transmitter fitting into the receiver box.
Scheduling/Timeline Reflections
The Gantt Chart we made to plan out our project timeline post-Fall Break.
We ended up sticking pretty close to this Gantt chart we made planning our project timeline - partly out of pre-scheduled necessity (one of our group members was out of town for some of the last week). Some of the final assembly occurred later than intended but other than that everyone stayed mostly on schedule. This was due to us having a realistic conversation about timing and other obligations while making the schedule - trying to front load the work as much as possible ended up being extremely helpful due to everyone having finals and other obligations that would end up taking up a lot of time.
Overall Thoughts/Reflections
Overall, our project went pretty smoothly. Our decisions were made as a team and heavily influenced by Maggie's input, and we stuck pretty closely to our plan from the time of prototype critique to final critique. A decision that we made was to make a prototype that purely gauged sensory choices, which meant that our prototyping took care of most of the major decisions that we needed to make. This allowed us to really "head down" work from that point forward, which was helpful; if we had prototyped the code, for example, we likely would have needed to make many more design choices on the road to the final critique without Maggie's input.
Breakthrough moments included getting the transceiver to work and getting the e-ink display to work. These were two aspects of the project that gave us some trouble along the way, but nothing we couldn't handle! We didn't have any major breakdown moments, thankfully, but we did have some minor frustrations with getting the laser to engrave the images facing the correct way. Once we got those settings down, we were all good to go!
Conclusions and Lessons Learned
We learned a lot of things along the way to completing our final project. Some of the things we learned came from our final critique, where we presented our project to our client, the class, and a handful of guest critiquers. One piece of feedback we received about our transmitter was “Needs softer corners.” The decision for the transmitter to have a very clean acrylic fabrication was deliberate to avoid some of Maggie’s sensory needs; she doesn’t like certain textures, and we knew that acrylic was a safe one and didn’t want to choose something softer without knowing if she would like it or not. Another piece of feedback we got was “Easier to rotate transmitter (sliders on bottom),” which is a small detail that we hadn’t though of that would make the project much better through a small change. Small fabric “feet” on the base of the transmitter would make this project much better at a super tiny cost of effort and money – since it’s mostly designed for on-table use, having little sliders on the base of both the transmitter and receiver makes sense. We also got the feedback of “Allow for the pressing of multiple buttons. Allow for “unpressing” of buttons.” Pressing multiple buttons and un-pressing buttons would be a great addition to the project, and would just be some pretty easy code changes to implement! “I wish both devices had LED lights to indicate the message sent/received and I with the functional buttons didn't look so similar to the fidget buttons” was another piece of feedback that we got; LEDs on the transmitter to show that a message was received was part of our original discussion, but this detail got lost along the way. It would be a great addition to the project and wouldn’t be that difficult to implement! Lastly, we got the feedback “See if you can shrink the form and make it more portable.” Ideally, we would love to be able to do this, but the project was already such a tight fit of wires and protoboards on the inside that, as the wiring currently stands, making the form smaller is pretty out of the question unless we do some major design changes.
We had a few takeaways from the experience of working with a person with a disability and designing something just for them. Overall, the process was super fulfilling; the concept of doing a project that might actually help someone in their day to day life is much more compelling than many projects we face in college, which are simply for a grade. Our main struggle in the process was communication; before we got to know each other, it was hard to get our client out of their shell. It was very fulfilling once we got closer to her and she felt comfortable getting excited around us, such as at our prototype critique when she got excited about choosing themes and talking about her favorite shows and characters. Something we would do differently next time is come more prepared for the client, like maybe talk to our contact beforehand to understand what their specific struggles are so that we go in with some initial ideas, or at least prepared with tactics to best communicate with them in the initial interview.
Overall, this has been a very fulfilling experience for all of us. The process definitely emphasized to us that soldering and fabrication always take longer than anticipated, so that is an element of the project that should be started as soon as possible. We started the fabrication process as soon as we started the project, and we still had a little bit of crunch time at the end. Something unfortunate that happened is our battery cases were somehow wired backwards out of the box; we connected the red and the black wires to our circuit, and the power switch turned ON when it was turned to OFF and turned OFF when it was turned to ON. One way to not have this happen in the future is to check that these wires are correct before soldering them. We also might devise a better way to close the enclosures other than superglue if we did the project again; it would be smart if we had included one side with screws or something so that it could be unscrewed in the event of internal technical failure so that someone could open it up and fix it without having to smash it open. Lastly, if we were to do the project again we would use a micro-controller with more memory so that we could display the letters on the e-ink display in a larger font.
Technical Details
Block Diagrams
Transmitter Block Diagram
Receiver Block Diagram
Electrical Schematics
Transmitter Electrical Schematic
Receiver Electrical Schematic
Code
Code Introduction & Citations
/********************************************************************
PHYSICAL COMPUTING 60-223 FINAL PROJECT CODE
Code for Augmentative & Alternative Communication Device
By Team Hickories (Aidan, Ella and Gemma)
Code Citations:
https://www.waveshare.com/wiki/Template:Arduino_Guides_for_SPI_e-Paper
https://github.com/soonuse/epd-library-arduino/blob/master/2.9inch_e-// paper/epd2in9/examples/epd2in9-demo/epd2in9-demo.ino
https://howtomechatronics.com/tutorials/arduino/arduino-wireless-communication-nrf24l01-tutorial
https://arduinogetstarted.com/tutorials/arduino-neopixel-led-strip
https://forum.arduino.cc/t/neopixels-and-button-control/910405/2
https://docs.arduino.cc/built-in-examples/digital/Button
*******************************************************************/
Transmitter Code
/********************************************************************
Transmitter Box Code for Augmentative & Alternative Communication Device
By Team Hickories (Aidan, Ella and Gemma)
Code Purpose:
Transmits data via wireless receiver corresponding to a button press.Each button has a message attributed to it which is sent to the receiver box.
Pin Map
Pin Number | Type | Connection
3 | INPUT | Button #1
4 | INPUT | Button #5
5 | INPUT | Button #4
6 | INPUT | Button #3
7 | OUTPUT | NRF24L01 Module RF24 CE
8 | OUTPUT | NRF24L01 Module RF24 CSN
9 | INPUT | Button #2
10 | EMPTY |
11 | OUTPUT | NRF24L01 Module MOSI
12 | OUTPUT | NRF24L01 Module MISO
13 | OUTPUT | NRF24L01 Module SCK
********************************************************************/
#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>
// Define Constants
#define COLORED 0
#define UNCOLORED 1
// NRF24L01 Setup
RF24 radio(7, 8);
const byte address[6] = "60223";
// Button Setup
const int buttonPins[] = {3, 9, 6, 5, 4};
bool lastButtonStates[5] = {HIGH, HIGH, HIGH, HIGH, HIGH};
void setup() {
// Initialize Radio
radio.begin();
radio.openWritingPipe(address);
radio.setPALevel(RF24_PA_MIN);
radio.stopListening();
// Button Pin Logic
for (int i = 0; i < 5; i++) {
pinMode(buttonPins[i], INPUT_PULLUP);
}
}
void loop() {
// Check Each Button for Change
for (int i = 0; i < 5; i++) {
bool currentButtonState = digitalRead(buttonPins[i]);
if (currentButtonState != lastButtonStates[i] && currentButtonState == LOW) {
char messageToSend = '1' + i;
radio.write(&messageToSend, sizeof(messageToSend)); //Radio Transmission
}
lastButtonStates[i] = currentButtonState;
}
delay(50);
}
Receiver Code
/********************************************************************
Receiver Box Code for Augmentative & Alternative Communication Device
By Team Hickories (Aidan, Ella and Gemma)
Code Purpose:
Receives message data from transmitter box according to which button is pressed. The corresponding message is printed on the E-ink display. Then, the Neopixel strip lights up green to indicate a new message. Once read, the user presses the reset button and the message and Neopixel strip clears.
Pin Map
Pin Number | Type | Connection
2 | INPUT | NRF24L01 Module RF24 CE
3 | INPUT | NRF24L01 Module RF24 CSN (SS)
4 | EMPTY |
5 | INPUT | Reset Button
6 | OUTPUT | NeoPixel
7 | OUTPUT | E-ink Module BUSY
8 | OUTPUT | E-ink Module RST
9 | OUTPUT | E-ink Module DC
10 | OUTPUT | E-ink Module CS (SS)
11 | SHARED | NRF24L01 Module MOSI & E-ink Module DIN (MOSI)
12 | OUTPUT | NRF24L01 Module MISO
13 | SHARED | NRF24L01 Module SCK & E-ink Module CLK (SCK)
********************************************************************/
#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>
#include "epd2in9_V2.h"
#include "epdpaint.h"
#include <Adafruit_NeoPixel.h>
// Define Constants
#define COLORED 0
#define UNCOLORED 1
#define PIN_NEO_PIXEL 6
#define NUM_PIXELS 8
// NRF24L01 Setup
RF24 radio(2, 3);
const byte address[6] = "60223";
// Button Setup
const int resetButtonPin = 5;
// e-ink Display Setup
Epd epd;
unsigned char image[1100];
Paint paint(image, 0, 0);
bool isDisplayOn = false;
bool lastResetButtonState = HIGH;
bool messageReceived = false;
// NeoPixel Strip Setup
Adafruit_NeoPixel NeoPixel(NUM_PIXELS, PIN_NEO_PIXEL, NEO_GRB + NEO_KHZ800);
void setup() {
// Radio Start
Serial.begin(9600);
// Initialize Radio
radio.begin();
radio.openReadingPipe(0, address);
radio.setPALevel(RF24_PA_MIN);
radio.startListening();
// Initialize Button Pin
pinMode(resetButtonPin, INPUT_PULLUP);
// Initialize e-ink Display
if (epd.Init() != 0) {
Serial.println("e-Paper init failed");
return;
}
epd.ClearFrameMemory(0xFF);
epd.DisplayFrame();
// Initialize NeoPixel Strip
NeoPixel.begin();
NeoPixel.setBrightness(10);
setNeoPixelColor(33, 138, 255);
}
void loop() {
// Check for Messages
if (radio.available()) {
char receivedMessage;
radio.read(&receivedMessage, sizeof(receivedMessage));
Serial.print("Received message: ");
Serial.println(receivedMessage);
displayMessage(receivedMessage);
messageReceived = true;
}
// Check Reset Button Change
bool currentResetButtonState = digitalRead(resetButtonPin);
if (currentResetButtonState != lastResetButtonState && currentResetButtonState == LOW) {
if (isDisplayOn) {
clearDisplay();
messageReceived = false;
setNeoPixelColor(33, 138, 255);
}
}
lastResetButtonState = currentResetButtonState;
delay(50);
}
//Displaying Message Logic Helper Function
void displayMessage(char buttonNumber) {
isDisplayOn = true;
//NeoPixel Change
NeoPixel.setBrightness(255);
setNeoPixelColor(0, 255, 0);
//Text Display Initialization
paint.SetWidth(40);
paint.SetHeight(220);
paint.SetRotate(ROTATE_270);
paint.Clear(UNCOLORED);
//Message List
const char* line1;
const char* line2;
switch (buttonNumber) {
case '1':
line1 = "Can I have";
line2 = "more time?";
break;
case '2':
line1 = "I'm feeling";
line2 = "nervous/shy.";
break;
case '3':
line1 = "I'm not comfortable";
line2 = "with that.";
break;
case '4':
line1 = "I'm having";
line2 = "trouble focusing.";
break;
case '5':
line1 = "I need";
line2 = "some space.";
break;
default:
line1 = "";
line2 = "";
break;
}
//e-ink Draw Logic
paint.DrawStringAt(0, 8, line1, &Font16, COLORED);
paint.DrawStringAt(0, 24, line2, &Font16, COLORED);
epd.ClearFrameMemory(0xFF);
epd.SetFrameMemory(paint.GetImage(), 0, 0, paint.GetWidth(), paint.GetHeight());
epd.DisplayFrame();
}
//e-ink Sleep Logic
void clearDisplay() {
isDisplayOn = false;
epd.ClearFrameMemory(0xFF);
epd.DisplayFrame();
NeoPixel.setBrightness(10);
setNeoPixelColor(33, 138, 255);
}
//NeoPixel Color Logic Helper Function
void setNeoPixelColor(uint8_t r, uint8_t g, uint8_t b) {
for (int i = 0; i < NUM_PIXELS; i++) {
NeoPixel.setPixelColor(i, NeoPixel.Color(r, g, b));
}
NeoPixel.show();
}
Design/Drafting Files
DXF box files with the button cutouts for laser cutting. Customizable with any photos/messages you would like to add!
Base box model credit to MakerCase (https://en.makercase.com/#/).