Team Hickories: Aidan, Ella and Gemma

Client: Maggie

Project 3, or “a useful implement for a person living with a disability” had our group of three working with a client from the organization CLASS (Community Living and Support Services) in order to make an assistive device that was uniquely tailored to them and their needs. This project spanned about a quarter’s worth of weeks and took many different iterations and ideas to reach where it ended up.  We had an initial interview consultation with our client Maggie, a second interview follow up during out ideation phase, a prototype critique and a final critique - all of which helped us flesh out what we were making and how we were making it. For this project, Team Hickories made an Augmentative & Alternative Communication Device. 

For more information on our initial interview process, click here.

What We Built

For this project, we built an Augmentative & Alternative Communication Device for Maggie. The function of this device is relatively simple. The polygonal transmitter box has one button on each side, and each button transmits a different message to the receiver box. The message that each button sends is indicated by a laser etched message printed on each respective side. The receiver box receives the signal and displays the message on an E-Ink display, also lighting up a light strip on top to green to indicate a new message. When the person reading the message has read it, they click a button on the top and the message and light strip clears.

Devices in Action

Narrative Description

Maggie is sitting in the computer lab at CLASS and is notified that its time to move to a different space; she is almost done with the art she is working on, and leaving the space right now feels overwhelming, but communicating this feels hard. She pulled out the device and sets the receiver near her on the desk so the CLASS individual can see the screen, and presses the "Can I have more time?" button. They tell her she can have 5 more minutes and she spends that time preparing to move to the next space, feeling comfortable and calm having communicated her needs. 

How We Got Here (Prototype and Process)

Prototype 

This prototype was built to primarily answer the following design question:  

• How does the user respond to the size, shape, and input types?

The prototype we built for the critique was firmly focused on giving Maggie as many options on tactile inputs, shape and form as possible while still being manageable. We built three boxes out of two kinds of material (a cardboard-like material and clear acrylic). These three boxes were varied in size, shape and number of sides. We also embedded a number of different kinds of buttons, switches and sliders. Additionally, we brought a number of index cards to the critique along with an LED hooked up to a wireless transceiver which communicated with a button in order to model what the function of the box would look like. After discussing with Maggie, we put the most important phrases to her on the cards.

Prototype Ideation and Fabrication Process

Original Ideation Sketches (for Prototype)

The original sketches of our first ideation of the design of the device - in this version there was only one box with a message.

Prototype Brainstorming and Fabrication

Prototype Findings

Our guiding question for our prototype,  "How does the user respond to the size, shape, and input types?" was answered through our prototype critique with Maggie. We identified which buttons, switches, and sliders she preferred (she liked the feel of the momentary switches) and which side and shape of the boxes she liked. She preferred the cube and the pentagon, and when it came down to deciding between the two, she liked the idea of the pentagon fitting more of her favorite characters and more messages and buttons than the cube would. Having the physical prototype there for her to touch and feel and hold was very helpful in this process, and once we had answered our guiding question, we still had a bit of time left. With this extra time, we talked about her favorite TV show and game characters, and how we could incorporate these characters as well as different colors into the final project design. 

Feedback that we integrated into our final design included using the buttons that Maggie liked the best, shaping the receiver and the transmitter boxes in the shapes that she preferred, and putting her favorite colors and characters into the aesthetic design of the project. We also spent a lot of time during the prototype critique discussing the most important messages that should go on the box. We made a list of messages with Maggie and then narrowed it down with her to the most important 5 messages, which were the ones that ended up on the project. 

The only feedback from the critique that we received that we didn't incorporate into the final design was a plethora of favorite characters and messages that simply didn't fit on our design, since an important aspect of the design was it being portable. We worked through the long list of characters and messages and narrowed it down with Maggie to just 5. 

Surprises that we encountered during the prototyping process included having a hard time getting firm answers from Maggie on some topics. We worked through different ways of framing our discussions to get the most helpful feedback out of our critique, but we are glad that we had a long period of time for the critique to ensure we got the information we needed from her. 

Overall Process

Code and Wiring

Final Enclosure Fabrication

Soldering and Final Assembly

Scheduling/Timeline Reflections

We ended up sticking pretty close to this Gantt chart we made planning our project timeline - partly out of pre-scheduled necessity (one of our group members was out of town for some of the last week). Some of the final assembly occurred later than intended but other than that everyone stayed mostly on schedule. This was due to us having a realistic conversation about timing and other obligations while making  the schedule - trying to front load the work as much as possible ended up being extremely helpful due to everyone having finals and other obligations that would end up taking up a lot of time.

Overall Thoughts/Reflections

Overall, our project went pretty smoothly. Our decisions were made as a team and heavily influenced by Maggie's input, and we stuck pretty closely to our plan from the time of prototype critique to final critique. A decision that we made was to make a prototype that purely gauged sensory choices, which meant that our prototyping took care of most of the major decisions that we needed to make. This allowed us to really "head down" work from that point forward, which was helpful; if we had prototyped the code, for example, we likely would have needed to make many more design choices on the road to the final critique without Maggie's input. 

Breakthrough moments included getting the transceiver to work and getting the e-ink display to work. These were two aspects of the project that gave us some trouble along the way, but nothing we couldn't handle! We didn't have any major breakdown moments, thankfully, but we did have some minor frustrations with getting the laser to engrave the images facing the correct way. Once we got those settings down, we were all good to go!

Conclusions and Lessons Learned

We learned a lot of things along the way to completing our final project. Some of the things we learned came from our final critique, where we presented our project to our client, the class, and a handful of guest critiquers. One piece of feedback we received about our transmitter was “Needs softer corners.” The decision for the transmitter to have a very clean acrylic fabrication was deliberate to avoid some of Maggie’s sensory needs; she doesn’t like certain textures, and we knew that acrylic was a safe one and didn’t want to choose something softer without knowing if she would like it or not. Another piece of feedback we got was “Easier to rotate transmitter (sliders on bottom),” which is a small detail that we hadn’t though of that would make the project much better through a small change. Small fabric “feet” on the base of the transmitter would make this project much better at a super tiny cost of effort and money – since it’s mostly designed for on-table use, having little sliders on the base of both the transmitter and receiver makes sense. We also got the feedback of “Allow for the pressing of multiple buttons. Allow for “unpressing” of buttons.”  Pressing multiple buttons and un-pressing buttons would be a great addition to the project, and would just be some pretty easy code changes to implement! “I wish both devices had LED lights to indicate the message sent/received and I with the functional buttons didn't look so similar to the fidget buttons” was another piece of feedback that we got; LEDs on the transmitter to show that a message was received was part of our original discussion, but this detail got lost along the way. It would be a great addition to the project and wouldn’t be that difficult to implement! Lastly, we got the feedback “See if you can shrink the form and make it more portable.” Ideally, we would love to be able to do this, but the project was already such a tight fit of wires and protoboards on the inside that, as the wiring currently stands, making the form smaller is pretty out of the question unless we do some major design changes. 

We had a few takeaways from the experience of working with a person with a disability and designing something just for them. Overall, the process was super fulfilling; the concept of doing a project that might actually help someone in their day to day life is much more compelling than many projects we face in college, which are simply for a grade. Our main struggle in the process was communication; before we got to know each other, it was hard to get our client out of their shell. It was very fulfilling once we got closer to her and she felt comfortable getting excited around us, such as at our prototype critique when she got excited about choosing themes and talking about her favorite shows and characters. Something we would do differently next time is come more prepared for the client, like maybe talk to our contact beforehand to understand what their specific struggles are so that we go in with some initial ideas, or at least prepared with tactics to best communicate with them in the initial interview.

Overall, this has been a very fulfilling experience for all of us. The process definitely emphasized to us that soldering and fabrication always take longer than anticipated, so that is an element of the project that should be started as soon as possible. We started the fabrication process as soon as we started the project, and we still had a little bit of crunch time at the end. Something unfortunate that happened is our battery cases were somehow wired backwards out of the box; we connected the red and the black wires to our circuit, and the power switch turned ON when it was turned to OFF and turned OFF when it was turned to ON. One way to not have this happen in the future is to check that these wires are correct before soldering them. We also might devise a better way to close the enclosures other than superglue if we did the project again; it would be smart if we had included one side with screws or something so that it could be unscrewed in the event of internal technical failure so that someone could open it up and fix it without having to smash it open. Lastly, if we were to do the project again we would use a micro-controller with more memory so that we could display the letters on the e-ink display in a larger font. 

Technical Details

Block Diagrams

Transmitter Block Diagram

Receiver Block Diagram

Electrical Schematics

Code 

Code Introduction & Citations

/********************************************************************

PHYSICAL COMPUTING 60-223 FINAL PROJECT CODE

Code for Augmentative & Alternative Communication Device

By Team Hickories (Aidan, Ella and Gemma)

Code Citations:

https://www.waveshare.com/wiki/Template:Arduino_Guides_for_SPI_e-Paper

https://github.com/soonuse/epd-library-arduino/blob/master/2.9inch_e-// paper/epd2in9/examples/epd2in9-demo/epd2in9-demo.ino

https://howtomechatronics.com/tutorials/arduino/arduino-wireless-communication-nrf24l01-tutorial

https://arduinogetstarted.com/tutorials/arduino-neopixel-led-strip

https://forum.arduino.cc/t/neopixels-and-button-control/910405/2

https://docs.arduino.cc/built-in-examples/digital/Button

*******************************************************************/

Transmitter Code

/********************************************************************

Transmitter Box Code for Augmentative & Alternative Communication Device

By Team Hickories (Aidan, Ella and Gemma)

Code Purpose:

Transmits data via wireless receiver corresponding to a button press.Each button has a message attributed to it which is sent to the receiver box.

Pin Map

Pin Number | Type   | Connection

3          | INPUT  | Button #1

4          | INPUT  | Button #5

5          | INPUT  | Button #4

6          | INPUT  | Button #3

7          | OUTPUT | NRF24L01 Module RF24 CE

8          | OUTPUT | NRF24L01 Module RF24 CSN

9          | INPUT  | Button #2

10         | EMPTY  | 

11         | OUTPUT | NRF24L01 Module MOSI

12         | OUTPUT | NRF24L01 Module MISO

13         | OUTPUT | NRF24L01 Module SCK

********************************************************************/

#include <SPI.h>

#include <nRF24L01.h>

#include <RF24.h>

// Define Constants

#define COLORED 0

#define UNCOLORED 1

// NRF24L01 Setup

RF24 radio(7, 8);

const byte address[6] = "60223";

// Button Setup

const int buttonPins[] = {3, 9, 6, 5, 4};

bool lastButtonStates[5] = {HIGH, HIGH, HIGH, HIGH, HIGH};

void setup() {

   // Initialize Radio

   radio.begin();

   radio.openWritingPipe(address);

   radio.setPALevel(RF24_PA_MIN);

   radio.stopListening();

   // Button Pin Logic

   for (int i = 0; i < 5; i++) {

       pinMode(buttonPins[i], INPUT_PULLUP);

   }

}

void loop() {

   // Check Each Button for Change

   for (int i = 0; i < 5; i++) {

       bool currentButtonState = digitalRead(buttonPins[i]);

       if (currentButtonState != lastButtonStates[i] && currentButtonState == LOW) {

           char messageToSend = '1' + i;

           radio.write(&messageToSend, sizeof(messageToSend)); //Radio Transmission

       }

       lastButtonStates[i] = currentButtonState;

   }

   delay(50);

}

Receiver Code

/********************************************************************

Receiver Box Code for Augmentative & Alternative Communication Device

By Team Hickories (Aidan, Ella and Gemma)

Code Purpose:

Receives message data from transmitter box according to which button is pressed. The corresponding message is printed on the E-ink display. Then, the Neopixel strip lights up green to indicate a new message. Once read, the user presses the reset button and the message and Neopixel strip clears.

Pin Map

Pin Number | Type   | Connection

2          | INPUT  | NRF24L01 Module RF24 CE

3          | INPUT  | NRF24L01 Module RF24 CSN (SS)

4          | EMPTY  | 

5          | INPUT  | Reset Button

6          | OUTPUT | NeoPixel 

7          | OUTPUT | E-ink Module BUSY

8          | OUTPUT | E-ink Module RST

9          | OUTPUT | E-ink Module DC

10         | OUTPUT | E-ink Module CS (SS)

11         | SHARED | NRF24L01 Module MOSI & E-ink Module DIN (MOSI)

12         | OUTPUT | NRF24L01 Module MISO 

13         | SHARED | NRF24L01 Module SCK & E-ink Module CLK (SCK)

********************************************************************/

#include <SPI.h>

#include <nRF24L01.h>

#include <RF24.h>

#include "epd2in9_V2.h"

#include "epdpaint.h"

#include <Adafruit_NeoPixel.h>

// Define Constants

#define COLORED 0

#define UNCOLORED 1

#define PIN_NEO_PIXEL 6

#define NUM_PIXELS 8

// NRF24L01 Setup

RF24 radio(2, 3);

const byte address[6] = "60223";

// Button Setup

const int resetButtonPin = 5;

// e-ink Display Setup

Epd epd;

unsigned char image[1100];

Paint paint(image, 0, 0);

bool isDisplayOn = false;

bool lastResetButtonState = HIGH;

bool messageReceived = false;

// NeoPixel Strip Setup

Adafruit_NeoPixel NeoPixel(NUM_PIXELS, PIN_NEO_PIXEL, NEO_GRB + NEO_KHZ800);

void setup() {

   // Radio Start

   Serial.begin(9600);

   // Initialize Radio

   radio.begin();

   radio.openReadingPipe(0, address);

   radio.setPALevel(RF24_PA_MIN);

   radio.startListening();

   // Initialize Button Pin

   pinMode(resetButtonPin, INPUT_PULLUP);

   // Initialize e-ink Display

   if (epd.Init() != 0) {

       Serial.println("e-Paper init failed");

       return;

   }

   epd.ClearFrameMemory(0xFF);

   epd.DisplayFrame();

   // Initialize NeoPixel Strip

   NeoPixel.begin();

   NeoPixel.setBrightness(10);

   setNeoPixelColor(33, 138, 255);

}

void loop() {

   // Check for Messages

   if (radio.available()) {

       char receivedMessage;

       radio.read(&receivedMessage, sizeof(receivedMessage));

       Serial.print("Received message: ");

       Serial.println(receivedMessage);

       displayMessage(receivedMessage);

       messageReceived = true;

   }

   // Check Reset Button Change

   bool currentResetButtonState = digitalRead(resetButtonPin);

   if (currentResetButtonState != lastResetButtonState && currentResetButtonState == LOW) {

       if (isDisplayOn) {

           clearDisplay();

           messageReceived = false;

           setNeoPixelColor(33, 138, 255);

       }

   }

   lastResetButtonState = currentResetButtonState;

   delay(50);

}

//Displaying Message Logic Helper Function

void displayMessage(char buttonNumber) {

isDisplayOn = true;

//NeoPixel Change

NeoPixel.setBrightness(255);

setNeoPixelColor(0, 255, 0);

//Text Display Initialization

paint.SetWidth(40);

paint.SetHeight(220);

paint.SetRotate(ROTATE_270);

paint.Clear(UNCOLORED);

//Message List

const char* line1;

const char* line2;

  switch (buttonNumber) {

  case '1':

    line1 = "Can I have";

    line2 = "more time?";

    break;

  case '2':

    line1 = "I'm feeling";

    line2 = "nervous/shy.";

    break;

  case '3':

    line1 = "I'm not comfortable";

    line2 = "with that.";

    break;

  case '4':

    line1 = "I'm having";

    line2 = "trouble focusing.";

    break;

  case '5':

    line1 = "I need";

    line2 = "some space.";

    break;

  default:

    line1 = "";

    line2 = "";

    break;

}

//e-ink Draw Logic

paint.DrawStringAt(0, 8, line1, &Font16, COLORED);

paint.DrawStringAt(0, 24, line2, &Font16, COLORED);

epd.ClearFrameMemory(0xFF);

epd.SetFrameMemory(paint.GetImage(), 0, 0, paint.GetWidth(), paint.GetHeight());

epd.DisplayFrame();

}

//e-ink Sleep Logic

void clearDisplay() {

isDisplayOn = false;

epd.ClearFrameMemory(0xFF);

epd.DisplayFrame();

NeoPixel.setBrightness(10);

setNeoPixelColor(33, 138, 255);

}

//NeoPixel Color Logic Helper Function

void setNeoPixelColor(uint8_t r, uint8_t g, uint8_t b) {

for (int i = 0; i < NUM_PIXELS; i++) {

  NeoPixel.setPixelColor(i, NeoPixel.Color(r, g, b));

}

NeoPixel.show();

}

Design/Drafting Files

DXF box files with the button cutouts for laser cutting. Customizable with any photos/messages you would like to add!

Base box model credit to MakerCase (https://en.makercase.com/#/).