Problem Statement
Police officers on the field find current communication methods with the deaf members of the public slow and inefficient. The client seeks a solution that improves the flow and ease of communication between both stakeholders, stakeholder A who is a deaf member of the public and stakeholder B who is a police officer untrained in sign language, so that conversations can be smoother and more expressive.
Intended Usage Flow
Signr, in action.
Fully deaf individual meet patrolling Police Officers (who patrol in pairs)
Police Officers seeks permission to record Deaf Individual signing
If they don't allow, app not to be used
If allow, proceed below
Police Officer A starts recording the conversation, with rear-facing camera facing the deaf individual
Police Officer A will be looking at and understanding the transcription while Police Officer B asks questions.
Police Officer B conveys questions to DI through pen/paper
The questions that the Police Officer B asked is not stored in the app.
Deaf Individual signs their answers to Police Officer A’s camera for transcription.
Police Officer A relays what he understood from the live transcription on his phone from the Deaf individual’s signing in step 5, to Police Officer B.
Police Officer B then conveys the next set of questions to the Deaf Individual.
The video will still be recorded as Police Officer B conveys his questions to Deaf Individual.
The transcription will not transcript anything if it cannot detect someone signing in frame in front of the camera.
Repeat Step 4 to 7 until all queries are resolved.
Police Officer A stops recording and will be brought to the “Save Video” page with editable default details like time and date of interview.
Feature Overview
Our key features recorded down as user stories
User Story 1: Login authentication with Google
Scenario 1 - Redirect to oAuth screen
Given that I am on login page
When I click on 'Sign In with Google'
Then I should see google oAuth login screen
Scenario 2 - Can only access app when logged in
Given that I am not logged in
When I try to access the app other than the login screen
Then I should be redirected to the login screen
Scenario 3 - Personal recording logs
Given that I am on the Recording logs page
And I have made recordings
Then I only see my own recordings
User Story 2: Convenient application navigation
Scenario 1 - check landing page features
Given that I am at "user" page
Then I should see "Recording" button
And I should see "View Past Recordings"
Scenario 2 - going to the logs page
Given that I am at "user" page
When I click on the "View Past Recordings" button
Then I should be brought to the "index" section of "recording" page
Scenario 3 - starting a video recording
Given that I am at "user" page
When I click on the "Home" button
Then I should be brought to the "new" section of "recording" page
User Story 3: Video recording and transcription of the conversation
Scenario 1 - Pre-recording checks
Given that I am at "new" section of "recording" page
Then I should see "Start" button
And I should see a video screen
Scenario 2 - Start recording
Given that I am at "new" section of "recording" page
Then I should see "Start" button
When I click on the "Start" button
Then I should see "Stop" button
And the “Stop” button should be “red”
When I click on the "Stop" button
Then I should be brought to the "show" section of "recording" page
Scenario 3 - User can view transcription
Given that I am at "new" section of "recording" page
When I click on the "Start" button
And I am signing a hand sign
Then I should see an English transcription of the hand sign
Scenario 4 - User can save transcription
Given that I am already recording a video with transcription
When I click on the "Stop" button
Then I should be brought to the "edit" section of "recording" page
User Story 4: Database with past recorded videos
Scenario 1 - finding a relevant video among past recorded videos
Given that I am on the "Recording Logs" page
Then I should see a logs table
And I should see Title, Date and Edit
And I should see "Home" button
Scenario 2 - link to correct past video screen
Given that I am on the "Recording Logs" page
When I click on the "Edit" button
Then I should be brought to the "edit" section of "recording" page
Scenario 3 - link to back to home
Given that I am on the "Recording Logs" page
When I click on the "Home" button
Then I should be brought to the "new" section of "user" page
User Story 5: Edit Past Video Transcription
Scenario 1- Validate edit recording UI
Given that I am on the edit page, "1" section
Then I should see "Title" and "Transcription" text area
And I should see "Update Recording" button
And I should see "Destroy" button
Scenario 2 - Title and Transcript is editable
Given that I am logged in
Given that I am on the edit page, "1" section
When I fill in "recording title" with "Sample Title"
When I fill in "recording transcription" with "This is the new transcription"
And I click on the "Update Recording" button
Then I should be brought to the "Home" page
Scenario 3 - Video is destroyed
Given that I am on the edit page
And I click on the "Destroy" button
Then a pop up will appear
System Architecture
What we used for our web application, and how they interact with each other
Cloud Storage
We stored and accessed the video recordings on Cloud Storage
Vertex.ai
We created our own dataset and trained and deployed our models on Vertex AI.
Compute Engine
We deployed our application using a Compute Engine
Ruby on Rails
We designed the application following the MVC framework using Ruby on Rails
Database Schema
Comprehensive Testing
to ensure our service remains bug free.
Cucumber
Test cases are written as user stories
Selenium
End-to-end testing, fully automated
Machine Learning with Vertex AI
Our solution utilizes VertexAI's image object detection model to deliver accurate transcriptions of a subject's signing.
We captured our own dataset and focused on the accuracy for a set of signs relevant to our scenario for this prototype.
Our application livestreams a series of images from the ongoing video recording to the deployed model so that transcriptions occurs live.
A filtering algorithm runs concurrently on the client side to filter any repeated predictions.
Try it out!
The Team
Park Bang Yong
Ma Qi Chao
Gabriel Phang
Rebecca Ling