Searching for data using NLP on custom hardware

This project is being designed, developed, and implemented to help increase productivity of employees within Google Hardware. This project will provide the ability for both technical and non-technical individuals to access information as simply as speaking the query.

A custom IoT voice assistant developed as a senior design project in the year 2019 - 2020. This voice assistant is unique since it can interact with a client specified database to help a user query the database with natural language queries. There are two ways to input a query into the system: a cloud GUI application and the physical microphone on the system. A user can hold down the button on the enclosure to speak a natural language query (for example, "How many Google Home Minis were shipped in the past month?"). The system converts this query into text, runs natural language processing on the textual query and uses an Elastic Search database to generate a response to the user's query. The result is then output through the speaker for the user to hear, or output as a table on the cloud GUI application.

A more technical description follows: Users are able to speak a query to the IoT device, a Raspberry Pi, which then using Google's Speech to Text API and natural language processing will convert the query (input from the microphone) to text and run the query against an index returning the result through a Text to Speech program, eSpeak. The result is converted back into human recognizable speech and then output through the speaker.

The technologies used included PostgreSQL, ElasticSearch, Node.js, Python, NLTK, SpaCy, eSpeak, Google Cloud Speech to Text API, PyAudio, RasPi GPIO, HTML and CSS.

This website serves as a source of information about the background, status, and reference for the project. Our client, Neerad Hakhu, is an engineering manager at Google Hardware. Learn more about the members of this team, here. Learn more about the technologies used to develop the voice assistant, here. Learn more about our project timeline, here.

Team 36 is constantly updating this website, stay tuned! We're excited about the progress we have made this year and hope you have fun looking through our project!