Speeding up the ordering process with artificial intelligence

Gmail and Office365 API

One of the important parts about this project is manipulating the mailbox in a way that doesn't interrupt the workflow at TVH. This means we needed to be able to move emails around based on their labels or folder structure. That is why we developed an API that integrates both with Gmail and Office365. This API is written in Python and has the following functionality:

• Move incoming mails to the dedicated label or folder
• Extract all the information from an email
• Put all the information per email on a Cloud Pub/Sub
• Be able to add and remove labels from an email
• Send and reply on emails

Backend

The backend is the backbone of this project. This will interact with all the other services. It will make sure that all emails are being processed correctly. This API is written in Python and has the following functionality:

• Listen on the Cloud Pub/Sub
• Send the email data to the text classifier
• Send data to the Named Entity Recognition
• Send instructions to the mail API
• Process requests from the web interface
• Create an order
• Text classifier

The text classifier is necessary to determine if an email contains an order or not. In the future this can also be expanded to classify different types of emails. For example order, cancelation, sign out, quotation ... . We ended up going for a neural network that was language independent. TVH offers customer support in 37 languages. Keeping that in mind from the beginning will save us time in the long run.

The API works as followed: the body of the email is converted to a sequence of tokens. These tokens are then embedded into vectors. We are transforming the body to vectors because a neural networks cannot process text, only vectors. The vectors are then processed by the neural network which outputs an output vector. By adding a classification on the whole text the API is able to return if the email is an order or not.

Named Entity Recognition

For the Named Entity Recognition we used the same approach as the text classifier. This means that we can extract entities from emails independent of the language. The difference with the text classifier is that for the NER we want to predict the label of a token, not for the whole text. This is done by adding the classification on the output vector of each token. This will return a label for each token. With this we can extract the necessary entities like delivery date for example. This API expects a body of text as input and will return all the tokens with their corresponding label.

Web interface

The web interface has two functions. The first is to verify if an order has been made correctly before sending it to the system. The second function is to save the corrections that are being made. This will help to improve the text classifier and the Named Entity Recognition. The web interface has a login page, overview page and an order page.

Result

An email comes in on the mailbox. The email is being labeled as 'In Progress' while all the information is put on the Cloud Pub/Sub. The backend retrieves all the information from the Pub/Sub and sends it to the text classifier. This returns that the email is indeed an order. The backend sends the body to the named entity recognition. This will return all the entities in the email. The backend will then use all the information to create an order. It will then send instructions to the mail API to remove the label 'In Progress' and add the label 'Processed'.

Now an employee of TVH can login onto the web interface and see a list of emails that need reviewing or that are reviewed. When the employee opens an order he will see the original email on the lefthand side and the order on the righthand side. The employee can then make changes as needed and click submit when done. This information is used to improve the different ai solutions, reducing the time even further.

The future

Brainjar is still working with TVH on improving the application even further. Right now we are working on implementing PDF. A lot of the customers of TVH make orders with PDF's. The problem is that each customer has its own layout and PDF structure. The next step is to also process those emails. Another future extension is adding additional classes of emails, for example: stop orders, change orders, ... .