BlenderBot 2.0 - Simplified

Facebook AI has blended long-term memory & contextual search to build Blender Bot 2.0, a state of the art chatbot and opensourced it on July 16, 2021.

It is an updated version of the Blender Bot, open sourced in 2020.

In this article, BB2 is used as a short form in place of BLENDER BOT 2.0.

FEATURES

Long-term memory : With long term memory lasting up to several months, the bot can store the information extracted from past conversations & use them in the present conversation.

Contextual search :While conversing, the model can perform contextual search for untrained data(words) & blend the search results into the conversation. Thus BB2 stays updated forever and can learn & gain additional knowledge.It can query using any search engine (currently tested on Bing API & nearest neighbor lookup)

Individual knowledge Bank : Bot stores the knowledge separately for every individual it speaks with. Thus information learned in 1 person's conversation won't be used in another during multi session chats

Limitations of other models

• The present language generation models suffer from short memory (gold-fish memory).

• Even if those models have long term memory,it is just static(remembers only the past training data) and the memory won't get updated.

ILLUSTRATIONS

Illustration - 1:

Consider the following facts.

In 2020, Donald Trump was the American president

In 2021, Joe Biden is the American President

For illustration purposes, let us assume all the models including BB2 were trained in 2020 .

When a human asks, "Who is the American President ?",

Other bots reply => Donald Trump

BB2 replies => Joe Biden

Thus BB2 stays updated due to its capability to search internet & give a reply according to the current situation.

Illustration-2:

BB2 can remember the context of previous discussions.

For example, if you ever spoke about BTS or K-pop to the bot, it might speak to you about Taehyung or Jungkook in the future as it knows you might be interested in BTS members. BB2 can even become your recommendation chatbot in future..!!!

MODEL ARCHITECTURE

BB2 uses a model based on Retrieval Augmented Generation(RAG) by Facebook. RAG can generate replies containing more knowledge than the knowledge learnt from the conversation .

During conversation, the model (which combines an information retrieval component with a seq2seq generator) seeks relevant information both in its long-term memory and from documents it finds by searching the internet.

Neural Module-1

Model pulls from the chatbot’s long-term memory store and decides what to add to it. This is achieved by using an additional neural module which generates the memory to be stored based on the conversational context.

Neural Module-2

To do search, we augment the traditional encoder-decoder architecture with an additional neural network module that performs contextual search queries & prepends the resulting knowledge to the conversational history which is encoded using the Fusion-in-Decoder method to generate a response.

Current large models store their learnings in their model weights.

Storing the ever growing internet data in models is impossible.

But models accessing the internet on fly is possible with BB2

DATASETS

FB collected data on a crowdsourcing platform & has released the following conversational data sets :

• Wizard of the Internet

• Multi-Session Chat

Wizard of the Internet ( Human conversations augmented with new information from internet searches ) provides supervision for BB2 on how to generate relevant search engine queries & supervision of relevant responses based on the search results

Multi-Session Chat ( Multisession, long-context chat with humans referencing knowledge from conversation sessions ) provides supervision for the chatbot on which fresh knowledge to store in long-term memory and supervision for relevant responses given those memories. Multitask training is done by combining the data sets, which enables BB2 to act simultaneously with all these skills.

BB2 is a successor of BB1. Hence BB2 was trained on tasks that were used to train BB1. BB1 was trained on Blended Skill Talk tasks — engaging use of personality, knowledge and the display of empathy

SAFETY

FB developed two new methods baked-in safety and robustness to difficult prompts to ensure safe conversations & reduce offensive responses, deceptive/damaging hallucinations by the bot. As BB2 can use internet, New safety challenges lie ahead which can be solved together by the AI community...!!!

SAMPLE CONVERSATION WITH BLENDERBOT 2.0