Language model

A language model (LM) is a type of artificial intelligence, designed to understand and generate language.

[C1] A corpus is a collection of texts used to train a language model. It determines the language model/s vocabulary and what words it can generate. 

An error is outputted likely as that word is non-existant in the corpus. If we give the model word it didn't see in a training progress, it's unable to suggest what word comes next as it's no storing info about that word. Such model only knows about words in its corpus, big or small. 

Types

Large language models (LLM)

LLM's corpus is often a combination of texts from various sources like chat rooms, Wikipedia, novels, etc.

[C1] Smaller ones (SLMs) may have one type of text like emails or Pablo Neruda poems. Things like a phone's predictive text, virtual assistants like Siri or Alex rely on a language model.

Modern email programs often try to predict the user's next word in a sentence, via probability. They store probabilities for each word that may be next, calculating these based on word sequences in the corpus.

E.g., If we wrote an email by starting with "Thank you for the update...", the next word likely is "on" or "about", not something like "truck.

N-gram model

N-gram model are simple and offer a good model to understand how language models' predictions work. IO the letter "N", the types of N-gram model's names have numbers or prefixes in them to hint at the number of words are used as context in a phrase the model uses to predict the phrase's last word. E.g., A "bi"gram model uses 1 word to predict a phrase's 2nd (and last) word since its N = 2, then 2 - 1 = 1 word is used to predict that word. Another example: 8-gram model uses 7 words to predict the 8th word.

Overall, an N-gram model uses "'n' - 1" words to predict the 'n'th word.

A context window is an AI model's "working memory" that store that number of words the model can process and recall. 

But it's an issue if 'n' if a n-gram model's 'n' gets too large. E.g., A 6-gram model needs 5 words that appeared in that order to predict, so it's limited in its prediction. Thus, a text autocorrector uses a smaller 'n' than a plagiarism detector as fixing a word's spelling needs smaller context than finding a plagiarized string of words.

• [C1] A bigram model uses 1 word to predict its second word. Bigram is a 2-word sequence so N - 1 = 1 word is used to predict the last word. 

It answer questions like "given this first word, what's a likely next word?". The last word a user give provides the cvocntext of te next word. E.g., If we gave a bigram model a long prompt like "To be or not to be, that's the question," it'll use "question" as context. This is because it's the last word when the model predicts and ignores the prompt's rest, a processs called Markov assumption, which says that a future word's probability depends only on the current word.

• A trigram model uses 2 words of context to predict a word. Trigram is a 3-word sequence

• 
  

Bigram models 

[C1] n-gram models use predictions based on the most likely next word. To figure out what that word is, we translate words into numbers/probabilities.

A unigram model ("uni" = 1) that produces a prediction without context. A word a unigram predicts the most often is "the", as it's the most commonly used English word.

E.g., For a book, the n-gram algorithm first goes through the whole book and counts how many times each word is used. 

Consider this list of all bigrams in the phrase “Turn off the heat and stir in the butter and herbs.” in the book:

• turn + off: 1

• off + the: 1

• the + heat: 1 

• heat + and: 1 

• and + stir: 1 

• stir + in: 1 

• in + the: 1 

• the + butter: 1 

• butter + and: 1 

• and + the: 1 

• the + herbs: 1

[C1] The list of the 100 most frequent bigrams in that book are:

• in + the: 1675

• of + the: 1338

• to, the: 521

• the + sauce: 469

• for + to: 485

Therefore the most likely word to be after "the" is "sauce" and the most likely one to be before "the" is "in".

[C1] In a prompt ending by "egg", these are the most likely words to be after "egg":

[C1] Therefore, "whites" has 34% of the time to be after "egg and "yolk" is 8.2% of the time.

[C1] An n-gram model reach their limit in more complex tasks. E.g., It can't predict when the cake will be made or makes it, which is instead done by complex models like neural networks.

Large language model

[C1] A model must go through/be "exposed" to a set of data many times to be competent. Each data training session is called an epoch. E.g., This LLM has completed 1 epoch, with 800 GB of English text. But even at 800 GB of training data, the model hasn't learnt how to make words, which are most common, or how to string words together in a phrase. It randomly chooses the next characters.

LLMs are neural networks. Before training, predictions are stored in them are random, so is the output. After, say, 1,000 epochs, such model can make sentences better, but don't always make sense.

[C1] A LLM's neural network can improve per training session by comparing its prediction to the original data, each epoch to learn the pattern of English and create fully coherent sequences. The model's prediction is likely off by some amout. The contrast between the predicted and actual values is a "loss", which reduces each loss.

But even after so many epochs, a model's loss won't reach 0 as the output isn't the same text the model was trained on. Training a model til its loss reaches 0 is undesired, otherwise te model's prediction will fit the training data exactly and can't generate "novel" text, referred to as
"overfitting", i.e., it the training data's replicates patterns so well it can't account for new data/create new patterns. If an overfitted model receives a new prompt, it won't generate novel text, but repetitive ones. The model replicates the training data too closely, resulting in both training data's reproducted text or repetitive strings. So a monitor must train a model closely as a model starts perfoming well.

Small language model

Operation

[C1] 

Preprocessing

Preprocessing, alongside tokenization, is one of the 2 steps to turn raw text into a clean dataset.

Both can help standardize a corpus' text prior a model is trained on it. We can know what preprocessing decisions were made by analyzing the LM's output. 

E.g., A bigram model trained on the Big Tech companies' Term and Conditions (T&C)'s output won't have capitalization and punctuation when its input does. 

Capitalization

[C1] 

2 words like "My" and "my" use 2 different characters. i.e., To humans it means the same, but not to computers. A reprocessing algorithm can decapitalize and remove punctuations to decrease the vocabulary size for the model to see more examples of each word.

A spam email with uneven capitalization, mispelling, and emojis can be caught by a spam filter. We should treat capitalization in an LM depending on the situation. Preprocessing by LLM (Pythia, ChatGPT) often keeps capitalizations as their datasets are big.

Punctuation

[C1] Exclamations often convey excitement and don't a certain meaning. Periods can mean a phrase's end or an abbreviation. Thus, we use special characters to the start and end a phrase. A preprocessing algorithm may use <s> </s> for this. Regardless, we treat punctuations as words in an LLM to create definitions. 

[C1] Choices we make in preprocessing indicates how an LLM acts. 

Tokenization

[C1] For a sentence written as "my ai did my research.", an LLM only store "my" once in its vocabulary. Tokanization is the 2nd step to make text digestible easier for computers. It breaks a corpus into units the model can train on, like this via vertical bars: "I | tested | the | model|."

 This method to split words in spaces (including tabs and line breaks) is whitespace tokenization, one of many tokenization method use to convert texts as smaller manageable units, tokens.

[C1] "starting," started", "start" have akin meanings but affixes have crucial info. We can their stem from its affixes: "I | test|ed | the |  model|."

This broken down phrase has 6 tokens. The drawback of this is that writing a program to split words needs to know many for each language's grammar. 

Byte-pair encoding

[C1] Byte-pair encoding (BPE) is a corpus tokenizing method, an algorithm building tokens, prior an LLM can train on it,.

IO splitting text into tokens, we can use an algorithm building tokens from characters.

E.g., Via a corpus with: "email," "emailing," "imagine," "imagined.", we must know all letters in it.

If a BPE algorithm ran forever, it eventually rebuilds the original text, with 2 ways to stop this:

• By specifying the vocabulary size (number of tokens); or

• Run the algorithm some number of times.

A token vocabulary build via BPE can be passed to a neural network for training. Tokenizing is crucial to build a vocabulary used to train an LLM. 

References

• [1] Wikipedia

  • [1.1] Large langauge model

  • [1.2] Small language model

  • [1.3] Byte-pair encoding

Courses

• [C1] How AI Works - Brilliant