Pre-Lab

By the end of this learning module, students will be able to:

What is Autoencoder.
What is WordNet
Ways of word to numeric.
What is Word2Vec, working process and optomizations.
Terms of Autoencoder.

Word Transformation Techniques

Autoencoder?

An autoencoder is a type of neural network designed to learn efficient representations of data. It consists of two main parts:

An Encoder: which compresses the input data into a lower-dimensional, dense representation
A Decoder: which reconstructs the original data from this compressed form.

The network is trained to minimize the difference between the original data and its reconstruction, allowing it to learn meaningful features and patterns in the complex data.

WordNet

WordNet is a large lexical database of English words that groups them into sets of synonyms called synsets which helps in understanding the relationships between words. WordNet helps computers interpret the nuances of language, such as context and word similarity so that the computational time is going short. Following figure respresinting some working phases of WordNet.

Flaws : Despite its usefulness, WordNet has some limitations. It primarily focuses on English and may not cover all domain-specific terms. Additionally, WordNet's fixed vocabulary can be less adaptable to new words and evolving language trends. It also can't compute accurate word similarity as well.

Word to Numeric conversation

one-hot vector

Words are often represented by one-hot vectors, where each word is encoded as a unique binary vector with a single 1 and the rest 0s. These vectors are mainly orthogonal position based, meaning they are at right angles to each other and do not share common dimensions. This makes it challenging to measure similarity or combine information from both vectors.

Example: Let's consider a search for "dog park":

If we represent "dog" and "park" with one-hot vectors:

"dog" might be represented as [1, 0, 0]
"park" might be represented as [0, 1, 0]

One-hot encoding representing it as different words for its dimensions. They only indicate that "dog" and "park" are different words. So the meaning within a sentence can be changed. As it is a great problem, hence here we have a sollution as well.

Some enrich sollution

Distributed Representations

Distributed representations means creating numerical vectors for words that capture their meanings based on their context. Instead of treating words as isolated, we model how each word depends on the surrounding words. Words that appear in similar contexts will have similar vector representations, reflecting their relationships and meanings more effectively.

For example : The cat sat on the mat.

In this context, the word "cat" is closely related to "sat" and "mat." Hence, the vector like [0.2, 0.7, -0.3], and "mat" might be [0.1, 0.6, -0.2]. Which reflecting their relationship.

Distributional Semantics

Distributional semantics works by analyzing the contexts in which words appear together to derive their meanings. Instead of treating words as isolated units. This approach assumes that words frequently found near each other in text share similar meanings

Example: The words "king" and "queen" often appear in similar contexts, like "royal," "throne," or "palace." Hence it provides close vectorization for same context.

Word Embedding

Word embedding is a technique used in natural language processing to represent words as numerical vectors. These vectors capture the meaning of words based on their context and relationships with other words. It provide dense, lower-dimensional vectors that reflect semantic similarities.

For example, words with similar meanings or that appear in similar contexts have similar close vector representations, making it easier for machines to understan .

Word2Vec

The Word2Vec algorithm trains a simple neural network with a single hidden layer and the focus is on learning the weights of the hidden layer. These weights represent the word vectors or embeddings. It has a self-supervised algorithm, where it doesn't require manually labeled data—making it efficient and scalable for large text. In the training section, W2V learns the frequency with which certain word pairs appear together in a text. During training, the model sees numerous examples of common pairs and allowing it to learn strong associations between them. If same pair of words appears in the same context, then it provide close vectors value.

Break through of W2V

Here our input text is "The quick brown fox jumps over the lazy dog ".

In W2v, we need to set window size which typically covering a few words before and after the target word. In here a window of size 2 is used.

For example, with "quick" as the target word and a context window of 2, the model creates pairs like ("quick", "the"), ("quick", "brown"), and ("quick", "fox").

Also focus all the "Training Samples", only ("quick", "brown") or ("brown", "quick") repeating more than once which mean these pairs of words have high relationship among others. So if the same pairs appear in the context then it provide close vectors value.

Pairs for ("quick", "brown") Pairs for("brown", "quick")

Vector for "quick": [0.2, 0.8, 0.5] Vector for "brown": [0.4, 0.9, 0.3]

Vector for "brown": [0.4, 0.9, 0.3] Vector for "quick": [0.2, 0.8, 0.5]

Given a window size of 2, can you identify which word pairs might have a strong relationship?

Optimization of W2V model

Suppose we have we had word vectors with vocabulary of 10,000 unique words and 300 features. In w2v, neural network has two weight matrices: a hidden layer and an output layer. Both layers would have a weight matrix with 300 x 10,000 = (3 million weights each ). Training such a large model using gradient descent can be slow and requires a massive amount of data to avoid overfitting. Although computing power has improved since 2013, optimizing and training a model with millions of weights and billions of samples remains challenging. Hence we have Optimazations for W2Vec for effective performances.

Subsampling:

Frequent words like "the," which appear in many contexts and offer little meaningful information. By reducing the frequency of such common words, the model focuses more on informative word pairs. The method involves removing words based on their frequency, with higher-frequency words being more likely to be excluded from the training data. The 'sample' parameter controls this subsampling. A smaller sample rate means frequent words are less likely to be kept in training data. Here👉

Wi is the word, z(Wi ) is the fraction of the total words and P(Wi) is the probability of keeping the word

Negative sampling:

Negative sampling improves training efficiency by updating only a small subset of weights for each training sample. Instead of adjusting all weights, the model updates the weights for one "positive" word and a few "negative" words (like 5), which helps manage large vocabularies. For each training pair like ("fox", "quick"), we update the weight for "quick" and the weights for 5 randomly selected negative words. This reduces the number of weight updates to just a small fraction of the total, making training faster and more manageable. Negative samples are chosen based on their frequency, with more common words being more likely to be selected as negatives.

AutoEncoder Model

Concept: In an autoencoder model for natural language processing, words are initially represented as one-hot vectors, where each word in a vocabulary of 10,000 unique words is encoded with a vector that has 10,000 components, with only one component active. The model consists of an encoder and a decoder. The encoder compresses these high-dimensional one-hot vectors into a lower-dimensional, dense representation that captures the word's meaning in relation to other words. The decoder then reconstructs the original one-hot vector from this compressed form, outputting a probability distribution over the vocabulary to indicate which words are likely to be nearby or contextually related. This structure enables the model to learn and represent word relationships effectively by understanding and encoding similarities between words.

Hidden Layer for AutoEncoder

A hidden layer in an AutoEncoder is a crucial component that captures the essential features of the input data by reducing its dimensionality. Positioned between the input and output layers, the hidden layer typically has fewer neurons than the input layer, forcing the model to learn a compressed representation of the data. For example:

The model learning word vectors with 300 features. The hidden layer is represented by a weight matrix with 10,000 rows (one for each word in our vocabulary) and 300 columns (one for each hidden neuron). This matrix helps the model learn a 300-dimensional vector for each word, capturing its meaning and relationships with other words where weight lookup table plays crucial.

Weight Lookup Table

The end goal of all of this is really just to learn this hidden layer weight matrix. Look at the figure 👉

When we multiply a 1x10,000 one-hot vector by a 10,000x300 matrix, the result is that the one-hot vector effectively "selects" the row of the matrix corresponding to where the "1" is. This means we directly retrieve the 300-dimensional word vector for that specific word, ignoring all the zeros in the one-hot vector. That means the hidden layer of this model is really just operating as a lookup table.

Output layer

In the output layer, the 1x300 word vector is fed into a softmax classifier. Each output neuron, representing a word in the vocabulary, generates a value between 0 and 1, with all values summing to 1. Each neuron multiplies its weight vector with the word vector, applies the exp(x) function, and then the result is divided by the sum of all outputs to ensure they add up to 1. This process allows the model to predict the probability distribution over the entire vocabulary.

All over effects of AutoEncoder model

Effectiveness of model in learning word vectors lies in its ability to capture the similarities in word contexts. If two words, like "intelligent" and "smart," frequently appear in similar contexts, the model is driven to learn similar vectors for them. In essence, the model "understands" the relationships between words based on their usage and reflects this understanding in the way it encodes them into vectors. This allows the model to capture the semantic similarities between words, even if they are not exactly the same.

Page updated

Google Sites

Report abuse