Embedded Files
We employed fine-tuned RoBERTa model to effienctly classify ["ham" or "spam"] our Email.csv dataset.
By the end of this learning module, students will be able to:
Why does NLP need to know.
How does NLP works.
Filter out spam email.
Terms of Transformer model.
What is Pre-Trained RoBERTa.
Effctive uses of RoBERTa for text classification.
Mathemetical concepts of RoBERTa.
Why NLP ?
Natural Language Processing (NLP), is a branch of technology that focuses on how computers can understand and interact with human language. Imagine talking to your phone or typing a question into Google NLP is what makes it possible for those systems to understand and respond to what you’re saying. It has importance in software applications, and its common uses (e.g., in email platforms, voice-based assistants, search engines) . This understanding allows it to do things like find spam emails, understand emotions in text, and translate languages more accurately.
NLP's working stages
Human language can be broken down into four main parts to ensure NLP applications to work effectively.
phonemes (basic sounds),
morphemes and lexemes(word parts and meanings),
syntax (sentence structure), and
context (meaning in situations).
Why email security concern!
Filtering out spam and scam emails using NLP is important because it helps protect your inbox from unwanted or harmful messages. By identifying and blocking these emails, NLP keeps your inbox organized and reduces the risk of falling for scams, ensuring that you only see the emails that matter. Key points often defined a email to be spam and those are:
Text Analysis.
Content Structure.
Contextual Analysis.
Transformer model
A transformer model is a neural network that learns context and meaning by tracking relationships in sequential data, like the words in this sentence. It has Self-Attention layer which helps the model to understand different words in the text.
Pre-trained RoBERTa
Figure 1: RoBERTa class labeling process.
The pre-trained RoBERTa model is a transformer-based language model that has been trained on large corpus of text data to understand human-like language. Pre-trained means the model first learns to predict missing words and identify real with fake sentences,
After pre-training, the model can be fine-tuned on specific dataset like text classification, using what it has learned to perform well even with less data. For the reason, pre-trained RoBERTa model highly effective for various NLP tasks.
RobertaForSequenceClassification
This diagram shows a model that learns from both the whole sequence and individual tokens. The encoder creates and refines token representations, using sequence-level loss for overall context and token-level loss for precise details. This dual approach helps the model be accurate and generalize well.
Figure 2 : Multi-level learning process of RoBERTa
RoBERTa for spam email detection
RobertaForSequenceClassification is effective for email classification because it understands text context well, thanks to its RoBERTa backbone. RoBERTa is pre-trained on vast amounts of text, allowing it to learn language patterns.
When fine-tuned on specific email data, it adapts to classify emails accurately. This combination of strong pre-training and fine-tuning makes it a great choice for categorizing emails.
Mathemetical concepts of RoBERTa
RoBERTa utilizes the transformer architecture so it is relies heavily on the self-attention mechanism. First equation is used to compute attention score .
1. Where Q (query), K (key), and V (value) are derived from the input embeddings, and dk is the dimensionality of the key vectors.
2.Each transformer layer uses a feedforward network for nonlinear transformation of attention outputs. where W1, W2 are weight matrices, and b1 and b2 are bias vectors.
3.where E is the input embedding, T is the token embedding, P is the positional embedding, and S is the segment embedding.
4. For text classification tasks The output logits are calculated.
where h is the pooled output from the last transformer layer, and W and b are the weights and biases of the classification layer.
5.The model is trained with cross-entropy loss, which measures how close the predicted probabilities are to the true labels.
where yi is the true label and y^i is the predicted probability for class i.
About Dataset
The analyzed dataset is sourced from a paper and includes balanced binary target classes. In this dataset, the target classes are labeled as "Ham" (not-spam) and "Spam."
The dataset contains 301 instances labeled as "Ham" and 301instances labeled as "Spam."
The dataset includes two features: "Message," which represents personal and informal conversations, and "Category," which labels the type of message.
Dataset can be found at the LINK
Page updated
Google Sites
Report abuse