Hands-on-Lab Practise

The code is described step by step. Follow each steps properly.

If you are facing code indentation issues while copying and pasting, you may follow the code from google coolab.

Follow the code for better understanding: coolab link

Google Coolab accessing

Copy and paste the following link to open google colab

https://colab.research.google.com/notebooks/welcome.ipynb

Then click File --> New notebook after that Click Runtime-->Changes runtime to T4 GPU

Code parts begins

To install all these libraries, copy the code to your coolab cell.

Code

!pip install nltk sklearn gensim

Applications of these libraries

nltk (Natural Language Toolkit): It provides tools for text processing tasks.

gensim: This library focused on topic modeling and document similarity analysis .

pandas: Offers powerful tools for data manipulation and analysis.

sklearn: Provides machine learning algorithms and utilities for model training and evaluation.

Segmentation

CODE

import nltk

nltk.download('punkt')

from nltk.tokenize import sent_tokenize

text = "Stay focused at classtime. Always cover up pre-lab materials."

sentences = sent_tokenize(text)

print(sentences)

nltk.download('punkt') : Downloads the Punkt models for segmenting sentences and tokenizing text.

from nltk.tokenize import sent_tokenize : It helps to split text to sentence.

text: A sample text string to be segmented.

sentences = sent_tokenize(text): Segments the text into a list of sentences and stored at sentences

This is a crucial step in NLP as it helps to understanding the structure .

Tokenizer

CODE

from nltk.tokenize import word_tokenize

text = "Stay focused at classtime. Always cover up pre-lab materials."

words = word_tokenize(text)

print(words)

import word_tokenize : Which splits sentences into individual words.

After that our sample text is tokenized

Remove stop words

Removing stop words is crucial in text analysis as it eliminates common but insignificant words ("and," "the," "is") that don't contribute meaningful information. By focusing on the remaining terms, we can better understand the core content and patterns within the text.

CODE

from nltk.corpus import stopwords

from nltk.tokenize import word_tokenize

nltk.download('stopwords')

stop_words = set(stopwords.words('english'))

text = "Stay focused at classtime. Always cover up pre-lab materials."

words = word_tokenize(text)

wor = [word for word in words if word.lower() not in stop_words]

print(wor)

import stopwords: That will help to eleminent eliminates common but insignificant words

nltk.download('stopwords') : Downloads the stopwords corpus.

stop_words = set(stopwords.words('english')): Retrieves and converts the list of English stopwords into a set for faster lookups.

word_tokenize used for make token the sample text.

After that, it filter out all the matched stop words and stored the filtered output in "wor"

Stemming

Stemming is a critical preprocessing step in many natural language processing (NLP) tasks due to its ability to reduce words to their root form, which helps in normalizing and simplifying the text data.

CODE

from nltk.stem import PorterStemmer

stemmer = PorterStemmer()

text = "Stay focused at classtime. Always cover up pre-lab materials."

words = word_tokenize(text)

stemmed_words = [stemmer.stem(word) for word in words]

print(stemmed_words)

import PorterStemmer: Imports the PorterStemmer class, used for reducing words to their root form.

PorterStemmer(): Creates an instance of the PorterStemmer.

Lemmatization

Lemmatization reduces words to their base or root form by considering their context and part of speech. This process helps normalize text, making different forms of a word consistent.

CODE

import nltk

from nltk.tokenize import word_tokenize

from nltk.stem import WordNetLemmatizer

nltk.download('punkt')

nltk.download('wordnet')

lemmatizer = WordNetLemmatizer()

text = "The cats are running and jumping. The cat was tired but kept running."

lemma_process = [lemmatizer.lemmatize(word.lower()) for word in word_tokenize(text)]

Reassemble= ' '.join(lemma_process)

print("\nOriginal text:\n", text)

print("\n\nLemmatized text:\n", Reassemble)

print('\n\n')

Import Libraries: Imports necessary functions from NLTK for tokenizing and lemmatizing text.

WordNetLemmatizer : Used for Lemmatizing words.

lemma_process: Tokenizes the text into words, converts each to lowercase, and applies lemmatization.

Reassemble: Joins the lemmatized words back into a single string.

Text to Numeric [Techniques]

Converting text to numeric formats is essential for enabling machines to process and analyze textual data.

Bag of Words (BoW)

CODE

from sklearn.feature_extraction.text import CountVectorizer

corpus = ['Stay focused at classtime.', 'Always cover up pre-lab materials.']

vectorizer = CountVectorizer()

X = vectorizer.fit_transform(corpus)

print(vectorizer.get_feature_names_out())

print(X.toarray())

import CountVectorizer : Which converts a text input to a matrix of token counts.

corpus is our test data.

vectorizer.fit_transform(corpus) : Fits with input text and transform it into a document-term matrix.

get_feature_names_out: Get the feature names from input text. [Vocabulary]

TF-IDF

CODE

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer()

corpus = ['Stay focused at classtime.', 'Always cover up pre-lab materials.']

X_tfidf = tfidf_vectorizer.fit_transform(corpus)

print(tfidf_vectorizer.get_feature_names_out())

print(X_tfidf.toarray())

import TfidfVectorizer : Converts a collection of text documents to a matrix of TF-IDF features.

tfidf_vectorizer.fit_transform(corpus): Fits with text and transforms it into a TF-IDF matrix.

Word Embedding

CODE

from gensim.models import Word2Vec

from nltk.tokenize import word_tokenize

sentences = [['Stay', 'focused', 'at', 'classtime'],

['Always', 'cover', 'up', 'the', 'prelab', 'materials']]

model = Word2Vec(sentences, min_count=1, workers=4)

word_vectors = model.wv

# Accessing the vector for a specific word

vector = word_vectors['classtime']

print(vector)

import Word2Vec: For word embeddings.

min_count=1 : Sets the minimum word frequency for training;

workers=4: Uses 4 CPU cores for faster parallel training.

OneHotEncoder

CODE

from sklearn.preprocessing import OneHotEncoder

import numpy as np

categories = [['apple'], ['banana'], ['apple'], ['orange']]

encoder = OneHotEncoder(sparse=False)

one_hot_encoded = encoder.fit_transform(categories)

print(encoder.categories_)

print(one_hot_encoded)

import numpy as np: It is often used for numerical operations.

OneHotEncoder(sparse=False): Creates an instance of OneHotEncoder with sparse=False to return a dense array.

one_hot_encoded = encoder.fit_transform(categories): Fits the encoder to the categories and transforms them into a one-hot encoded array.

These are crucial preprocessing steps for advancing NLP tasks.

Page updated

Google Sites

Report abuse