Hands-on-Lab Practise

The code is described step by step. Follow each steps properly.

Google Coolab accessing

Copy and paste the following link to open google colab

https://colab.research.google.com/notebooks/welcome.ipynb

Then click File --> New notebook after that Click Runtime-->Changes runtime to T4 GPU

Code parts begins

Import libraries

CODE

import pandas as pd

import numpy as np

import tensorflow as tf

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Dense, Embedding, Flatten

from tensorflow.keras.preprocessing.text import Tokenizer

from tensorflow.keras.preprocessing.sequence import pad_sequences

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import LabelEncoder

from sklearn.metrics import accuracy_score

pandas : This function is for working with structured data (csv files)

numpy : Provides support for large, multi-dimensional arrays, matrices and mathematical operation on them.

tensorflow : A deep learning framework used to build and train neural networks

sequential and dense: To define the neural network model architecture, with layers arranged sequentially.

Tokenizer: For converting text into sequences of tokens (numbers) and ensure uniform length.

Upload dataset

Click the icon to upload mail.csv dataset to your coolab server.

df = pd.read_csv('/content/mail.csv')

X = df['Message'].values

y = df['Category'].values

and then run the code

A short preprocessing

CODE

tokenizer = Tokenizer(num_words=10000)

tokenizer.fit_on_texts(X)

X = tokenizer.texts_to_sequences(X)

X = pad_sequences(X, maxlen=100)

label_encoder = LabelEncoder()

y = label_encoder.fit_transform(y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

The tokenizer used to convert the text into sequences of integers, where each integer represents a word based on its frequency in the datase and the target labels are encoded into numerical values using a label encoder.

Linear Activation function

Simply outputs the input directly without any transformation andand is ideal for neural network models when predicting a continuous value.

CODE

model_linear = Sequential([

Embedding(input_dim=10000, output_dim=128, input_length=100),

Flatten(),

Dense(64, activation='linear'),

Dense(1, activation='sigmoid')

])

model_linear.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

model_linear.fit(X_train, y_train, epochs=3, batch_size=64, validation_split=0.2, verbose=0)

y_pred_linear = (model_linear.predict(X_test) > 0.5).astype(int)

accuracy_linear = accuracy_score(y_test, y_pred_linear)

print(f"Linear Activation - Accuracy: {accuracy_linear:.4f}")

This code defines and trains a neural network model using a linear activation function in one of its dense layers. First the input is flatten and passed to a dense layer with a linear activation function, followed by an output layer with a sigmoid activation as we have binary class. Linear Activation - Accuracy: 0.9174

Sigmoid Activation function

Sifmoid makes it ideal for output layers where the goal is to predict the probability of a particular class.

CODE

model_sigmoid = Sequential([

Embedding(input_dim=10000, output_dim=128, input_length=100),

Flatten(),

Dense(64, activation='sigmoid'),

Dense(1, activation='sigmoid')

])

model_sigmoid.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

model_sigmoid.fit(X_train, y_train, epochs=3, batch_size=64, validation_split=0.2, verbose=0)

y_pred_sigmoid = (model_sigmoid.predict(X_test) > 0.5).astype(int)

accuracy_sigmoid = accuracy_score(y_test, y_pred_sigmoid)

print(f"Sigmoid Activation - Accuracy: {accuracy_sigmoid:.4f}")

Sigmoid Activation - Accuracy: 0.7934

Tanh Activation function

Tanh (hyperbolic tangent) activation function, which outputs values in the range of -1 to 1, is particularly effective in scenarios where the data is centered around zero. Additionally, tanh tends to work well when the network architecture benefits from non-linear transformations that can enhance the learning of complex patterns.

Code

model_tanh = Sequential([

Embedding(input_dim=10000, output_dim=128, input_length=100),

Flatten(),

Dense(64, activation='tanh'),

Dense(1, activation='sigmoid')

])

model_tanh.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

model_tanh.fit(X_train, y_train, epochs=3, batch_size=64, validation_split=0.2, verbose=0)

y_pred_tanh = (model_tanh.predict(X_test) > 0.5).astype(int)

accuracy_tanh = accuracy_score(y_test, y_pred_tanh)

print(f"Tanh Activation - Accuracy: {accuracy_tanh:.4f}")

Tanh Activation - Accuracy: 0.8264

ReLu Activation function

It transforms the input x to max⁡(0,x) which means it passes positive values through while setting negative values to zero. The non-linearity introduced by ReLU enables the network to learn complex patterns without the added computational cost of more complex activation functions.

CODE

model_relu = Sequential([

Embedding(input_dim=10000, output_dim=128, input_length=100),

Flatten(),

Dense(64, activation='relu'),

Dense(1, activation='sigmoid')

])

model_relu.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

model_relu.fit(X_train, y_train, epochs=3, batch_size=64, validation_split=0.2, verbose=0)

y_pred_relu = (model_relu.predict(X_test) > 0.5).astype(int)

accuracy_relu = accuracy_score(y_test, y_pred_relu)

print(f"ReLU Activation - Accuracy: {accuracy_relu:.4f}")

ReLU Activation - Accuracy: 0.8017

GELU activation Function

GELU combines the advantages of both linear and non-linear activation functions. This design allows GELU to introduce a probabilistic non-linearity that can capture complex patterns more effectively.

CODE

model_gelu = Sequential([

Embedding(input_dim=10000, output_dim=128, input_length=100),

Flatten(),

Dense(64, activation=tf.keras.activations.gelu),

Dense(1, activation='sigmoid')

])

model_gelu.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

model_gelu.fit(X_train, y_train, epochs=3, batch_size=64, validation_split=0.2, verbose=0)

y_pred_gelu = (model_gelu.predict(X_test) > 0.5).astype(int)

accuracy_gelu = accuracy_score(y_test, y_pred_gelu)

print(f"GELU Activation - Accuracy: {accuracy_gelu:.4f}")

Tanh Activation - Accuracy: 0.8264

These are crucial preprocessing steps for advancing NLP tasks.

Page updated

Google Sites

Report abuse