Sample of Master's Thesis Project - Machine Learning and Artificial Intelligence Techniques Used

Project Overview

My master's thesis project was based on a dataset derived from online drug reviews from contributors to the drugs.com website (www.drugs.com) which was formerly located at the UC Irvine Machine Learning Repository website. The goal of my research was to demonstrate the efficiency of legacy machine learning methods versus neural networks and LLMs (large language models) for identifying adverse drug reactions for new medications introduced to the prescription drug market.

Data Preprocessing Pipeline

The following steps were taken to create a data processing pipeline in Google Colab Notebooks:

• Removal of null and duplicate values and substituting NaN values with blank values using the Pandas library in Python

• Changing all text to lower case

• Removing special characters

• Removing stopwords using the NLTK library in Python

• Removing features such as health condition and adding features such as category (used for sentiment analysis)

• Using the TF-IDF Vectorizer tool from the NLTK library in Python for feature selection ranking along with Textblob and VADER libraries in Python for ranking review sentiment

• Creating new datasets via the Pandas library in Python for ingestion by the models

Methodology: Two-Stage Approach

Stage 1: Sentiment Classification of Drug Reviews

I implemented and compared multiple model architectures:

Legacy Machine Learning Models

• Random Forest (scikit-learn)

• Multinomial Naive Bayes (scikit-learn)

Neural Network Models

• Convolutional Neural Networks (CNN) using Keras

• Long Short-Term Memory networks (LSTM) using Keras

Large Language Models

• RoBERTa (HuggingFace)

• DeBERTa (HuggingFace)

All models underwent appropriate training techniques:

• Legacy and neural models: K-fold cross validation with hyperparameter tuning

• LLM models: Fine-tuned using zero-shot prompting with additional training using HuggingFace Evaluate with an 80/20 train/test split

After evaluation across accuracy, precision, recall, and F1 metrics, the DeBERTa model with Textblob sentiment ranking demonstrated superior performance for identifying negative reviews.

Stage 2: Adverse Drug Reaction Identification

The negatively-rated reviews from Stage 1 were then processed through a pre-trained medical LLM from the HuggingFace Model Repository. This model was fine-tuned on social media posts reporting confirmed adverse drug events. Manual annotation was performed to establish ground truth and handle ambiguous cases on the model output.

Key Results

The confusion matrix below confirms the prediction accuracy of the medical LLM after reviewing the negatively labeled drug reviews - 93 of the reviews were rated as true positives, matching the ground truth labels I assigned through manual annotation.

The final adverse drug reaction detection model achieved approximately 85% accuracy, demonstrating the potential of this approach for pharmacovigilance applications.

Skills Demonstrated

• Multi-stage ML pipeline development

• Advanced NLP techniques

• Model comparison and evaluation

• LLM fine-tuning

• Medical domain application of AI

• Data preprocessing and feature engineering

Course Information

Master's Thesis COMP11024
University of the West of Scotland
April 2024

Title: Analysing Sentiment in Prescription Drug Reviews: Developing Contextualized Machine Learning and Large Language Models to Identify Side Effects