Sample of Advanced Data Science Coursework 

Project Overview

This data science project was my first data science and machine learning project, based on a synthetic airline flight dataset found on Kaggle at www.kaggle.com/datasets/iamsouravbanerjee/airline-dataset. The end goal was to determine which flights and/or airports experienced the most delays, so that service and operations improvements could be concentrated there. This approach was of particular interest to me due to my experience analyzing customer feedback data during my time working in customer operations support.

Data Preprocessing Pipeline

Preprocessing of the dataset was performed using the following steps:

• Removing null values and values containing "0" only and removing duplicate rows using the Pandas library in Python

• Renaming of the Airport Name feature to Departure Airport

• Adjusting the date/time format of the flight dates for consistency

• Filtering flight data by deriving the top 10 dates and airports to facilitate data plotting

 Feature Selection & Machine Learning Approach

Feature selection was performed using the SelectKBest and chi2 tools (for chi-square analysis) within the Scikit-learn library in Python with an 80/20 test-train split. Features with low value as indicated from the chi-square feature selection (such as passenger name) were removed from the dataset.

K-means clustering was used as a machine learning method to separate flights into 3 distinct status clusters:

• Cluster 0: Cancelled flights

• Cluster 1: On-time flights

• Cluster 2: Delayed flights

Exploratory Data Analysis

Multiple histograms were calculated as part of the exploratory data process to compare flights by flight status, airport, and month using the Matplotlib library in Python. Please refer to the example below.

The visualisation below shows flight status in aggregate for the top 10 departure airports. This reveals that Santa Rosa Airport and Santa Maria Airport had the most delays, while Mae Hong Song had the most cancelled flights.

 Skills Demonstrated

• Data preprocessing and cleaning

• Feature selection and engineering

• K-means clustering implementation

• Temporal  and categorical data visualization

• Business analysis and recommendations

• Python libraries: Pandas, Scikit-learn, Matplotlib, Seaborn

Course Information

Advanced Data Science COMP11068
University of the West of Scotland
May 2023