Exploratory Data Analysis (EDA)

Project Overview- Fraud Analysis and Rules Creation

In this project, I tackled a complex fraud detection case study, focusing on analysing fraudulent transactions and creating rules to prevent financial institutions from blocking such activities on their platform. I leveraged my analytical skills and experience in data analysis to develop a comprehensive solution.

Problem Statement:  The goal here is to minimize fraud as much as possible. We want to reduce the amount of money lost to fraudulent activities. However, there's a critical constraint that no more than 15% of genuine transactions get blocked (in genuine transactions we care about the number of transactions stopped and not about the dollar value). I’ll walk you through the approach I’ve used to tackle this challenge effectively.

Constraints:

You can use up to 4 rules (see below what a rule is). 2 out of the 4 rules must contain at least 2 conditions.

Definition of "rules": A rule is made of logical conditions, examples of possible conditions are:

• Amount >= $600

• IP class B = 256.256

• Merchant = British Airways

 A rule can be composed of a few conditions with AND relationship between them. For example: (Amount>=600) 𝐴𝑁𝐷 (𝑀𝑒𝑟𝑐ℎ𝑎𝑛𝑡 = 𝐵𝑟𝑖𝑡𝑖𝑠ℎ𝐴𝑖𝑟𝑤𝑎𝑦𝑠) .

I need to prioritize the rules that I create and summarize how much fraud 4 rules can stop (in $ value) and how many genuine transactions they will stop (in % out of all overall transactions)

Dataset used: A list of 500 transactions were provided in a  Data File xlsx with 13 variables

Steps followed: 

1. Data load and inspect: I began by loading and inspecting the dataset to understand its structure, types of variables etc

2. Data cleaning: Next, I cleaned the data to ensure its quality. This involved handling missing values, trimming blank spaces, correcting inconsistencies, and ensuring that the data was in a suitable format for analysis.

3. EDA: I performed exploratory data analysis to uncover underlying patterns and insights.

4. Data Visualization: Visualising data helped me to better understand the distribution of data, relationships between variables, and potential indicators of fraudulent transactions.

5. Feature engineering: I created bunch of new features such as day , time of transaction which could be useful in identifying fraud.

6. Rules creation: Finally, I developed rules to identify and prevent fraudulent transactions. I analyzed various aspects such as merchant behavior, IP addresses, transaction timing, amount etc to create a robust set of rules.

Iterations:

To create the most efficient recommendation, I went through several iterations:

1. Initial Exploration: First, I explored the dataset and looked at various factors like merchants, IP addresses, transaction times, and amounts. My goal was to find patterns that could indicate fraudulent transactions.

2. Rule formulation and testing: I started making a list of rules for identifying fraudulent transactions. This began with extracting lists of IPs, IP country,  merchants that had no genuine transactions and only fraudulent ones.

3.  Refinement and combing transactions: Next, I combined these rules to maximize the detection of fraudulent amounts. I made combinations of various merchants and IPs and refined the rules to ensure they were independent and not overlapping incorrectly.

4. Optimization and validation: I then optimized these rules and validated their accuracy to ensure maximum fraud detection within the given constraints.

5. Final Evaluation and Selection: Finally, I finalized the rule set and evaluated its overall performance. I selected the top 4 best-performing rules. I confirmed that these rules stopped a significant amount of fraud while blocking minimal genuine transactions.

Rules Creation/Recommendation :

• 1st Rule is to block 74 merchants I have identified who have conducted 121 fraud txns and 0 genuine txns. So by blocking all these fraudulent merchants we can save upto $66,889 USD.

• 2nd Rule: (Merchant = Ebuyer UK) AND (Amount USD >= 420) is for Ebuyer UK, an electronic e-commerce retail merchant, experienced the highest number of fraudulent transactions. There is a possibility that fraudsters are conducting fake transactions on the website. Point to be noted is that all fraudulent transactions exceeded 420 USD. Therefore, I implemented a logical condition to block all such fraudulent transactions, resulting in the blocking of 10 genuine transactions in the process. So by implementing this rule we can save upto $44,914 USD

• 3rd Rule: I have identified 13 IPs through which 24 fraudulent txns are done and no genuine txns amounting to $32,011. Hence I recommend to block these IPs. I made sure that these IPs are not overlapping with all other 3 rules. Infact no rule is conflicting with other rule. I have linked of all these IPs in the next slide.

• 4th Rule: (Merchant= Watford Electro) AND (IP_B IN (87.74, 84.9, 81.148, 216.250)) is for another merchant namely Watford electro has second highest number of fraudulent txns amounting $9860 USD. Hence I took all IPs of this merchant through which fraudulent txns are done blocking 2 genuine txns in the process. 

I have priortised these rules on decreasing order of amount prevented from fraud.

Impact:

Let's talk about the financial implications. Through the implementation of the rules I developed, we achieved substantial cost savings.

1. My efforts led to the prevention of USD $153,674 in fraudulent transactions across the four rules that were created.

2. Secondly, our approach resulted in the successful blocking of 78.6% of identified fraudulent transactions. This showcase the effectiveness of our strategies in mitigating financial risks.

3. While it's crucial to stop fraud, we also made sure that genuine customers aren't negatively affected. I'm pleased to inform that only 2.4% of genuine transactions were blocked.

4. Lastly, it's important to highlight that all predefined conditions for rule creation and constraints were met

For a detailed walkthrough of the methodology, findings, and recommendations, please view my FRAUD ANALYSIS CASE STUDY deck

Project 2

Project Overview- Employee Attrition Analysis

Problem Statement:

Here, we are have a small dataset that has stats about employees. I will be performing EDA on this dataset to understand:

• Target variable (whethe there is attrition or not)

• What are the factors contributing attrition

Dataset Used: Job Attrition on Kaggle

We are given about 1500 samples out of which ~1300 samples were not attrition and only 200 samples showed attrition. Initially in order to get an idea, on what factors an employee attrition depends, I plotted few bar plots to check if there is any relationship between attrition and-

• Gender

• Employees doing Overtime

• Job role 

• Marital status

• Total working years

• Performance rating

• Department 

• Education

• Business travel

• Education field

• years at company

• Work life balance

• Number of companies worked 

• Age

I discovered several factors, such as work-life balance, business travel, and employee age, which provide insights into average attrition rates. Employees facing challenges with work-life balance and frequent travel are more likely to experience attrition. Additionally, those aged between 18-28 and 56 and above tend to have higher attrition rates.

Steps followed as part of EDA

🔍 Step 1: Load and Inspect Data

Utilized Python with Pandas, NumPy, Matplotlib, and Seaborn to load and visualize the dataset. Peek at a few rows to understand the data's structure. Checked nulls/duplicate data if any.

🧹 Step 2: Data Cleaning

Dealt with missing and duplicate values strategically: Remove/ fill / drop them using relevant methods. I have replaced them with 0's here.

📈 Step 3: EDA and visualization

Visualized target variable distribution with box plots. Spot and investigate intriguing outliers. Uncovered insights from all the variables by checking relationship between the variables using regplot.

💡 Step 4: Feature engineering

This involved the process of selecting, modifying, or creating new features from existing dataset that are relevant and useful for a machine learning task.

🎯 Step 5: Outlier Handling

Checked % of outliers in the variables using Z score and remove/replace outliers based on domain knowledge and analysis impact. Outliers can be either replaced with mean/median/mode or 0's. In my dataset it does not make any sense to use mean/median/mode hence I replaced with 0's.

🔤 Step 6: Handling Categorical Data

Machine learning algorithms work with numerical data, so categorical variables need to be transformed into numerical representations. This can be done through one-hot encoding, label encoding, or other methods. In our dataset there is no categorical variable.

🤖 Step 7: Model Building

Trained the model by dividing dataset into Training and Testing data. I have used 33% of the dataset as test data. Fit the model and predict the value for variable "Income_per_acre" .Check the accuracy of the model and if it can be improved further by normalizing dataset

📐 Step 8: Data Normalization

Using Standardscalar/ MinMaxscalar technique normalised data to ensure that all features have a similar influence on the machine learning algorithm, preventing certain variables from dominating the model simply because they have larger scales or units. Normalize dataset and re run the model to check if accuracy of model changes positively.