Project Overview

I performed data transformation and exploratory analysis on a super messy survey dataset titled "Ask a Manager Manager Salary Survey 2021 (Responses) with Python. Additionally, I utilized Power BI for visualization. I embarked on this personal project to have experience cleaning real-world datasets and to have insight into salary disparities across industries, age groups, work experience, education qualification, gender, and races.

Dataset Description

The dataset, titled "Ask a Manager Salary Survey 2021 (Responses)," is a real-world collection of survey data featuring 17 variables and over 28,000 rows focused on salaries and workplace demographics. Sourced from AskAManager.org, the data primarily reflects responses from the United States, with additional inputs from international participants.

• 'Gender' tells us the gender of workers.

• 'Industry' tells us the industry of the workers.

• 'Age Groups' tells us the age group of the workers.

• 'Work Year Experience' tells us the year of experience of the workers.

• 'Location (Country, State, City)' tells us the location of workers.

• 'Salary' tells us the Annual salary of workers.

• 'Education' tells us the highest Education qualifications of workers.

• 'Monetary compensation' tells us the bonuses or overtime in an average year.

• 'Race' tells us the race of the workers.

• 'Job Title' tells us the job title of the workers.

Tools Used

• Python, Jupyter notebook (Cleaning and exploratory analysis)

• Libraries: Pandas, Numpy, matplotlib and, Seaborn.

• Power BI (Data Visualization)

Key Question(s) for Analysis

• How does age group influences salary?

• Which industry pay the most?

• Which Educational Qualification earns the most salary?

• What is the most popular industry?

• How does gender vary by educational qualification?

• How do work year experience influences salary?

Data Wrangling and Steps

• Imported Libraries needed like seaborn, pandas, numpy, and matplotlib.

• Handling Encoding Errors: The first hurdle was an encoding issue. I resolved it by using the latin-1 encoding to ensure proper reading of the data.

• Assessed the data programmatically using the info() function.

• Renaming Irregular Columns: The column names were inconsistent, so I standardized them using Python’s rename() function with a dictionary argument. Additionally, I applied the lower() function to make them lowercase and strip() to remove trailing spaces.

• Renamed Rows:

  • Country column: Renamed entries like US, USA, United statew, America to United States and       UK, Uk to United Kingdom for clarity using the replace function in pandas.

  • Race Columns: I regrouped the rows to multiracial, Biracial, and moniracial for easy                   readability. Categories with one race was named monoracial, Categories with two races was named Biracial, and categories with more than two races was renamed multiracial.

  • Gender: Renamed rows with entries like Other or prefer not to answer and Prefer not to answer to 'No answer'. 

  • Industry: Replaced Computing or Tech with 'Science & tech', Education (Higher Education) with  'Education', Government and Public Administration with 'Public Administration', Nonprofits with    'NGO', Accounting, Banking & Finance with 'Banking & Finance', Regulatory Affairs- nutraceuticals   and 'Regulatory Affairs', Education (Primary/Secondary)' with Education', and Manufacturing :       corporate admin support with 'corporate admin support'.

  • Recategorized some rows in my work experience column for easy understanding e.g        41yearsormore to 41years+ using the replace function.

• Changed data types:

  • Changed the salary column from string to float using replace function to remove the commas. Then I converted all the salary to USD using the lambda function.  Then I filled in null values with the mean before converting the data type from f to integer. 

  • The monetary compensation, I converted it to USD using the current exchange rate, using lambda function. Then I also filled in null values with the mean before converting to integer.

• Handled Outliers:

  • I handled outlier in the salary column by filtering the datasets by lower bound and upper bound.

• Deleted columns

  • Removed unnecessary columns in my data frame such as Unnamed: 18, Unnamed: 19, Unnamed: 20, Unnamed: 21, Unnamed: 22, Unnamed: 23, additional_context_income, How many years of professional work experience do you have overall?, and State which was not needed for my analysis.

Education vs average salary

Education vs Gender

Data Visualization with Power BI

• Created a dashboard to have more insight into salary by Education, Work Experience, Age group. and industry.

• I added Slicer like Race categories and gender.