Data Prep & Code

Data Preparation and Exploration

Our dataset, sourced from historical weather records, underwent meticulous preparation to ensure it met the requirements for a supervised learning task. Initially, the dataset featured various atmospheric conditions, including temperature, humidity, and wind speed. Our objective was to predict rainfall based on these conditions. To achieve this, the 'Precip Type' column was encoded into binary form (1 for 'rain', 0 otherwise), providing us with a clear target variable for classification.

Data Cleaning Highlights:

- The 'Precip Type' column, indicating rain, was transformed into a binary target variable for clarity.

- Selection of features was based on their relevance to predicting rainfall, focusing on numerical columns like temperature and humidity for the Naïve Bayes model.

Splitting the Dataset:

- The data was divided into training (80%) and testing (20%) sets to ensure our model could be validated against unseen data, a crucial step in assessing its predictive accuracy and generalizability.

In the Naive Bayes implementation conducted on the weather history dataset, the test-train split was created using the train_test_split function from the sklearn.model_selection module. This function segregates the dataset into two parts: one for training the model and the other for testing its performance. The split was specified to allocate 80% of the data for training and 20% for testing, which is a common practice in machine learning to balance between having enough data for learning and enough data to validate the model's predictive power on unseen data.

How the Test-Train Split Was Created:

The test-train split was created with the following key parameters:

• Data features and target variable: The features (e.g., temperature, humidity, and wind speed) were used as inputs (X), and the target variable (e.g., binary encoded 'Precip Type') was used as the output (y).

• Test size: Set to 0.2, indicating that 20% of the data was reserved for the test set, and consequently, 80% was used for training.

• Random state: A seed was provided to ensure reproducibility of the split. This means that each time the code is run, it generates the same split of data, aiding in consistent model evaluation.

Importance of Creating a Disjoint Split:

Creating a disjoint split between the training and testing sets is crucial for several reasons:

• Model Evaluation: It allows for an unbiased evaluation of the model. The model is trained on one set of data (training set) and tested on a completely unseen set of data (testing set). This process helps in assessing how well the model generalizes to new, unseen data.

• Overfitting Prevention: By keeping the training and testing data separate, we can prevent the model from overfitting. Overfitting occurs when a model learns the training data too well, including its noise and outliers, to the point where it performs poorly on any new data. A disjoint split ensures that the model's performance on the test set is a genuine measure of its ability to generalize.

• Model Tuning: The test set serves as a final check before deploying the model in the real world. It allows data scientists to compare different models or configurations and select the one that performs best on the test set. This selection process is crucial to ensure the model's reliability and effectiveness in making predictions on real-world data.

• Confidence in Model's Predictive Power: A disjoint split gives stakeholders confidence in the model's predictive power. It demonstrates that the model can perform well on data it hasn't seen during training, which is similar to how it will be used after deployment.