Deep Dive in Machine Learning with Python
Part — XV: Initial Data Analysis — II
Welcome to another blog of Deep Dive in Machine Learning with Python. In the previous blog, we worked with the Autism Spectrum Disorder (Children) dataset to understand how to effectively perform the first two stages of Initial Data Analysis.
In today’s blog, we will continue with the further stages of IDA and fill the MISSING values in ‘ETHNICITY’ and ‘WHOS_COMPLETING_TEST’. Also, build our first Machine Learning Regression model to predict the missing values in ‘AGE’.
First-hand cleaned DataFrame
Step-3: Fill Missing Values
Step-3.1: Fixing Missing values in ‘Ethnicity’
Ethnicity count distribution where Country is ‘Jordan’
Ethnicity count distribution where Country is ‘Egypt’
Ethnicity count distribution where Countries are ‘Qatar’, ’Saudi Arabia’, ’Russia’ and ‘Pakistan’
Ethnicity count distribution where Countries are ‘Syria’, ’United Arab Emirates’, ’Lebanon’ and ‘Libya’
Ethnicity count distribution where Countries are ‘China’, ’Kuwait’, ’Iraq’, ’Latvia’, ’Austria’ and ‘Malaysia’
Here, we explored people of which ethnicities from the above countries participated in this dataset. Now, by seeing the trend we will replace the missing value with the dominant ethnicity.
Step-3.2: Fixing Missing values in ‘Whos_completing_test’
In this feature, we have a paramount difference between ‘Parent’ and the rest of the categories. So, labeling ‘Parent’ as 1 and other categories as 0.
Step-4: Drop not-required columns
In this step, we will remove the Columns from the dataframe which have missing values and we have already fixed them by creating their new columns without any missing value.
Step5: Convert categorical data into numerical form
In this step, we will convert the qualitative data in ‘Ethnicity’ and ‘Country’ into numerical values.
Step-5.1: ETHNICITY column in numerical form
Here, you can also use LabelEncoder to perform labeling, however, I defined the mapping in a dictionary to explain things in an easy to apprehend manner.
Step-5.2: COUNTRY column in numerical form
As in the ‘Country’ column, we have differences amongst the categories counts, hence, bucketed all the countries with a count of ≤ 7 in a single class and labeled as 9.
Step-5.3: AGE_DESC column in numerical form
All the rows in AGE_DESC marked as 0.
Step6: ML Model to predict the missing values in the AGE variable
Here, we divided the DataFrame into two sections:
- Test Records where AGE is 0
- Train Records where AGE is not 0
First, we checked the dimensions in Parent, Train & Test records DataFrames. Then, we instantiated the RandomForestRegressor and trained it on the subset(created using train_test_split) of Train Records DataFrame.
Here, the model predicted the age for subset records. Then, we ran it on the Test DataFrame which contains records where age is 0.
And, we got the values that can replace the missing data in ‘AGE’.
Congratulations, we come to the end of this blog. To summarize, we covered the remaining stages of Initial Data Analysis (IDA).
Follow to get notified for the upcoming posts where we will work on building the understanding around Exploratory Data Analysis using an example dataset.
If you want to download the Jupyter Notebook of this blog, then kindly access below GitHub repository:
Thank you and happy learning!!
