Data science has risen to prominence in the last decade due to its capabilities in predictive algorithms. While many business verticals value the benefits of predictive algorithms using Data Science, insurance companies place a lot of importance as data science and predictive algorithms helps them keeps premium low. Data is always been at the core of what insurance companies do analyzing data such as claims, what kind of a vehicle one drives, how many miles do they drive per day among other.

The data science field is gaining strength with improvements in technology, availability of statistical libraries to compute regression or classifications of data collected. Actuaries, the data scientists at insurance companies as they were called a decade ago, used to collate data from different sources and analyze the premium and claim data to identify fraudulent transactions that helped them keep the premiums low. If anything, data science technology of today has given far more tools to perform their analysis.

The data has a few ordinal, categorical data that needs to be parsed and categorized properly.

Our goal is to predict a binary outcome of 1, to indicate safe driver, or 0, to indicate that the drivers' data needs a review. We will also look at the continuous variables and fill in the missing data with the mean or median in order to not skew our results.

After cleaning up the data and filling in missing data we will look at the features and their correlation so that we can drop highly correlated data which may impact our results.

We are importing all the required packages¶

Import the data into python¶

In the above data, target is our dependent variable, which shows 1 as safe drive and 0 as not safe driver. We have other independent variables like the type of car the driver is driving, their personal and professional details.¶

Checking the descriptive statistics to see if there is any visible impossible values (like family size in negative)¶

Here we don't get to see any problem with the data

Checking is the data has any missing values¶

We get to see that there are 2 variables with missing values

Here we are checking the frequency distribution of the dependent variable to check if it's a balanced data.¶

We see it's almost a balanced data with proportion of 1 and 0 being 70% and 0% respectively.

We are now extracting the categorical variables, to transform/drop them as required.¶

Among the categorical variables we retain the following:

1. Gender
   
2. Marital_Status
   
3. Vehicle_Type, and
   
4. Age_bucket
   
   EngineHP_bucket, Years_Experience_bucket, Miles_driven_annually_bucket, credit_history_bucket have a corresponding continuous variable. Creating each with their own dummies along with the continuous variable does not make sense. We will keep the Age_bucket as there is no continuous variable to represent age.
   
   We can split the dataset by State (one sub-dataset for each state) and analyze each state by itself. As each US state has its own regulations it may make sense to analyze each state by itself. We could aggregate our results across states later to get a national statistic.
   
   Or, for now, we could drop the State column and analyze the data across the nation later.

Drop these 5 columns: ID, EngineHP_bucket, Years_Experience_bucket, Miles_driven_annually_bucket, credit_history_bucket¶

The Miles_driven_annually feature has some null values. Let us explore which particular cells have NaN and ingest them with the median data.

It may make sense to ingest the median of Vehicle_Type=='Truck' as all the NaN values are for Truck only. Let us look at the median of Miles_driven_annually by each vehicle type.¶

Replace NaN values in Miles_driven_annually with the median value for Truck. There may be better ways to impute missing data. But we have just 8 NaN cells out of some 30,000+ rows which is less than 0.03%. So, imputing with median for all the 8 cells is not going to skew our results.¶

Check for null values again to make sure we did not miss any accidentally¶

Check the data types of all remaining features¶

Looking at the feature values above, the range of values of each vary a lot. For example 'Miles_driven_annually' is in the 10s of thousands, whereas 'credit_history' is in the 100s and 'annual-claims' is in single digit. Due to the varying magnitudes of the feature values we will scale the features with Z-scores using sklearn.preprocessing.scale.

Check if there are any NaN values one more time¶

Restore the column names from the original dataset¶

We now have the scaled feature set. Now we need to concatenate the categorical features back with our scaled¶

dataset before running OneHotEncoder or dummies.¶

Univariate analysis and dummy variable creation¶

Now we are going to extract all object variables (categorical variables) and check their values¶

Now we are going to check the frequency distribution of each categorical variables, and treat them according to their distributions¶

Observation: We can see that the variable state has too many values to create a dummy variable. Hence, we will drop it. For the rest of the variables, we will create a dummy variables.¶

Here, below, we separate our feature set from the label target and convert all the categorical variables to numeric. Then split the feature set into training and test data sets.

Let us convert some of the categorical features into numeric giving weightage to each variable.

1. Gender: 1 = Female and 2 = Male
   
2. Marital_Status: 1 = Single and 2 = Married
   
3. Vehicle_Type: Use LabelEncoder
   
4. Age_bucket: Use LabelEncoder
   
   We are not using dummies or OneHotEncoder because these create sparse matrices and increase dimensionality. By giving a 1 or a 2 for say Marital_Status we give higher weightage to Married by assigning a value of 2.

Segregating the dependent and independent variables as X and y¶

We found out much earlier that our target label is 70% success (good driver or target == 1) and 30% failure (bad driver or target == 0). Let us do class balancing using SMOTE and see the distribution.

Split the resulting balanced data set as train and test¶

Decision Tree Classifier¶

We could achieve a score of 66.6% using Decision Tree¶

Random Forest Classifier¶

We could achieve a score of 75.7% using Random Forest¶

Stochastic Gradient Descent Classifier¶

We could achieve a score of 49.9% using Stochastic Gradient Descent¶

Ridge Classifier¶

We could achieve a score of 50.3% using Ridge Classifier¶

Gradient Boosting Classifier¶

We could achieve a score of 69.8% using Gradient Boosting Classifier¶

Extreme Gradient Boosting Classifier¶

We could achieve a score of 75.8% using Extreme Gradient Boosting¶

Analytics Educator is the best institute for Data Science courses, based out of Kolkata. We specialize in providing training on data science even to students coming from non-technical background with zero programming or statistical knowledge. We help the associates to learn data science and get job in this field.¶

We provide a 100% money back guarantee on learning. It means that each and every student of analytics educator will be able to understand every line of codes and algorithm, else we will refund the money back.¶

The readers of this blog might mail their opinion how to further improve the model and you will get our contact details here.

If you want to read more such case studies then click on Whom should you ask for donations for a charity or Identify if a patient has cancer

Regression problems can be found at House Price Prediction and Insurance Premium Prediction