Last week we talked about the imbalance problem on our dataset. After the analyzing the post re-sampling method results, we decided to apply these re-sampling methods to all of our algorithms. So, all results that we will talk about in this post are obtained with re-sampled dataset.
Before moving on, we wanted to see the results of a Neural Network on our dataset. During this process we have tried different techniques on our network. Such as :
Data normalization
Batch normalization
Weight initialize
Learning rate decay
Dropout
Weight regularization
Early stopping
We have tried these parameters for our network :
And we obtained :
As we talked about in previous posts, we got our best results with Decision Tree and Random Forest. Therefore, we decided to apply some of the other tree based algorithms.
We have tried LightGBM, AdaBoost and XGBoost. As we guessed, we got our best results with XGBoost. It is not surprising as XGBoost is the winner of the most of the machine learning competitions. We obtained ~%60 micro F1 Score with XGB.
After finishing with algorithms we sat back and thought what can be done in order to increase the success of the model. %60 F1 Score isn’t that bad with a very noisy data like this but couldn’t be satisfied with this. So, we remembered what we had thought at the very beginning of the project : we have an ordinal regression (or classification) problem instead of a regular classification problem. This means, order of our class labels are important. We can’t treat every incorrect prediction as same. Let’s explain this with an example:
Let’s assume we have a building which has a damage grade of 4 and our model couldn’t predict it correctly. In this case, our model can predict 5 or can predict 1, too. If it predicts 5 instead of 4, we can say that this is not a very very bad prediction after all. However, if it predicts 1 instead of 4, we have a serious problem. So, we need to come up with a solution to solve this problem. Actually this is not an easy-peasy problem to solve and there are many studies related to this topic, like this one : Herbrich, Graepel, Obermayer. Hence, we will try to solve this with a naive approach.
If close damage grades are similar to each other, why not group them together? So we grouped our damage grades into 3 groups:
- Damage grades 1–2 -> Low
- Damage grade 3 -> Medium
- Damage grades 4–5 -> High
As mentioned earlier, this is definitely a naive approach and a better job can be done for sure. Lets see the XGBoost results again with 3 types of labels :
With different re-sampling methods we were able to get ~%70–75 score which is quite good.
This is all from us for this project everybody. With many different algorithms and techniques we obtained different results and a similar kind of a model definitely can be useful for future earthquakes. All the code work can be found here : https://github.com/OnerInce/ml-earthquake
We also created a short video for our project, too. You can watch it from here :
See you in future works !
✌️
