Solving A Real World Data Science Tasks With Python : Predictive Maintenance Classification — Episode-II- “Building Your Own Metrics”
For Previous Article : Here
Hello everyone, If you remember, in our previous article we used machine learning to minimize production errors in a factory. At the end of the study, we found the accuracy scores of 7 different models, whether there will be an error or not, and we shared them with you. However, in real-life problems, we may need more detailed findings. In such a case, accuracy scores are not enough for us. In this article, we have descended into a more detailed layer.
In this article, we generated our error metric ourselves. We think that a scenario in which the error metric is adapted to the real problem is also important in terms of showing how machine learning algorithms contribute to a business in practice. Standard metrics have been used in machine learning problems so far. For example, accuracy, precision, recall, f1-score. You can reach our article on metrics at this link. But what if we need to use a metric that we created outside of these metrics in our business? Perhaps standard metrics are not always the best choice for real-life problems. If you’re ready, let’s start.
First of all, we changed the problem from a binary class classification problem to a multi-class. Because in reality, the types of machine errors and the costs that these errors reflect on the business can be very different.
Secondly, we made some cost assignments for each mistake, taking into account our imaginary business.
Third, we ranked the success status of the models according to the new metric we created.
The purpose of this problem is, in the past, when there was an error in the production line and after the error occurred, we stopped the line. Now we have renewed the system in the factory. We placed sensors on the production line that see real-time data. We connected these sensors to a computer and database and loaded a machine learning algorithm on the computer. Now, if the computer deems it necessary before the error occurs, it will stop the line on its own initiative and will tell the type of error that will occur before the error occurs.
Of course, not every estimate may be correct, but the goal is to minimize the cost. We will try to see with which model the line under artificial intelligence management gives the best results.
Before we move on to the codes, we need to give one last piece of information. We should know the error types and the costs of these types before we start coding.
Error Types
For all error types, the cost of predicting and taking action is fixed and is 5 units. Except this;
Power Failure = 45 units
This error is caused by a problem in the electrical component. Requires machine refurbishment if not resolved beforehand
Tool Wear Failure = 30 units
This error is caused by the incorrect insertion of the machine tip. As a result of tip breakage, it causes us to order a new tip again.
Overstrain Failure = 20 units
It results in the breaking or cracking of a part of the machine. In this case, it is necessary to mount the relevant module of the machine on the old machine.
Random Failures = 15 units
It refers to the expression of 4–5 different types of faults that occur because the machine works too hard in a single class.
Heat Dissipation Failure = 5 units
It causes faulty production as a result of overheating of the machine.
First, we imported the necessary libraries.
Then we loaded our data file and looked at the first five lines with the head() command.
We dropped the UDI and Product ID columns in the df data file and created a new column named nf by feature engineering.
Then we created a new variable called df_target. Because we want our changes not to affect the df file. We have assigned the column Failure Type, which is the column we will guess, to a different variable.
Then some columns had string values. We encoded them.
After cleaning the necessary data, we divided the data into two parts train and validation data.
In this section, we created our error metric that we mentioned above. We defined this metric in the calculate_cost function. Assuming that there will be an error in the first part, we defined the cost of taking action. We have defined the unit costs of not being able to predict the errors that will occur. We calculated the number of times I will take action regarding the error types in the second paragraph, using the confusion matrix structure. Thus, we found the total cost of taking action for each error. In the third paragraph, we calculated the number of failures to predict errors and their costs. Finally, we found a total cost value by collecting all these costs and had it returned.
In this section, we imported the library of each machine learning algorithm we will try and called our calculate_cost() function, which we defined above, based on the predictions made by that model.
We turned the “classifiers” variable that we defined above into a data frame and sorted it from least to most.
According to the new metric we produced, the most successful model is seen as the Begging Classifier.
I would like to express my gratitude to my esteemed advisor, Alparslan Mesri, with whom we worked together in the writing of this article. See you in our next article :)
