Model Evaluation(Part 5)

In the previous lesson, we presented the training and testing of our model. This was on the basis of creating our ML pipeline. The following figure illustrates the workflow.

In this final lesson, we set out to evaluate our model. It can be performed using:

val evaluator = new MulticlassClassificationEvaluator()
   .setLabelCol("CategoryNameIndex")
   .setPredictionCol("prediction")
   .setMetricName("accuracy")

Next, is to determine the accuracy of the model as follows:

val accuracy = evaluator.evaluate(predictions)

We print the accuracy to the console and neatly end the spark job with spark.stop().

println("Accuracy= " + accuracy)
  
   spark.stop()

And here is the output:

Accuracy = 0.935483870967742

About the Author

Taiwo O. Adetiloye has a PhD in Information and Systems Engineering. He is very interested in large-scale data processing and analytics using AI and ML frameworks like Spark, Keras, Tensorflow, Neo4j Graph and MxNet. Favourite programming languages are Scala, Python, and Go-lang. He is constantly developing himself as an information technologist working and acquiring new skill sets to help him in his career growth. He is a highly motivated individual, good communicator and great team player with a passion for creativity and a drive for excellence. He has got over 9-years experience in the Data Analytics Industry. Presently, he works as a Data Analytics Engineer in the Healthcare Sector.

You can connect with me on LinkedIn.