Happily Ever After?
What’s Next After Model Deployment? Monitor and Visualize Your Project Results with Tableau.
Once the team of developers and business side handshake on the model to be used in the data science project, now it’s time to deploy it. As is known, the data science project lifecycle consists of different phases, this lifecycle never ends, and everyone involved in the project puts some feedback to evolve a better version.
Let’s see where this story parks in the data science project journey:
We are already finished with the modeling phase in the previous story Implementation of XGBoost on Income Data and checked relevant evaluation metrics in Interpreting Performance Evaluation Metrics for Classification Problems published in Hypatai. Assume that after all those comes and goes with the business, you already put a great effort into the deployment phase and your model is in pilot since last week. Now it’s time to demonstrate what value you have added to the existing system, be proud of yourself, and receive some feedback if any modification needed.
I use the dashboarding, storytelling, and visualization tool called QlikSense, frequently in my job for presenting the project deliverables. In real-life, people work with big data, which is why I mostly use the connection between QlikSense and Apache Hive to access the relevant data and create qvds to access them easily later. QlikSense also provides a scheduling mechanism not to worry about recreating up-to-date results manually. A similar server connection can also be established through Tableau, however, since we will be working with the data we worked on in the previous stories, a simple connection to a CSV file stored on the local computer will be enough in our case. The reason why I used Tableau this time was to get familiar with the tool which I do not use in my daily routine and to assess the pros and cons compared to Qlik.
What I caught during my small example Tableau viz creation phase were :
- Tableau is more of a drag-and-drop style tool, and QlikSense is more based on scripting.
- When it comes to Tableau, it has fancier, better-looking graphs with stylish colors, appealing to the eyes of the audience.
- Personally, I do not prefer to pick any of them for EDA purposes. Phases before modeling need a quick trial-and-error approach if we want to be agile and to produce an mvp as soon as possible. Libraries like Pandas, Matplotlib, and Seaborn in Python already provides us this flexibility as developers.
Now we are ready to start!
The data used in Tableau is the test set of our income dataset we used in the previous stories. The test set was not used in the model-building phase at all, therefore it can mimic the new coming data in the pilot phase after deployment. I also needed some extra columns like predicted income probability and predicted income level coming from the XGBoost model we established in the story Implementation of XGBoost on Income Data. So I added those columns, as well as a new fake date column for trend plotting purposes, and to demonstrate date picking in Tableau. Prepared visualization data and the associated notebook could be found on my GitHub page here.
To get a more interactive and full experience of what we will be talking about, you may reach the viz I created in my Tableau Public profile from here. The story I created is composed of the three-page dashboards prepared in one day. So there is room for improvement for visual fulfillment, but still, it is a good start. 😇
Here are the pages showing the general and detailed views of our pilot :
The Overall View shows the general picture and a summary of our main inputs and outputs. Remember that we labeled income as >50K when the predicted probability is greater than the optimal threshold (0.21) we found in the previous story Interpreting Performance Evaluation Metrics for Classification Problems. We see that the average predicted probability (0.236) is close to the optimal one.
Just by looking at the average, a quick insight would lead us into seeing an almost equal share of both labels (>50K and ≤50K) when we see the proximity of these two probabilities. However, as shown in Detailed View, the probability distribution is not as uniform as we expected. The real distribution (which is shown on the left) is more like an exponential distribution (I did not make an approximation to the distribution, so real distribution might be different). Distributions for other variables (stacked by predicted income level) are shown for marital-status, capital-gain, and age variables which were not selected by chance. Remember that those variables were the top three in our feature importance list (See Implementation of XGBoost on Income DataImplementation of XGBoost on Income Data).
It is important to monitor and detect any sudden change in the distribution of important variables, which may lead us to revisit our data collection phase, our preprocessing method, or even the model itself. Looks like our Tableau story does a good job of providing important insights both for the business side and the developers, and serves as a platform for the ‘Feedback’ step in our data science project lifecycle.
Trendlines shows the trend on the average value of each important variable, again with breakdowns into the two-income labels. Our alert mechanism works with a deeper level here because we can actually catch the milestone of the change by checking the time frame.
The plot showing the distribution of predicted income probability shown in Detailed View is right-skewed and most values lie between 0 and 0.4. Therefore the user may want to have a closer look. Thanks to the zoom in/out slider, we may see the magnified results.
I also created a date picker to select the desired date range. Both the zoom in/out slider and the date picker filter out relevant records in all pages and show the results accordingly.
The story I created in Tableau Public works well with the data we have at hand. However in real life,
- It would be nice to add other visualizations like plots or charts showing the status of other models or processes using the predicted income we generated as input.
- In addition, other KPIs showing the validity of taken actions based on the income we supplied would be vital.
- Maybe there was an existing system assigning income levels in a traditional way and people from business preferred to go in pilot not in a full fashion but rather with an A/B test approach. In that case, some visuals showing the comparison and performance of obtained results would be relevant.
Hope this was an insightful story for you. 🎯
Stay tuned!
