Evolving From Descriptive to Prescriptive Analytics: Part 5, Selecting the First Project

By Chad Marston and Shaikh Quader

In our four previous posts, we talk about building a team with the proper skills and tools to succeed at data science. Now, it’s time to take on our first project. The goal for this project is to obtain real-world experience and to deliver tangible value to the business. Our strategy is to get started with a project even as we’re building skills and tool infrastructure, so we’ll be more prepared to take on a highly visible project.

We started with three project selection criteria:

1. We need to have subject expertise and/or established networks to draw on

2. We need to already have the core data — or know we could get quick access

3. We need to be able to provide tangible benefits to our immediate organization

We know that our Customer Support organization regularly resolves client issues with speed and quality, which typically results in a positive experience for our clients. But if clients don’t feel they’re getting resolution at the pace they need, they’ll sometimes escalate their issues to Support Management, Sales Reps, Executive Sponsors — basically anyone they know in IBM who they hope can get them what they need. Over time, Customer Support has made a vigorous effort to capture information that would help them understand these client escalations. They’re eager for clues that would signal client frustration and want to prioritize issues in order to avoid it.

We determine that a machine learning project to avoid client escalations fits all three of our criteria:

1. Subject Matter Expertise: In a previous role, I led global escalation management for over a decade. I know this space and have relationships with many others that do as well.

2. Data: We have the data since we’ve been providing it to the business through traditional reporting and descriptive analytics.

3. Tangible Benefits: Reducing escalations would benefit both our clients and our business. We can also measure the impact of reduced escalations.

During this project, we start to learn invaluable lessons about the technical challenges of machine learning as well as our project selection criteria. Regarding the latter, we confirmed that these are critical criteria to the success of a machine learning project, but we didn’t fully grasp what it meant for a project to meet these criteria. Some detail…

1. Subject Matter Expertise: This isn’t just about having the domain expertise, which we had — it’s also about engaging the stakeholder community early and often in the project. That engagement provides value beyond building the solution as it helps us identify the right data sources, understand and validate the data, test the model, and deploy the model in production.

The major value we didn’t appreciate in this first project is the benefit of having business stakeholders formally on the project team when you engage the community of other business stakeholders. A message delivered from a peer is far more powerful than one coming solely from the IT team. Having a stakeholder already in your project changes the perspective from IT delivering a solution to a joint solution. That difference instantly earns trust.

2. Data: Our goal in this project was to accurately predict whether open support cases would be escalated through our formal complaint process. The model was able to do this 86% of the time. Going into the project we knew we had access to the formal, fully documented complaint process. We later came to realize and appreciate the many informal ways clients escalate which was not captured in this data set.

Because not all of these informal shadow procedures were documented, our training and testing data was an incomplete view of the business. We realized that engaging subject matter experts to obtain a business-centric view of the data that’s required and available is just as important as a data-centric view of that data.

3. Tangible Benefits: “If you build it, he will come” from the movie Field of Dreams has become part of the American lexicon, but it’s not entirely true. We were quite proud of the model we built with 86% accuracy but Customer Support didn’t use the results at the scale we expected.

We learned that accurate results are only a piece of the equation as actual adoption is the only real path to value. Key to adoption is working with the stakeholder community to understand the optimal way for them to turn model output into decision making and action. A successful machine learning project has to design and build the experience of consuming the models.

We now have a very important addition to our project selection criteria, a new number 1:

Stakeholder involvement and commitment: The users that we expect to benefit from the model need to be part of the project team from the start. In addition to the roles mentioned previously, they also set the success benchmark for the machine learning models and commit to use the models if we meet their benchmark.

Our team’s goal with this first project was to learn by tackling a real business problem that would also provide benefit to the business. We certainly learned a lot, in fact far more than we expected. However we didn’t maximize the business value. We now ensure the involvement of model consumers early and often to keep focus on the ultimate goal of business value.

We hope you’ll check out the previous posts in this series:

• Part 1: Leadership Support
• Part 2: Acquiring the Right Skills, Faster
• Part 3: Fast Start Data Management
• Part 4: Eating the Ugly Frogs