Exploratory methodology to untap value of AI/ML solutions

Some tools and ideas to guide exploration of advanced analytics initiatives

A lot of companies are starting to create customized AI solutions to enhance decision making in different areas of business.

Many of these AI solutions start by a simple idea, that with a little processing can be posed as a hypothesis. An AI/ML model can be built to materialize the idea using data and can be used experimentally to get to know if they work for the expected purpose and their value to the business. So, through quick model creation and experimentation it is possible explore new ideas and build AI and ML solutions that can escalate afterwards.

This try-and-see attitude before escalating something might sound very natural… we resort to this experimental approach many times to try new things in our daily lives. Even so, if there aren’t any rules or methodology being applied in the process of experimenting it is not that easy to trust in the results.

Experimenting while putting in practice these rules and methodologies in the day-to-day work can be very challenging. Even though exploring ideas should be quick and dirty building and testing there has to be a certain way of work and environment that tends to crash against bureaucracy, approval proceses, infrastructure, workflows, and so on.

Another thing that can decelerate or even kill experimentation in the business world is the mindset. In most of the companies people are set to comply with a stablished process, avoid mistakes and celebrate success stories, whereas experimenting is all about being open to failure (always limiting the risk), learning, constantly question the way we do things and knowing when and how to recognize new value streams.

This point is very important because the business is set to expect results, whereas the purpose of experimentation is to learn. This difference in expectations make a great impact on how we do things, because to aim results means to execute an action that we are almost sure will have a positive impact which contradicts with experimenting. And, what’s the result? Great ideas and models that never see the light, depriving the opportunity to change the status quo.

If experimentation is avoided then value can’t be identified. For this reason, I want to present an exploration framework that can help to methodically experiment new ideas and create useful AI an ML models to unleash impact in the real world.

First of all, I would like to clarify that exploration not only includes experimentation… there are 4 main steps that are essential for exploring AI/ML initiatives:

1. Discovery: The main objetive is to detect the opportunities that can make the most impact with AI. First business problems/opportunities are detected and then we see if there is enough data to work with AI, integrate the relevant data sources that could solve the business problem and calculate the business case of each initiative relying on data.
2. Modeling: The goal is to design and create the preliminary models that enables first experimentations, and to optimize and improve them over time.
3. Experimenting: The purpose of this stage is to use the models in the real world to test how they work and how the decision process in the business if affected by it. The positive impact and the risk of experimenting is limited by the scope of the experiments, but by extrapolating the results one can calculate empirical business cases that guide further iteration or implementation.
4. Prototyping: If the analytical solution has demonstrated value, we handover the model and all the knowledge gained on how to apply it in real use cases, and define the MVP that can be applied in the business to escalate the solution to the whole business.

These stages are not strictly sequential and tend to overlap with each other, being the most common thing to iterate between modeling and experimenting.

In order to move efficiently between these steps we can apply a certain framework (concepts, practices and criteria) that guide the way to get solid outputs from each phase and build good insights to asses how to move forward.

At Spike, we have implemented a framework that enables management of all initiatives in one exploration pad, so it can help to prioritize ideas between them to generate the maximum impact on business with the use of analytical solutions. It will also bring tools and guidance on how to communicate the value of exploration and relate with key stakeholders.

In the next figure you can take a look at the framework:

Let’s go through it!

1. Discovery

To kick off its good to do some pre-work on the domain of exploration, like a bibliographic review (read and summarize studies, papers, articles, etc) to understand the industry, different business models and the state-of-art in AI. We document this review in Notion!

At the same time it’s very helpful to document some peculiarities of the organization itself in which one is going to explore (stakeholders, processes, history) that will be necessary to design a customized experimentation methodology.

The first interactions with stakeholders aim to map all kind of ideas, problems and opportunities that exist in the domain. An ideation workshop is a good idea to carry this out, because it can help stakeholders make a pause on their daily work and think about this ideas. A good practice is to diverge as much as possible in the ideation workshop and not thinking in limitations on data, time, or feasibility, but just purely detecting problems and opportunities (not even solutions). Afterwards, to ground the ideas and detect where the value could be, it is good to work on defining all the problems with a certain structure and map them to a key stakeholder to associate the issue with a certain domain of action.

Then, we arrange what we call Pre-flight sessions with the selected key stakeholders, those who will be referents for a certain scope in the domain of exploration and that can share the most relevant insights of an area of work. In this session the main goal is to asses different angles of the issue that we are interested in tackling, and that’s why we find very useful to go through a checklist of questions that target different kinds of information, mainly about the business and data. To focus in just a set of problems, a good practice is to start the session with a summary of all the issues captured previously, in the workshop and the pre-work review, that have to do with the scope of the domain to focus just in them, and from there try to detect the most painful to go through the questionnaire.

The set of questions that have to do with:

1. Business: problem definition, context (wow, structure), objectives, target KPIs, business case value, risks.
2. Data: Target data sources, governance, quality of data, frequency.
3. Execution: High level timeline, Delivery mock-ups, high level into-production workflow.

For example, if we were to explore in a Supply Chain area there might be different managers with different scopes, objetives and problems. So each of them can be the key stakeholder for its scope, and the correct person to have the pre-flight with.

At this point, it all might make sense. Truth is that when having the pre-flights we have to be very careful not to be induced very quickly by a HIPPO and lead to biases on prioritizing points of exploration. In order to avoid this, always try to verify the business case with data, and comparte with other problematics that are known for the domain to keep other options available. A good way to avoid this bias is to go back and check with the pre-kickoff investigation.

On the other hand, a cool thing about working top down is that it brings clarity on how the team works, the data flows, and the decision processes are managed. The main stakeholder will bring great vision on the main sources of opportunity, readiness, and team capabilities.

The output of the Pre-flight session is a summary of each initiative, in which we should be able to define each blank space of this Canvas:

This is really useful to make clear what the understanding of the problem is, the ultimate goal and the main needs to tackle it. It can also be a powerful tool for alignment and communication.

It should be clear what sources of data are needed to build the AI/ML solution and what KPIs are key for impact measurement. Therefore during the discovery it’s important to find and integrate the necessary databases to give the first sense of feasibility and also to evaluate the Impact of business case with data.

With all the Canvas in hand, we can use the Impact estimation together with a scalability and effort measure to prioritize the AA initiatives.

As a result we can define a roadmap of exploration with all the different initiatives. The roadmap is a great way to organize the exploration pad, see all the portfolio of initiatives, and give a sense on what should be happening next. But, watch out! The exploration itself can’t be very structured and have pre-defined next steps. The whole purpose of exploring is to find new insights and information on the way and to have flexibility enough to take them into account and follow new paths and hypothesis if that’s reasonable. So the roadmap isn’t about the timeline itself, but about the knowledge gained in the past and what makes sense to do in the following weeks, not much further in time.

Once the roadmap is clear and data is available, it’s OK to start modeling.

2. Modeling

Take into account that the modeling phase not only means programming an algorithm, but it also involves exploring different the models applicable to the case, deciding which fits better, building the model and validating it.

It is important to be as transparent as possible with stakeholders on how the model is working. Sometimes explainability might be possible, but sometimes it isn’t (like in deep learning models). But even when explainability can be built, it takes lots of work to do it. So when exploring something new it doesn’t always makes sense to start building full explainability. One way to overcome this point, is to conceptualize how the model works, provide some examples of it and show some classic KPIs such as: accuracy, precision and recall.

Once the model is functional, we jump into experimentation, and through experimentation is that we gain knowledge to incrementally improve the model and test.

If the model is proven to deliver impact and is handed over for exploitation, it will be necessary for the technical counterpart to have all the technical documents and repositories available for handling. So during the exploration it is a good practice to keep the model, data ingestion and predictions as documented as possible.

Another important thing, is the level of automation in the process of using the model. When experimenting it’s okey not to have everything automated, as this might take time to do so and it’s not always going to payoff. So keep in mind that when starting to experiment with the model it makes sense to work manually with some parts of the model, but if the experimentation demonstrates value and starts broadening the scope it will be inevitable to work on automation.

So, it doesn’t matter if we are at the first or final version of the model, it is always necessary to pass it through experimentation to capitalize it’s value and make an impact on business and learn.

3. Experimentation

Such as building models in modeling, experimentation is not only executing the experiment.

To start it si very important to design the correct experiment and assure applicability to the business. For example, it is not always possible to build a control group to perform A/B tests and we have to resort to more complex experimental designs. This kind of decisions are made at the starting point of experimentation. The most important definitions to take on are about: variables, controls and KPIs.

All in all, experimentation consists in designing the experiments, implementing, measuring results, and defining next steps. Next steps might be to modify the hypothesis, iterate, or exploit the model.

To assist next step definition we can calculate the captured value on the experiment and extrapolate this to have en empirical business case which we can comparte with the theoretical business case and drive conclusions.

Moreover, always keep in mind that the most important thing about experimenting is the knowledge gained from intervening the reality, so its a good practice to keep an insight backlog to then visualice all the learning path and keep it documented. This can be very useful to visualize the learning outputs, which are the most important assets in exploration, and also as inputs for the prototyping phase if value is demonstrated.

4. Prototype

Once an initiative has shown empirical evidence of impact through iteration of experimentation we drive the analytical solution to the exploitation phase which consists in going into production mode and full escalate to impact the whole reality of the business. For this to happen and to help the business capture all the value estimated in experimentation phase there are two important aspects of the handover: The model itself (how to train, predict and maintain it) and the operation of the analytical results in the business use cases (actions, workflows and impact generation, human-solution interaction).

At this point, it is expected to have a solid versión of what the business case is if the AI/ML solution is implemented and a clear vision of what the MVP must incorporate.

For this purpose, defining what the MVP should look like incorporating all the knowledge of iterating with the prototype is a good way to converge into a tangible and functional solution with the exploitation teams. The MVP must consider all the insights generated throughout the experiments, and incorporate the analytical solution. The MVP must be built by a development team to assure we are incorporating the MLOps and product development best practices (QA, CI/CD, Issue tracking).

Handover meetings with end users and exploitation teams are essential to being able to transfer the knowledge gained through experimenting, and also handover the technical parts: pipelines, APIs, visualizations, model life cicle, etc to the corresponding team.

All the the above must be well documented and accesible for the handover teams.

And that’s it!

I hope this summarized explanation on what the framework is about and how to use tools like the canvas can make an impact now to your business and enhance exploration!

Many thanks to Mane, Angie and JP.