Non-supervised AI for SMEs: Making success tangible with benchmarks.
This article is part 5 of our series on non-supervised AI for SMEs.
Authors: Johannes Otterbach, Clara Swaboda, Design: Clara Swaboda
“You cannot improve what you cannot measure.” This nugget of common sense knowledge is key to fueling digital transformation through AI, but it is often overlooked. Building ever better models is the motivation that drives both AI research and business applications. But how do we know how good an AI is? The answer is: through benchmarks. Benchmarking is the practice of comparing tools to find the best-performing technology. In AI research, the current model is usually compared to the state-of-the-art model in the field.
“As we all know, no two problems are the same in the real world. For AI projects, this means that the performance of two models cannot be readily compared because the problem and data are often unique.”
However, in business applications, benchmarking is much more difficult. As we all know, no two problems are the same in the real world. For AI projects, this means that the performance of two models cannot be readily compared because the problem and data are often unique. Moreover, applying the metrics used in AI research to business problems might not yield valuable results. Developing successful AI solutions requires a deep understanding of the relevant business KPIs and a way to easily measure and express them in a single, or a few, numbers.
Take this example: A bank has data on whether their clients repay or don’t repay their loans. This data can be used for setting up an AI project. What the company is really interested in, however, is not whether individual clients pay their loans back or not, but how much money the bank loses overall. This business need can be translated into a suitable metric: the money loss rate. Thinking about how to express a KPI in an AI metric is crucial to transforming an AI project into a solution to a business problem.
“Thinking about how to express a KPI in an AI metric is crucial to transforming an AI project into a solution to a business problem.”
After having formulated a business problem as an AI problem and translated a business benchmark into an AI benchmark, model development can begin. In most AI projects, all data points are labeled. This makes it easy to see whether the classification a model makes is correct or not. Based on this information, the performance of the AI model can be computed. Non-supervised AI methods however can work entirely without labels. So how can we measure how well non-supervised AI is doing? Our vision is to train a large base model on the raw unlabeled data to roughly discover patterns. Then, this model is fine-tuned on a small dataset of labeled data points for a specific task (read more in “How non-supervised AI will enable B2B adoption”).
Here is where the benchmarks are applied. We would like to know how well the model performs on the task it is supposed to solve. The general rule is: The closer the benchmark reflects the task, the better one can assess the model’s impact when deployed. This piece of wisdom is true for every AI project but even more so for non-supervised AI where we have fewer data points. Those data points should be of high quality and the benchmarks should be designed in such a way that they reliably assess their quality.
“The general rule is: The closer the benchmark reflects the task, the better one can assess the model’s impact when deployed.”
If the benchmarks indicate that the performance of the model is insufficient, it is helpful in most cases to take a critical look at the data. Providing more unlabeled or labeled data points to the model can improve performance. At times, more labeled data points are needed to make the model perform better on the given task. But benchmarks can also help in identifying which kind of data points are missing in the current unlabeled data set. This additional unlabeled data is then used to train the next generation of the base model.
In a nutshell: Benchmarks are the barometer that gives us an answer to the question of whether AI is actually solving our business problem. Even the best data, the best infrastructure, and the best talent are not enough if the AI does not solve the problem it is designed for or if we cannot reliably measure what the AI is doing. The take-away message from our blog series on non-supervised AI for SMEs is: technical and business knowledge have to be intertwined to generate sustainable business value with AI and drive the digital transformation of SMEs.
