DATA STORIES | GENAI | KNIME ANALYTICS PLATFORM

AI or not AI?

The Occam’s razor principle from the GenAI perspective

We are in an AI frenzy. GenAI is invoked as the panacea to solve all possible data problems. While GenAI is surely powerful and it can often provide a solution, sometimes I think that we are complicating our own professional life much more than needed by forcing GenAI into all our projects.

Recently, I was supposed to adapt an old solution for fraud detection to the new version of KNIME Analytics Platform 5.2. Since I was at changing, I thought of also exploring the possible benefits of GenAI for parts of the workflow.

What I have learned on the way, is that there are tasks where applying AI is a sure win and there are tasks when the time invested cannot compare with an easy, ready, out-of-the-box technique.

So, what is it? AI or not AI? And if AI, when?

Fraud Detection in the Old Days

The original fraud detection task aimed at isolating possible fraudulent documents from a dataset of 100 PDF files of investment contracts. Each contract followed a legal template, where customer data, money amount, date, and ID code were customized for each instance. Customer data was also extracted from the CRM database, including the customer’s preferred language of communication.

The process for the old solution included:

1. Import and parse each PDF file

2. Extract the contract information (date, ID code, customer email, money amount)

3. Find outlier contracts by applying one of the many available fraud detection techniques, based on clustering, statistics, Inter-Quartile Range, or Isolation Forest

4. If an outlier is found, send an alert to the customer in their preferred language, explaining that some checking might be necessary to verify the legitimacy of that contract.

Find all previous workflows in folder KNIME for Finance/Fraud Detection on KNIME Community Hub.

The most complex part of the solution was the email writing part. Indeed, the little clause “in their preferred language” requires the creation and maintenance of circa 50 email templates for 50 different languages. For that, a set of translators was required to be on standby at each update cycle, especially if the email texts had to comply with original company’s statutory documents. This approach generates a complex process for the updating and maintenance of the email texts.

GenAI for Alert Email Text Generation

One area where GenAI excels is surely text generation in many languages. Using GenAI to generate the email text in the customer’s selected language seemed like the easiest improvement to apply in the original workflow. We just needed to create a simple prompt, asking to generate:

• an alert email for suspicious fraudulent activity
• in <preferred language>
• around contract number <ID code>
• for <amount> dollars
• signed by <customer name>
• signed on <date>.

After authenticating with the AI provider of choice and selecting the appropriate LLM, we just fed the prompt into the LLM Prompter node, and the email text was generated in the selected language. No more need for storage of templates in whatever languages, no more need for maintenance of such templates. Just a prompt and the email is out!

Figure 1. On the left: the Authenticate — Connect — Prompt sequence to include GenAI within a KNIME workflow. Here it is used to generate an alert email for a possible fraud. On the right: the resulting email.

Now, you could object that the email is a bit too general and misses some context. The amount is surely a reason for suspicion, but maybe the email could refer to a more formal document identifying the top reasons for suspected frauds or explaining the different types of investment contracts. Thus, we adopted a Retrieval Augmented Generation (RAG) approach to tune the selected LLM on the knowledge base from a formal document.

Find the new workflow “Fraud detection with dataviz, IQR and GenAI for alerts”, including GenAI generated email texts with context, in folder KNIME for Finance/Fraud Detection on KNIME Community Hub.

Figure 2. On the left: the workflow segment implementing RAG, to generate an alert email with context. On the right: the resulting email. In yellow the part that is different from the previous email text.

GenAI for Fraud Detection

What about using GenAI also to find the outliers, i.e. the potential fraudulent contracts? In the old application, in order to discover the outliers in the dataset, we have tried:

• Z-score. This technique transforms the data as to have a Gaussian distribution. Then it applies a threshold to both queues of the Gaussian to find the outliers.
• Inter-Quartile Range (IQR). This technique assumes that the outliers are located beyond k*IQR from the 1st and the 3rd quartile, where IQR is the Inter Quartile Range measure.
• Isolation Forest. This technique measures the number of cuts it takes to isolate each point in a dataset. The hypothesis is that outliers can be separated from the rest of the data with less cuts than non-outliers. Thus, a threshold on the number of cuts labels the outliers.
• DBSCAN, a clustering-based techniques. The DBSCAN algorithm has this great property of isolating distant unreachable points and of classifying them as “noise”. This algorithm also requires a radius value for the inclusion / exclusion of data points into clusters.

Figure 3. Six popular techniques for outlier detection.

Could GenAI be of help for this task as well? We have tried it. The implementation is quite easy, with the usual sequence of three nodes, i.e. Authenticate — Connect — Prompt, and, as expected, the LLM produces an answer.

On one side, this approach is easier than others. It requires no knowledge of the algorithm, no structure of the query (prompt is just text), and it can easily work on all sorts of data. However, we found also a few disadvantages.

• Small prompt variations lead to completely different results. Sometimes you get one outlier, sometimes four, sometimes they overlap, sometimes they do not. All of that, just following small changes in the prompt text.
• The LLM works as a black box. We know neither the logic nor how the request is parsed. There is zero-transparency on how the results are obtained. The algorithm is not just complicated, as it was for neural networks, it is not available.
• There is little control. LLMs can be tuned, but only a few tuning options — mostly temperature & prompt engineering — are available. For example, there is no way to apply a threshold to make the algorithm more (or less) conservative, like we did for the more traditional techniques. Proper LLM fine-tuning, although possible, would require large datasets, time and money.
• Result accuracy and ability to detect outliers depends on context window. An outlier is such when compared with other datapoints. For small datasets, the context window used by LLMs should be sufficient. However, for very large datasets, the context window might not be large enough and this can affect the accuracy of the results.
• Pre- and post-processing are required. In this case, we do not just pass a text prompt to the LLM, but also a dataset. The dataset must be forced into the prompt and therefore some pre-processing is required. The same is true for the LLM outputs, in text format, that requires some post processing to extract the relevant output. In addition, depending on the data and the prompt, such pre-and post-processing can become quite complex.

Four Factors for the “AI or not AI” Decision

So, let’s go back to our original question: AI or not AI? Will AI provide a solution to all my data science problems? If not, what are the factors to keep in mind when choosing whether a task is suitable to an AI solution or not? From this experience, I would carve four main factors to make our decision between GenAI and classic techniques.

1. Correctness. Most important of all, results must be correct.

2. Transparency. Do I need to understand what the model does?

3. Control. Can I tune the model as to skew it towards specific results? Can I do that with just prompt engineering and temperature?

4. Cost. Most LLMs cost, in money and human labour. If I need days to tune the prompt, prepare the data, and postprocess the results, then an out-of-the-box technique might be preferrable.

In this project, for example, we had two chances of using GenAI: one to automate the email creation and one to extract the outliers. In the table below you can see the two tasks and their decision factors.

Table 1. Decision factors pro or con using AI for a project task.

Correctness. While the email texts are on average good enough; the results obtained for the outlier detection are very much dependent on the prompt we gave.

Transparency. For the email creation, I do not need to understand what the algorithm is doing. After all, I can read, or I can find somebody to quickly read the text created in the selected language. For the outlier detection, it would be important to understand why I end up with 3 or 4 or 2 or 10 outliers and where the truth lies.

Control. For the email creation I can use temperature, RAG, and prompt engineering to get a text more authoritative or more based on company’s documents. For the outlier detection, I cannot set a threshold and decide whether to allow more or less outliers through the grid.

Cost. We could have also implemented an automated email text generation but it would have been a lot of work (We need to create and train a neural network, assemble a sufficiently large text corpus, have sufficiently powerful machines for the training, etc…). For the outlier detection, we could have just drag-and-dropped the Numeric Outliers node to get the outliers with the IQR technique.

Based on all these factors, adopting AI to create email seems totally worth it; however, adopting AI to find outliers looks like a classic overkilling example.

AI or not AI?

The Occam’s razor principle states that “other things being equal, explanations that posit fewer entities, or fewer kinds of entities, are to be preferred to explanations that posit more.” (see Britannica Encyclopedia for this quote). We have heard this principle over and over, together with the cannon used to kill a fly and everything looking like a nail to hammer. The meaning being that we should choose the more appropriate (simpler) solution for each problem.

All considered, in many cases GenAI represents a valid alternative; but in many other cases, classic data analysis techniques provide a simpler, more transparent, less costly solution than GenAI.

In conclusion, while the idea of using GenAI for all our tasks seems quite tempting, we need to be careful with each situation. If a simpler, cheaper, more transparent, more controllable technique is available, then we should give our preference to this technique rather than making our solution excessively and uselessly complicated.