DATA STORIES | AI & LOCAL LLMS | KNIME ANALYTICS PLATFORM
Chat with local Llama 3 Model via Ollama in KNIME Analytics Platform — Also extract Logs into structured JSON Files
Integrate and interact with the latest local LLMs with a low-code tool
Large Language Models are still going strong and the development is fast. There has been a lot of interest in my recent articles about KNIME and local LLMs using GPT4All. This time I would like to explore the use of the much hyped Llama 3 by Meta via the Ollama wrapper and query the model with the help of the low-code analytics platform KNIME.
Just load the KNIME Workflow and give it a try (you will have to install and run Ollama) — https://hub.knime.com/-/spaces/-/~_gPUhkuAieieXCsi/current-state/
Maybe just download the whole LLM workflow group …
I will use mostly generic nodes and the KNIME Version 4 — so to have a maximum of compatibility and flexibility — since it happened twice in the last few month that, for example, GPT4All changed their file formats and KNIME nodes did not catch up immediately.
In order to get this thing started you will have to install Ollama (https://ollama.com/) and initially start a chat with a specific model. When you first call this function the model will be downloaded to your drive (in this case Llama3/instruct):
ollama run llama3:instructThe model will be used on your local computer with your local resources. So data might not leak into the internet.
As D. Paurat keeps reminding me: instruct models will more closely follow -well- instructions directly while chat variants might be more suited for … chat. So you might want to experiment with that — and your own data.
Ask Llama 3 via simple KNIME POST Request
In KNIME the simplest use case is to create a small JSON prompt in a Table Creator node and then send it to Ollama via a POST Request. The results will be brought back via a JSON structure that you can then use and inspect:
You will have to make sure Ollama is running in the background and you will be able to contact the localhost with this URL:
http://localhost:11434/api/generateYou should check a few settings in the node settings. Namely the timeout. Since the model will need some time to process you will have to adapt the timeout. Also in Error handling you could choose settings for retries.
The Request Body will be in a “Value” Column containing the name of the model to use and the prompt:
{ "model": "llama3:instruct",
"prompt": "How many colors are there in a rainbow?", "stream": false } 
The response will be captured in a JSON file (Content-Type / application/json) whose meta info will be shown under the “Request Headers” tab.
The results are then stored in a response column you extracted with the JSON Table node, and that you can further use and display:
Llama 3 does not have a specific setting for a role or separate instructions. But you could easily provide one on your own:
{ "model": "llama3:instruct",
"prompt": "You are a helpful assistant knowledgeable in science,
particularly physics and geography. Provide polite and thorough answers
in simple language suitable for beginners. Always strive to be clear
and concise in your explanations.
The question: How many colors are there in a rainbow?", "stream": false }Loop through several questions
You can now also use this setup to loop through several questions at once and then collect the results. This will be especially useful if you want to process larger amounts of data or automate extractions (we come to that in a moment).
As you might have noticed. I have introduced another node that would pre-process the prompt in order to escape special characters so the data can be handled in KNIME and can reach the Llama 3 model even if you use quotes or line breaks:
Your local LLM Chatbot with KNIME Data Apps and Llama 3
You can also build out your local chatbot with the help of KNIME Data Apps. It will use the same REST API under the hood but present the results as a chat.
Inside the component the prompt and instructions will be put together, special characters will be escaped and the answers will be stored in a KNIME .table for further reference.
Note. The chat will not have any previous knowledge of the questions but each question will be handled individually.
Use Llama3 to extract data from Logs into structured JSON Files
One common idea is to use LLMs to extract data and maybe insights from un- or semi-structured data into machine-readable tables (that is JSON files). In this example I have some Log files with several information I want to extract. I instruct the model to respond with not just a chat or text but with the information in a JSON file that always should have the same columns.
We then use KNIME to untangle the results and make them readable in a classic table — yes you might have to adapt that to your needs and think about how to process them further.
Also in this case the power of your local machine will be used so you will not have to pay for tokens at a hyperscaler. Although you might want to check out the licenses and terms and conditions of your specific LLM model.
You instruct the model in a prompt to bring back a result that you can then turn into a table. Depending on the LLM you are using you might have to experiment with different prompts.
Please convert the following log file data into JSON format.
Ensure the JSON output has a consistent structure with
"date", "time", "severity", "component", and "message" fields.
Here's the log data:The response initially looks like this. The result is enclosed in backticks and has a JSON format. The loop will iterate it over your collection of Logs. Llama3 does a good job at following your instructions.
You also could do this using a bunch of RegEx and there are log parsing tools out there. This is just to demonstrate the concept of using LLMs to get back structured data. You might have to experiment with settings. In this case I in fact let the LLM also write the prompt and the structure of the resulting JSON file from a sample of the Logs.
Use KNIME nodes to extract and process the JSON Files
KNIME can now process the JSON structure within the results. In this case replacing the triple backticks with a placeholder and then using a RegEx extraction (from the Palladian extension) to get the pure JSON string back.
You can then convert the string(s) and extract the fields as lists and ungroup them to get a ‘regular’ table with columns.
These results you can now use to further process your data:
I hope you enjoyed the options you have with Llama3 and KNIME. If you want to read more about LLMs you can take a look at these articles or explore the KNIME Blog.
In addition you might also like these articles:
In case you enjoyed this story you can follow me on Medium (https://medium.com/@mlxl) or on the KNIME Hub (https://hub.knime.com/mlauber71) or KNIME Forum (https://forum.knime.com/u/mlauber71/summary).
