DATA STORIES | TIPS AND TRICKS | KNIME ANALYTICS PLATFORM
Mastering KNIME: Unlocking Peak Performance with Expert Tips and Smart Settings
The low-code tool KNIME is great and a lot of guides are out there to help you master it. You are encouraged to read the official ones. Here are some more hints…
You can just start using the data analytics tool KNIME and be happy. Over time some questions, maybe quirks and even problems might (or might not) occur and this blog is here to help you — with my 10+ years of experience with KNIME and a lot of conversations from the KNIME Forum (https://forum.knime.com/). What follows is relevant at least up to version 4.7.x — the new KNIME 5.1 and KNIME 5.2+ will have a different user interface and concept so these things might not apply though the general concepts mentioned do.
If you encounter a problem you also might want to ask a question in the KNIME Forum. To do that: read how to write a good forum entry or ask for help, then provide a sample workflow and maybe provide a detailed log file in KNIME 4 or DEBUG-Log in KNIME 5.
A large open source tool
Though seemingly obvious, KNIME is a quite extensive open source tool based on Java (and some Python lately) and it is undergoing dynamic developments with new releases every few months and it runs on Windows, Mac and Linux — so some issues might appear and the graphical presentation needs a fair amount of resources. And as with all software, you will need to familiarize yourself with the environment so maybe start with the User Guide (here is the one for the latest old GUI 4.7 and the new interface 5.x):
Overview of Topics you might want to consider …
Please be aware: there might not be one perfect setting for every task and system. Read the entries and find the ones that fit your needs. And be aware: often there is no substituting a good setup and strong hardware performance with super clever settings — having said that there are some things you could check out (more details on the subjects further down the article):
- Overall performance of your system (allocated RAM, CPU, Disk speed). Give KNIME what you can but leave some for your operating system and other tasks (20+%) (more details below).
- Permissions — do you (and KNIME) have full access to all files and disk spaces (there was an issue with MacOS once)?
- Virus scanner — KNIME has a lot of small files, make sure no virus scanner is aggressively blocking them.
- Settings about data storage and Java heap space (and temporary folders) — we come to that in a moment.
- If you encounter problems maybe restart your KNIME once with a “-clean” option in the knime.ini.
- Maybe do a clean new installation in a new folder (if you are on a Windows system just un-zip the zip version into a clean folder) — https://www.knime.com/downloads.
- Check the integrity of your storage/disk — be aware of possible issues with OneDrive [special characters like hash (#) and brackets “(“ “)” must be allowed/supported] (more details below).
- Think about a suitable backup strategy for your work. Things can go wrong. You might want to revert to an earlier version of your work (more details below).
On to more details …
Know your paths and folders
I would always encourage people to know about their system and their paths — which data is stored where. I often see people unaware where their files are and what they mean, and then you might run into problems.
You are encouraged to read the “KNIME File Handling Guide” — yes it is lengthy but then you will know all you need about how KNIME handles files and paths.
These are the three storage locations for KNIME:
- The folder where the KNIME Software resides. You can delete or re-install the software and even keep several versions in parallel. Just keep in mind: you will be able to open older knime-workflows but newer ones might not be used with older software versions.
- The “knime-workspace”. This is where your actual workflows (your work) is being stored. This data you should protect, version and backup! You can switch workspaces and you can import and export the files either as a single file or as a folder (structure).
- The KNIME temp space. Here you will have the temporary files KNIME uses.
You should learn how to configure these File and Folder Variables and how to use them:
Also: you might not be immediately aware but you can capture a configuration you have entered (like a path for a file to save) and use that in a Flow Variable. To be honest I oversaw this option for quite a long time …
KNIME Memory and Performance
One constant theme is the performance of KNIME workflows. One important thing is the physical memory (RAM) allocated to KNIME — this will show up as “Java Heap Space”. Since KNIME ist a visual/graphic tool that would store the intermediate data in each node it will need a good amount of resources and especially memory.
Somewhat older but still relevant blog: “Optimizing KNIME workflows for performance”
The default setting is 2 GB of RAM out of the box. While this is good for a lot of tasks, you might need more. A rule of thumb for me is half or 2/3 of your systems memory if you plan on doing a lot of things with KNIME, and also depending on the size of your data.
You will have to locate the knime.ini file and change the value “-Xmx” like this:
- -Xmx4g (4 GB)
- -Xmx10g (10 GB)
While giving KNIME RAM is good you should leave something for your system otherwise everything might get stuck.
For the new KNIME 5 which uses Apache Arrow under the hood you should also take into consideration that other processes like Python might also need RAM that is not ‘covered’ by the settings in KNIME. So if you experience problems you should try to adapt the settings to something like 60 KNIME vs. 40 for the rest.
OK, and disk speed is still a thing. An old HDD might not be the best choice. Opt for a fast SSD!
KNIME internal data storage
One thing to check is the internal storage format of KNIME. In more recent versions the underlying storage has been shifted to Apache Arrow underneath. Which also improves the data transfer between KNIME and Python. Some initial problems with that have been fixed.
In KNIME 5.x you can choose between Row-Based Backend and Columnar Backend. The later should perform better but might have challenges for some data types. You can set this either in the KNIME preferences of for each workflow individually:
Speed up your KNIME workflows
There are several things you can do to speed up your KNIME workflows. More often than not it will have to do with some planning about what you want do and bringing that in sync with your resources.
Also consider this:
- Java Garbage Collection can help to free memory (other hints in the link).
- You can use the Cache node to collect data at certain points in your workflow. Might be a good thing to place this at the end of complex operations and before exports.
- You can decide to not save parts of your workflow to save space.
- Streaming might be an option to speed op some operations. You will get the results for a chunk of rows faster instead of doing each step for all rows and then move on — depends on your task.
- Another option is Parallel Execution — which of course would need more resources but might speed up your tasks.
These are options to consider if you have special use cases or experience problems. Please do not just randomly implement these.
If you have to handle very large files there are some strategies using formats like Parquet or H2/SQL, see: “KNIME Snippets (1): Collect and Restore — or how to handle many large files and resume loops”
In case these settings are not enough you can always split you work:
Split you workflows if they become too large
If your workflows become too large and you can no longer handle them safely (and you cannot increase the resources) on option is to split them into a main and (several) sub-workflows (see example one, example two, example three, Example four with KNIME 5 — best to download the whole workflow group).
Backup is only for the faint of heart?
Backup is important with all your workflows and data and regardless of KNIME or other programs. Keep in mind: KNIME desktop does not have a recycle bin mechanism (yet) where you can recover your workflows if you accidentally delete them in the KNIME Explorer window — the commercial KNIME Business Hub does have such a recovery mechanism. KNIME although does have an auto-save mechanism for open workflows.
With backup also think of versioning — the KNIME Business Hub would help with built in versioning and collaboration but you might also consider it for your KNIME desktop version.
You can have your own improvised local Backup with KNIME nodes — it is not pretty but it would work :-). Also more recently I built a workflow that would pack your project folder into ZIP-Chunks and do a backup with a timestamp (https://hub.knime.com/s/wIUpbtzXabhKEfsI)
Another example will have a backup workflow within your KNIME project and store everything in a .zip file that you can store on a cloud drive. Also there is this example of how to do this by the great mwiegand:
Another way to store your data is using a cloud service. In any case:
Take care of your workflows and data!
If something happens to your workflows and you do not have backup: Stop what you are doing and seek professional help in restoring the data from your hard drive! Yes this will cost money but you have to decide if loosing the work will cost you more.
KNIME and Clouds — OneDrive and DropBox
You can have your workflows on a cloud drive as long as the data is (also) present on your local folder and you keep some things in mind. In general very long paths and file names might give you problems, so try to avoid them.
Concerning MS OneDrive these are the things to consider:
- Some versions of OneDrive have severe limitations about special characters that KNIME does need (# hashtags namely). Please make sure your version would allow them. Otherwise you might damage your workflows. If you have your workflows in folders instead of (zipped) .knwf files you can try and import them from a folder.
- OneDrive also has limitations about the length of names. So if you have very deep paths with long names you might encounter problems. Keep in mind that KNIME adds sub-folders beneath the primary workflow-folder.
If you plan on using clouds to work with KNIME on several systems you might want to consider some important aspects:
- Make sure to fully sync your data after closing KNIME.
- Do not have KNIME open on both systems at the same time.
- Make sure you have a backup nonetheless.
Further dark arts and KNIME Tuning
If you still encounter problems with performance you can dive deeper into the dark arts of tuning other parameters in KNIME. One place to explore is the KNIME FAQ.
- Although the Column Rename node offers the possibility to also change the data type this might not be a good option — cf. this thread for details.
- Some possible changes with the internal settings of the data storge “SNAPPY” (proceed with caution).
- Some deeper settings in the knime.ini file (https://docs.knime.com/latest/analytics_platform_workbench_guide/index.html#setting-up-knime-ini).
- If you want to handel very large Excel files you can try to employ Python / OpenPyxl.
Again: these could be measures of last resort or for the very experienced users. Before diving into these muddy waters try thinking about your tasks and plannings and maybe ask in the KNIME Forum, which is a good idea anyway :-)
Further things to read about tweaking KNIME:
.. like these useful general considerations regarding performance:
If you enjoyed this blog make sure to follow me on Medium and also on the KNIME Community Hub and Forum (https://hub.knime.com/mlauber71).
