How to re-execute failed nodes automatically in KNIME

Have you ever encountered the situation where for some unexpected reason one of your nodes fails, and the workflow stops? There you must re-execute the node manually to run the workflow. Running the workflow in KNIME Server gives you the option to execute the workflow automatically on a schedule but what if you need to re-execute the workflow right away?

Here we have an example workflow in which we are trying to get a list of country names using an API. To do that, we pass a URL to the Get Request node. If we lose our internet connection, the node produces missing value by default. But we can change this behavior by selecting the “Fail on connection problems” option in the configuration window to get an error instead of missing value. Losing internet connection is one of those unexpected cases we mentioned earlier. Maybe the connection is lost for a brief amount of time (longer than the timeout value), but that is enough to make our node fail.

Now let’s see how we can overcome this issue:

First, we use a Table Creator node to generate the URL which will be passed to the Get Request node. The API which we are using to get the country names is available at this URL:

http://countryapi.gear.host/v1/Country/getCountries

We input the URL, name the column as “url” and run the node. In a typical flow, we would use the Get Request node to take the URL, send the request and get the response and so on. Let’s be typical and do the same for now! Pick the Get Request node and in the configuration window of the node, select the “url” column for the URL column option and check the “Fail on connection problems” option.

Execute the node and then pick a JSON To XML node to convert the response. Replace the “body” column in the configuration window and execute this node as well. Then we need an XPath node to extract the country names. Click on “Add XPath” and input this XPath:

/root/Response/item/Name

We enter Name as the New column name option and select Multiple Rows.

Let’s save the settings and execute the node. Now we have the country names in a column named “Name”. We use a Column Filter node to include the “Name” column and exclude all the other columns.

So, for now, we have our desired output. But what if the connection is lost while the workflow is executing? The execution of the workflow would fail with an error from the Get Request node even if the problem is temporary (longer than the timeout that we set).

We are going to use KNIME error handling nodes in a loop to overcome this issue.

Just before our Get Request node, we put two nodes: A Generic Loop Start node and a Try (Data Ports) node. These nodes have no specific configurations. An optional Wait node is recommended to prevent the re-execution process from going crazy! You can put this Wait node anywhere inside the loop.

Now just after the Column Filter node, we put a Catch Errors (Data Ports) node. This node also does not require any configurations but needs another input for the second input port. This is the data port which will be passed if an error occurs inside the Try/Catch construct. So, what should we feed this port to achieve our goal? To answer this question let’s go a bit forward. We want to close the loop using a Variable Condition Loop End node in which we set some value as the condition to determine if the loop should be repeated or finished. So, we need to generate some value which we call it the “error value” here. This error value should be passed to the loop end node if an error occurs inside our Try/Catch construct. You know how to do this. Yes, we input this value inside a table by using another Table Creator node and pass it to the second input port of the Catch Errors node. Just be aware that we MUST provide the same table structure as the one which would be fed to the first input port of Catch Errors node. Otherwise, our Variable Condition Loop End will object the difference. By saying “the same structure” we mean the column names, the column types and the order of the columns.

So, let’s pick a new Table Creator node and create a column named “Name” and input some value into the first row. This value should be something unique and different from the values we expect to have in our main table. For example, if we input “Afghanistan”, then our loop will never stop since the first value in our list of countries is “Afghanistan”. Here we choose the value “error” and execute the node. Now the output port of this Table Creator node should be connected to the second input port of the Catch Errors (Data Ports) node.

We have one final step to complete the process. A Table Row to Variable node to transform our data table values to flow variables and feed the loop end node. In our case, the output of this node would be a flow variable named “Name” with the value of “error” or “Afghanistan” depending on whether we encounter an error or not. The output of this node then should be fed to the Variable Condition Loop End node, and in the configuration window of the loop end node, we select the “Name” flow variable and choose the “not equal” condition (!=) and input our “error” value. So, the loop keeps repeating unless the value of the “Name” flow variable is not “error”. We also should check the option “Collect rows from the last iteration only” so that we only keep the table coming from our main flow.

Well, that’s it. Now if you disconnect from the internet and run the workflow, it would not stop with the error from the Get Request node and keeps looping till you reconnect to the internet so that the Get Request node will execute successfully and the loop ends.

Be very careful with using this trick. It can cause an infinite loop if you do not consider all the conditions like the one we mentioned earlier about choosing the error value or having a permanent error situation.

You can download the workflow here and watch the video of this tutorial here:

Originally published at https://blog.statinfer.com.