Running ts-utils as a Kubernetes batch job

Photo by Matteo Grassi on Unsplash

From an operational standpoint, one of the most important new features available in HCL Commerce v9 is the introduction of the Utility image.

There is very little documentation about ts-utils on the current HCL Commerce knowledge center and it exclusively refers to how to use it in your local Commerce developer. In fact, ts-utils is not even included in the out-of-the-box Commerce Helm chart.

However, ts-utils has the potential to be a very powerful tool to execute Commerce related jobs if used properly.

HCL Commerce v9 deployment

Why the Utility server is potentially very useful

In the previous versions of Commerce, every time you had to run an administrative task from the command line using one of the many Commerce utilities, you had to create a specific Commerce runtime instance to have all Commerce libraries available.

An additional Commerce installation was needed just to execute dataload or stagingprop without affecting the performance of other environments. And things got even more complicated if the utility to execute was, for example, wcs_encrypt which required aligned Commerce instance information like merchant key to work properly.

Fortunately, Commerce v9 makes everything so much easier!

Just spin up a new ts-utils container and you’re ready to go: you can execute several Commerce utilities specifying the needed parameters on the command line to create the right context.

On the other hand, if you need to execute a Commerce utility that needs a concrete Commerce instance configuration, you can leverage the existing run engine commands available, adding something like:

run update-datasource-db <jndi> <database> <dbHost> <dbPort> <sslConnect>

to our custom ts-utils Dockerfile.

Why use Utility server as a Kubernetes job?

From Commerce v9, we have the ability to deploy in a Docker orchestration platform and Kubernetes is the most used container orchestration platform.

The flexibility of the Utility server can be combined with the granularity of Kubernetes job objects.

A job in Kubernetes allows executing a single or a complex set of logical operations from the batch job to the parallel processing leveraging templates.

Combining these Kubernetes job capabilities with Utility server flexibility, we can orchestrate complex Commerce operations like parallel dataload or staging propagation in a Continuous Integration pipeline.

This can allows us to finally bypass a historical limitation of Commerce utilities: the single-threaded execution.

For dataload for example, the CI process can create different sets of configuration files and spin up a full set of ts-utils containers to execute dataload in parallel.

Or for stagingprop execution, the out-of-the-box single-threaded staging propagation can be segmented and parallelized using different staging filters.

Executing our first Kubernetes job with the v9 Utility server!

But before running, we have to start walking.

Photo by Jukan Tateisi on Unsplash

As spinning up a ts-utils container is pretty easy, we’ll start creating a specialized instance of the Utility server connecting to an Oracle database and execute the dbupdate utility to upgrade the database schema to the latest Commerce level.

To connect the generic Utility server available out-of-the-box to an Oracle schema, we can use the CONFIGURE_MODE = Vault as explained in my previous post, or we can pass the needed parameter values to the container environment using the CONFIGURE_MODE = EnvVariables.

First, we build a properties file with all the environment variables that allows switching from the standard DB2 connection to the Oracle database.

Then we load this file in the Kubernetes Commerce cluster as a configMap object (oracle-config-test).

kubectl create configmap oracle-config-test — from-env-file=oracleConnectionEnvs.properties

Oracle environment variables as a configMap

Startup ts-utils container as a job

Now we’re ready to execute the simplest version of our job: no commands will be executed but Commerce entrypoint execution will connect the container to our Oracle backend passing the needed parameters through envFrom field:

First basic k8s job for ts-utils

We can then deploy the job on our Kubernetes cluster…

kubectl create -f ts-utils-first-job.yaml

…and get the confirmation of its creation…

job.batch/ts-utils-first-job created

…shortly the ts-utils pod will be up and running…

NAME READY STATUS RESTARTS AGE
ts-utils-first-job-4ffh8 1/1 Running 0 5s

…connecting to the container, we can confirm the startup was successful…

kubectl logs ts-utils-first-job-4ffh8 -f

ts-utils startup log without executing any command

As you can see in the logs, some Commerce run engine commands are executed as part of Commerce startup sequence.

Run updatedb command on our job

Next step, we’ll be to launch the command we want to test using the available Commerce extension point.

Reviewing the Docker start up process for Commerce images, you can add your custom logic on a specific file called

/SETUP/bin/custConfiguration.sh

Then we’re including our call to updatedb utility command in custConfiguration.sh and to get it available to the container, we transform the script into a configMap object

Modifying the previous job yaml file, we’ll create a new job file where the script is mounted as a volume (thank you all in this thread for the suggestions).

The interesting part is that we run the utility using the out-of-the-box pattern…

./updatedb.sh -instanceName demo

…because passing the custom environment parameters, Commerce startup scripts did the work for us configuring all instance values properly…

Update instance property file

As you can see in the resulting container log…

…the command is executed as expected right after the container startup (custConfiguration.sh)

Launch customization configuration
 — — — — — — — — — — — — — — — — — — — — — 
updatedb execution
 — — — — — — — — — — — — — — — — — — — — —

Conclusions

Commerce operational processes can be redesigned using the Utility server and Kubernetes jobs.

Inserting the flexibility offered by this new tool, we can orchestrate very complex processes for WCB, dataload and other utilities stepping over single-threaded limitation.

More complex examples will come in the next posts: stay tuned!

And don’t forget to vote and contribute via the HCL Commerce Ideas website!