Turn cache on in HCL Commerce v9
A caching infrastructure is a key feature for every e-commerce website.
HCL Commerce can offer a sophisticated caching layer to implement a complex caching strategy to handle a considerable amount of traffic.
If your production environment is a Commerce cluster composed of many nodes, one of the key features of this strategy is the ability to centralise the cache contents in an external object grid using IBM eXtreme Scale (WXS).
In HCL Commerce, the cached content is grouped in cache instances and the workload pattern of every instance will determine if it’s suitable to be stored in WXS or kept in the local WebSphere Dynacache.
IBM published some guidelines a while ago (but the principles are still valid) and a set of performance tests will help you to understand how to distribute the cache instances (especially the Commerce data cache).
In this post, we’ll explain to you how to set up a local Commerce authoring environment where the baseCache instance is connected to a local WXS grid extending the docker-compose file distributed by HCL Commerce.
We tested this configuration on Windows 10 with Docker desktop version 2.2.0.5 (Engine 19.03.8) and more importantly, we have to dedicate an important part of our laptop resources to get it to work
XSLD, eXtreme Scale in a container
The first piece to configure in our local Commerce authoring environment is the WXS server. Following the basic principle of idempotency, we want to create an environment based on code (docker-compose) “can be repeated an arbitrary number of times and the result will be the same as if it had been done only once”.
Then we chose to rely on the Docker distribution of WXS, WebSphere eXtreme Scale Liberty Deployment (XSLD) available in the Docker Hub. The image is based on the 8.6.1.4 version of WXS and the documentation is available on the WXS knowledge center.
Add the XSLD service to the HCL Commerce docker-compose file
To start our WXS integration, we want to add the XSLD image to the docker-compose file proposed by HCL Commerce in the knowledge center.
To simplify our deployment and keep it idempotent, we’re using the docker-compose version where DB2 is executed as a container.
We added then the XSLD service to the existing file
A couple of comments on this new service.
We had to build a custom image based on the original IBM to add curl command: it was needed to implement the healthcheck. And we had to update the docker-compose version to 2.3 in the configuration file, for healthcheck full support.
The container will be considered “healthy” only if WXS admin console will be loaded and we added curl command to do the check (neither wget or nc were available in the image).
We extended the port intervals to ensure XIO communications between Commerce application servers (WebSphere and Liberty) to work properly. The original port interval for XIO was
-p 4809–4821:4809–4821
but applying the XSLD guidelines for XIO, we changed the interval to
ports:
- “4809–4848:4809–4848”
And finally, we created a custom network to allow the communication between the host and the deployed containers with this command
docker network create — driver bridge xsld-net
and we added it to the docker-compose file
Note: again, not tested in Mac. Windows only.
XSLD is up and running: time to connect it!
Now we have an XSLD container up and running but our application servers are not using the remote cache data store. This is because we didn’t configure the WXS client deployed in ts-app, search-app and crs-app
To do so, we have to follow the integration process described in the HCL knowledge center and create three custom builds for ts-app, search-app and crs-app that includes the run engine commands necessary to enable the connection to WXS and move the designated cache instances there.
Following the HCL example, for ts-app the Dockerfile would be
for search-app
and crs-app
The application servers are linked to XSLD, but what about dependencies?
Yes, ts-app and search-app can connect to WXS after we created the custom images with previous Dockerfiles but initially XSLD will be raised up completely empty, without any grids.
Then we need to create a DYNACACHE_REMOTE grid in XSLD before WXS clients in ts-app, search-app and crs-app try to connect to the cache repository; otherwise, the connection will fail and the application servers will not startup properly.
These dependencies have to be materialized in our docker-compose file to ensure the application server will find the expected cache grid hosted in WXS when they’re starting up.
The correct startup sequence to accomplish the dependencies has to be:
- db container starts up
- db container started
- search-app starts up
- XSLD starts up
- XSLD container healthy
- DYNACACHE_REMOTE grid created in XSLD
- ts-app starts up
- ts-app container healthy
- crs-app starts up
Then we first added to the docker-compose file the dependency from ts-app to XSLD service
and we added the reference to the default network to ensure connectivity between the containers and the host machine.
Ok, but how to create the grid in WXS before the dependent services?
To comply with the described startup sequence, we have to wait until XSLD is fully started up and the DYNACACHE_REMOTE grid created before trigger ts-app container creation.
We decided to force this sequence using a support container called commerce-services. This image will contain the wait-for-it script created by Giles Hall (but you can use a similar one) and the json file needed to create the grid using WXS REST API.
Declaring the service in the docker-compose file
we’ll implement the required startup logic.
When XSLD container will be considered healthy, commerce-services container will start up executing the command
sh -c ‘/wait-for-it.sh xsld-1:9443 && curl -k -X POST -u xsadmin:xsadmin4Me! -H Content-Type:application/json https://xsld-1:9445/wxsadmin/v1/grid/DYNACACHE_REMOTE -d@baseCache.json’
The commerce-services container will wait for XSLD admin console to be available and will trigger the grid creation with the WXS REST POST command using these parameters
ts-app container raised up at the same time we’re executing the WXS REST command (the event is “XSLD container healthy” for both) but the grid is normally created in a few seconds and ts-app needs at least 2 minutes to look for the DYNACACHE_REMOTE grid object. Then the procedure is safe but it could be improved.
Every container is connected to WXS! Let’s fun begins!
We finally implemented the right startup sequence then we can first build our custom images
docker-compose -f docker-compose-auth-wxs-v9 --env-file .env build
Once all images are ready, it’s time to run the docker-compose configuration
docker-compose -f docker-compose-auth-wxs-v9.yml --env-file .env up
Going through the generated log, we can see the grid getting created by commerce-services container
services_1 |[0m wait-for-it.sh: waiting 15 seconds for xsld-1:9443
services_1 |[0m wait-for-it.sh: xsld-1:9443 is available after 0 seconds
services_1 |[0m {“taskid”:102}
docker-compose_services_1 exited with code 0
Checking the WXS console, we can see the grid as expected
and containers startup logs will report the successful connection with the grid
txn_1 |[0m [4/17/20 9:35:03:542 GMT] 00000001 ClientDomainC I CWOBJ1126I: The ObjectGrid client connected to the DYNACACHE_REMOTE grid in the cg1 domain using connection 0.
search_master_1 |[0m CWOBJ1126I: The ObjectGrid client connected to the DYNACACHE_REMOTE grid in the cg1 domain using connection 6.
store_1 |[0m CWOBJ1126I: The ObjectGrid client connected to the DYNACACHE_REMOTE grid in the cg1 domain using connection 0.
Open Aurora store and check the cache entries
Now all the Commerce infrastructure is up and running, it’s time to access Aurora store and navigate to verify cache entries are created.
First of all, we have to create the Solr index to navigate the catalog
curl -k -u spiuser:passw0rd -X POST https://localhost:5443/wcs/resources/admin/index/dataImport/build?masterCatalogId=10001
and once the index is created, we can open Aurora store homepage
and navigate to the Shoes category
Rendering this page, we’re triggering the creation of several entries for one of the cache instances we moved to WXS
run connect-searchcache-wxs services/cache/SearchFacetDistributedMapCache 2000
Then going back to WXS console, we can see the entries created
It worked! We’re storing remotely Commerce cache entries in our fresh new grid!
Conclusions
We progressively enhanced the original HCL knowledge center article to integrate WXS and you can find the sources to test the integration on your own here.
Just a note on executing the docker-compose file: I had an issue on Windows about path evaluation and I created a pwd_var.bat to fix it. You should run the bat script to allow the .env file to get created. Remember also that this docker-compose configuration is heavily consuming memory and disk in your laptop: please be patient.
Thank you for reading this last line: it was an extensive tutorial :-)
Any comments will be greatly appreciated.
