Scaling CA Endevor Web Services with Multiple Instances

Sometimes a Single Web Service Instance is Not Enough…

In the last couple of years, the number of options for interacting with CA Endevor off-host has grown dramatically. As more and more traffic comes from outside the mainframe environment, it is becoming critical to ensure that the APIs that service those applications are stable and able to scale to support larger and larger loads. Broadcom has made a number of improvements in the services themselves to address this: support for 64 bit Java and therefore larger heap memory, streaming of results and various other performance improvements. Despite this, sometimes the best way to deal with increased load is to scale horizontally by having multiple instances. Another benefit to this approach is you can increase up time by ensuring that when one instance goes down, others can take over and continue to provide service.

Added Instances = Added Maintenance?

You may be afraid that all of those instances might have impacts on maintenance. For instance, would you then need to update each one of the installation locations whenever there is a patch or new version released? Won’t that increase down time? In this case, one of our customers came up with an interesting solution to this and they agreed to let us share their configuration with you. With their set up, only a single directory containing the web services software is required and it can be used to launch multiple instances. This means less disk is used and maintenance becomes much easier. The following describes their configuration and a few different options for achieving the goal of running parallel, load balanced instances of the Endevor web services from a single install.

Customizing web services software files

To start an Endevor web service task on each LPAR you will need multiple versions of the ENDEVOR.cfg file so that the spawned task gets started on the same LPAR as the web service task that spawned it. More on this later.

Most customization that you make to the web service files can be shared in any instance, e.g. tomcat-users.xml or context.xml.

For this solution you need to update /tomcat/conf/context.xml to allow symbolic linking by adding these lines:

<Resources allowLinking=”true”>
</Resources>

As an Endevor administrator this customized version should of course be added to Endevor so rename the delivered version of context.xml and create a symbolic link:

mv context.xml context.xml.vanilla
ln -fs <endevor_dir>/context.xml <ws_software_dir>/tomcat/conf/context.xml

Implementing single Endevor web service software stack for multiple sysplexes

This solution makes use of the system variable &SYSPLEX and allows you to copy the same software stack to multiple sysplexes without changing the contents of the directories when copying.

server.xml

server.xml contains the IP address and port of the Endevor web service so this is unique to each web service instance.

In order to avoid the need to edit server.xml when copying the web service software from one plex to the next, you can create a symbolic link with the system variable &SYSPLEX, then create a uniquely named version of server.xml for each plex.

mv server.xml server.xml.vanilla
ln -fs £SYSSYMR/&SYSPLEX.server.xml.syml <ws_software_dir>/tomcat/conf/server.xml

If you don’t use Endevor to save your customization, then create one server.xml file per plex:

<plexname>.server.xml.syml

If you save your customization in Endevor then add a secondary symbolic link:

ln -fs <endevor_dir>/server.xml-<plex-meaningful-name> <ws_software_dir>/tomcat/conf/<plexname>.server.xml

Create one server.xml file per plex and add them to Endevor.

For server.xml, <ws_software_dir>/tomcat/conf would now contain:

server.xml                 Syml -> £SYSSYMR/&SYSPLEX.server.xml.syml
server.xml.vanilla         File
PLEX1.server.xml.syml      Syml -> <endevor_dir>/server.xml-PLEX1
PLEX2.server.xml.syml      Syml -> <endevor_dir>/server.xml-PLEX2

On PLEX1 <endevor_dir>/server.xml-PLEX1 will be used

On PLEX2 <endevor_dir>/server.xml-PLEX2 will be used

You will also need individual LPAR or plex directories for the logs:

<ws_software_dir>/conf/logs     Syml -> <ws_software_dir>/conf/lpar
<ws_software_dir>/conf/lpar     Syml -> £SYSSYMR/&SYSNAME
LPAR1                           Dir
LPAR2                           Dir
LPAR3                           Dir

Or the following if you want to use a link to an STC user directory for the logs:

<ws_software_dir>/conf/logs     Syml -> <stc-home-dir>/logs/lpar
<stc-home-dir>/logs/lpar        Syml -> £SYSSYMR/&SYSNAME
LPAR1                           Dir
LPAR2                           Dir
LPAR3                           Dir

Implementing Endevor web service on a multi LPAR sysplex with shared DASD

This solution makes use of the system variable &SYSNAME (LPAR name) and allows you to use the same software stack for multiple LPARs.

ENDEVOR.cfg

ENDEVOR.cfg contains the LPAR name so is unique to each LPAR, this is the “STC HostName” variable and defines the hostname/LPAR where the spawned task will be submitted.

mv ENDEVOR.cfg ENDEVOR.cfg.vanilla
ln -fs £SYSSYMR/&SYSNAME.ENDEVOR.cfg.syml <ws_software_dir>/webapps/endevor/ ENDEVOR.cfg

If you don’t use Endevor to save your customization, then create one ENDEVOR.cfg file per LPAR:

<LPAR_name>.ENDEVOR.cfg.syml

If you save your customization in Endevor then add a secondary symbolic link:

ln -fs <endevor_dir>/server.xml-<plex-meaningful-name> <ws_software_dir>/tomcat/conf/<plexname>.server.xml

Create one ENDEVOR.cfg file per LPAR and add them to Endevor.

For ENDEVOR.cfg, <ws_software_dir>/webapps/endevor/ would now contain:

ENDEVOR.cfg              Syml -> £SYSSYMR/&SYSNAME..ENDEVOR.cfg.syml
ENDEVOR.cfg.vanilla      File
LPAR1.ENDEVOR.cfg.syml   Syml -> <endevor_dir>/ENDEVOR.cfg-LPAR1
LPAR2.ENDEVOR.cfg.syml   Syml -> <endevor_dir>/ENDEVOR.cfg-LPAR2
LPAR3.ENDEVOR.cfg.syml   Syml -> <endevor_dir>/ENDEVOR.cfg-LPAR3
LPAR4.ENDEVOR.cfg.syml   Syml -> <endevor_dir>/ENDEVOR.cfg-LPAR4

Load Balancing Endevor Web Services on a Multi-LPAR Sysplex Using Zowe API Mediation Layer

One easy option for setting up load balancing with multiple Endevor instances is to use the Zowe API Mediation Layer. With the mediation layer in place, you simply need to register each of the instances behind the API ML so they share the same serviceId as described here.

Once configured, the API ML will automatically balance the load between instances that share the same serviceId.

Load Balancing Endevor Web Services on a Multi-LPAR Sysplex Using a z/OS Communications Server TCP/IP distributed VIPA

Another option for load balancing of Endevor web service requests is to start up the Endevor web service Tomcat task on each LPAR of a multi-LPAR Sysplex.

Set up a z/OS Communications Server TCP/IP distributed VIPA with the SERVERWLM parameter.

“If the DISTMethod SERVERWLM parameter is specified on the respective VIPADISTRIBUTE statement, the distributing stack selects from the available servers for a DVIPA/port and forwards connections based on a WLM recommendation indicating how well each server is executing on its system.”

In other words, the request will be sent to the best performing LPAR where the Endevor web service task is running, and if one LPAR is performing badly no requests will be sent there providing that there is a better performing LPAR with an Endevor web service task available.

More information on the IBM TCP/IP SERVERWLM parameter can be found on this web page.

Summary

A quick recap of what we’ve covered:

• Using multiple instances of CA Endevor web services helps to scale to high user loads, as well as provide high availability
• You can load balance using Zowe API Mediation Layer or Sysplex capabilities
• You can use a single software stack for all of the instances through clever use of symbolic links, reducing your maintenance work

Learn more about Zowe at this site; more Zowe blogs here.

If you have any questions or feedback or would like to talk about any of your CA Endevor use cases, please feel free to contact me at Vaughn.Marshall@broadcom.com.