Self-service Developer Workspaces in Endevor

Supports Agile Development Principles like Developer Isolation

Agile has become the methodology of choice for software development and for good reasons. Organizations compete in a marketplace driven by regular disruption so improving developer velocity is not only an advantage, it’s critical. To achieve maximum velocity, there are a number of key development practices that must be observed — ensuring developers can make their changes in isolation from other developers to avoid creating unnecessary dependencies is one of those.

In the world of distributed systems development, this is done by using branches to isolate your code and leveraging virtualization and containers to ensure changes can be built, deployed, and tested without impacting other developers’ work. But how does this translate to the world of mainframe development, especially where veteran mainframers prefer to work with traditional mainframe SCMs instead of Enterprise Git platforms?

Taking Inspiration from the Past

Parallel development in the world of mainframe is nothing new — organizations typically create parallel lifecycle paths in Endevor for this very purpose. They will have, for example, one or more parallel Endevor environments — DEV1, DEV2, DEV3, and so on — to allow developers to pick a place to start developing with the parallel changes merging at some point later in the life cycle. What is new is the practice of doing this for every separate feature or fix to ensure changes don’t get intertwined unnecessarily.

At Broadcom, we ran into that very problem early in our own Agile transformation so we created Endevor “sandboxes” — which use the application inventory classification field named “Subsystem” to act as a branch of sorts. Unfortunately, for many customers, this doesn’t work because it requires adjustment of all their build and lifecycle automation, an effort many administrators do not have the cycles to undertake. Additionally, not everyone’s application changes fit neatly into a single subsystem/sandbox. So we needed to come up with something better.

One way to make this more widely adoptable would be to take inspiration from how Endevor developers have always done parallel development — with parallel environments. Easier said than done! Setting up new environments for every feature or fix would create a huge administrative burden, not to mention being a great way to chew up a ton of unnecessary disk space.

Dynamic Environments in Endevor

Introducing Dynamic Environments and Deferred File Creation

Because developer isolation is so critical, we addressed those issues so that developers could isolate their changes regardless of how they choose to work. Dynamic Environments lets developers work with their own environment in Endevor whether they are working in ISPF or if they are using the environment to back their Zowe workspace or build their Git working directory.

To ensure that creating these environments did not become an administrative burden, we introduced new SCL that allowed you to define an existing environment like another reference environment, and a way to specify the backing datasets so they are only allocated on first use, thereby avoiding the unnecessary use of disk space that would be required by a full environment otherwise.

Getting Started

Enabling Dynamic Environments is really a 2-step process — enabling Dynamic Environments and then setting up Deferred File Creation to ensure the new environments don’t use unnecessary disk space. To enable Dynamic Environments, you must create a Global Control File (GCF) and then add a new parameter to the TYPE=MAIN macro in the C1DEFLTS defaults table. The GCF is a VSAM file that contains the environment definitions and related records for ACM. There is a sample JCL file for allocating the GCF contained in the CSIQJCL member BC1JJB08. The exact steps can also be found here.

To enable the Deferred File Creation feature, a member must be added to your PARMLIB data set that defines the parameters necessary to allocate files as new when they are being allocated for the first time. The Deferred File Creation member contains statements that will be parsed at element action execution time and used during file allocations if a file does not exist. You can use Endevor symbols to define your data set names within this member as well. Once you’ve added the new member, you need to define the name of the Deferred File Creation member to your C1DEFLTS table TYPE=MAIN section using the DFCMBR= parameter and then reassemble C1DEFLTS. The exact steps can also be found here.

One thing to keep in mind with these new environments is security. Access to define or modify environment information is controlled at the environment level by the “PRIMARY_OPTIONS” entry in the security table, BC1TNEQU. When MENUITEM is included in the PRIMARY_OPTIONS entry it resolves to a value for the primary option selected at run time, which for the ENVIRONMENT option is “ENVRMENT”. For example, in the delivered version of BC1TNEQU, we have this entry for PRIMARY_OPTIONS:

NAMEQU PRIMARY_OPTIONS, +
      L1=(‘C1’), +
      L2=(ENVIRONMENT), +
      L3=(‘PMENU’), +
      L4=(MENUITEM)

A request to define dynamic environment “WRK1 ‘’ would result in this pseudo data set name being checked for Read access: C1.WRK1.PMENU.ENVRMENT. If the user has read authority to C1.WRK1.PMENU.ENVRMENT, they will be able to define, update or delete the WRK1 Dynamic Environment, as well as the Systems, Subsystems, etc. within the environment. Note: Static environments defined in C1DEFLTS cannot be updated in this way, they must continue to be updated via changes to C1DEFLTS.

In order to avoid having to do continual security maintenance and facilitate self-service, having an environment naming convention is key. You may want to consider using a prefix or range of values that can be used for defining dynamic environments and creating a profile or standard access that can be given to a group of users for this purpose. For example, granting read access to this group for data set “C1.W%%%.PMENU.ENVRMENT” would enable them to define dynamic environments that begin with the letter W followed by 3 additional characters. Or you could create permissions for a set group of values to be used; WRK1, WRK2, WRK3, etc.

We recommend you incorporate the Dynamic environment name into all data sets and definitions that pertain to it. This includes the VSAM MCF files for the environment as well as the Base, Delta, Type, Source Output and Processor output files. This should make it easy to identify all of the data sets that belong to a dynamic environment and to establish rules to secure these data sets. For example, the base library for a type definition could be something like this, including the system, subsystem, environment, stage number and type in the data set name:

NDV.&C1SY..&C1SU..&C1EN..S&C1S#..&C1ELTYPE

More details can be found here.

Enabling Developers to Serve Themselves

To truly enable developers to go fast, you should consider making the creation of a new environment something developers can do themselves. There are a few approaches. If you are looking for an Endevor ISPF-native solution, you can enable the self-service creation environment via processor automation. You can, for example, have a type DYNENV which holds a simple text element defining the name of the requested environment and any input parameters needed for the automation such as the reference environment to create the new one as a clone. Invoking generate could then spawn the actual automation that creates the environment. Here is an example of what I mean.

When the SCL runs, definitions are copied as-is from one environment/system to another. One consideration is the fact that Endevor cannot automatically update symbolics; these are copied as part of cloning of TYPE definitions. If changes have to be made, that would have to be done after the cloning. It can also be done as part of the processor that creates a DYNENV element by giving the element content and using that content in the processor.

If you want even more control, such as including approvals by the Endevor admin, or would like to tie a form to the entire process you might also consider creating a front end for the process in a Service Catalog solution or a CI orchestrator like Jenkins. In this case, you’d be invoking the same SCL but you would submit it using Zowe CLI’s zowe endevor submit scl.

Using the Move processor with packages tied to approvals is another Endevor-native way you could add approvals to the mix.

What about Cleanup?

As with branches in Enterprise Git repositories, you probably don’t want all these project work areas to stick around forever so having a plan to clean up when development is done is recommended. If you want an Endevor ISPF-native solution, you can simply have a delete processor set up to run when the DYNENV elements described above are deleted. Here is an example of that.

Note: any environment containing elements that have not been promoted will not be allowed to be deleted (for good reason!) As an administrator, you can leverage the CSV utility (as well as Zowe CLI) to list out environments and find which ones are dynamic (via the “Dynamic” flag in the output). From there a simple List element CSV or Zowe CLI call will reveal if there are any elements left in the environment. If there are, you can look at the last update date. If that date ages out you can take action to follow up with the developer to clean up.

Taking it even further

Dynamic environments provide a project work area in native Endevor, accessible via ISPF or Quick Edit but in some cases, developers may want to access it via a branch in Git. If you are using an orchestrator to drive the process, consider using the new Bridge for Git Zowe CLI command that allows you to do just that as part of the overall self-service request:

zowe ebg branch add MY-CONTEXT MY-MAPPING MY-BRANCH -like TEMPLATE_BRANCH -endevor-environment DIFFERENT_ENVIRONMENT

Here’s an example using parameters:

zowe ebg branch add myorg dyn-env-repo-name de01 -like dev -endevor-environment DE01

So in the above example, I’m adding a new mapping to a Git repo in “myorg” called “dyn-env-repo-name” to a new branch called “de01” which maps the same systems and subsystems in the “dev” branch to the newly created dynamic environment called “DE01”.

You can even do any other environmental preparation activities you want to go with this. Things like spinning up associated CICS regions or zD&T instances, populating with test data, etc. If you are triggering this from a processor, a webhook can be used to layer an additional workflow that needs to run from off-host.

Summary

With Dynamic Environments & Deferred File Creation, Endevor developers can create the isolation so critical to achieving the velocity necessary to meet their customer’s demands in a highly competitive environment. At a high level:

• Dynamic environments support Agile by allowing developers to work in isolation and in parallel regardless of how developers choose to work
• Creating environments is a much lighter lift administratively
• Unnecessary disk usage is avoided with deferred file creation
• Add self-service to allow developers to create environments as needed
• Regularly clean up old environments automatically

Learn more about Zowe at this site; more Zowe blogs here.

If you have any questions or feedback or would like to talk about any of your Endevor use cases, please feel free to contact me at Vaughn.Marshall@broadcom.com.