source: concord.walmartlabs.com

Creating Custom Concord Plugins

Concord is a workflow server. It is the orchestration engine that connects different systems together using scenarios and plugins created by users. More details about the tool can be found here.

Creating Custom Concord Tasks

Tasks, or Plugins, are executable Java code which are called from Concord workflows to perform actions. These actions are usually more complex than what can be accomplished with the basic DSL and Expression Language syntax supported by Concord’s YAML-based project definition.

This post will expand on the basic task documentation as well as point to actual implementation examples in various Concord task source code.

Invocation Methods

Tasks can be invoked in two main ways: Task Syntax and Expression Syntax.

Task Syntax

Task syntax is a YAML definition of the task call and parameters. The task’s execute method is invoked by the runtime.

Expression Syntax

Expression syntax calls a public method of a Task directly.

Source Examples:

1. Puppet Task — Execute method use with task syntax
2. Resource Task — Public methods used with expression syntax

Leverage The Concord SDK

The concord-sdk module must be used to implement the Task interface. Further, the module includes a number of other useful classes which make implementing a Task easier.

The ContextUtils and MapUtils classes provide methods which simplify wrangling variables from the Process in which the Task is called. Since variables are returned as Object from a Context object, these utility classes can be leveraged to avoid the boilerplate of casting the values to appropriate types and checking for null values.

The Concord SDK also includes interfaces which can be used for injecting services to be used within a Task such as SecretService for retrieving Secret values and DockerService for running Docker containers during Task execution.

Source Examples:

1. Docker Task — Uses ContextUtils to obtain task parameter values
2. Git Task — Uses SecretService to get private key

Default Parameters and Precedence

It is advisable to support global default for some or all task parameters. This allows certain parameters to be defined once and re-used implicitly with each task call. For example, if a flow calls the Puppet task multiple times, it can repeat itself much less by using a global variable to hold some of the task’s parameters.

The Puppet task checks a particular puppetParams variable for default values.

Process variables can be defined (and redefined) in a number of ways. Understanding the order of variable definition precedence is important when a Task retrieves variable values from a Context object. From lowest to highest precedence:

1. Process default variables defined by the Concord system
2. Variables defined in workflow’s configuration.arguments section
3. Variables defined when starting a workflow (e.g. curl -F arguments.myVar=myVal)
4. Variables created with the — set: call or context.setVariable('name', 'val') during flow execution
5. Task in-parameters and expression parameters

Source Examples:

1. Jenkins Task — Support default parameters in jenkinsParams variable
2. Puppet Task — Supports default parameters in puppetParams variable

Returning Data to the Process

When setting Process context variables in a Task, use a Map to make data access in the Process more intuitive. Don’t set multiple variables. In the Task, a simple Serializable class can be converted to a Map with Jackson’s ObjectMapper class.

Source Examples:

1. Jenkins Task — Builds a result Map
2. Puppet Task — Converts a PuppetResult object to a Map
3. MS Teams Task — Converts a Result object to a Map

Exception Handling

By default, Tasks should throw an exception with a helpful message when an unrecoverable error is encountered. The plugin can provide an argument to ignore errors (e.g. ignoreErrors) which gracefully exits the task and sets a Process context variable with the message.

Source Examples:

1. Slack Task
2. Taurus Task
3. Confluence Task

Extra Logging for Debugging

Default logging for a Task should be limited to exception details and important task progress in order to avoid an overly chatty Process log. For deeper debugging purposes, a Task can provide a debug option to output more detailed information on the Task’s operations.

Log only pertinent info my default. Allow for debug logging with an optional parameter.

Source Examples:

1. Ansible Task — debug and verbose parameters for extensive logging
2. HTTP Task — debug parameter outputs request and response data

Using External Artifacts and Executables

Tasks may require an external artifact to perform their actions. Rather than package these files with the Task, use Concord’sDependencyManager to resolve them at runtime. This cuts down the size of the Task’s compiled binary, which in turn makes Process startup more efficient. The external artifact will not be resolved until the task is called, and will be cache-able for subsequent processes.

Download dependencies at runtime.

Source Examples:

1. Terraform Task — Downloads and execute Terraform binary artifact

Working with Secrets

Tasks can access Secrets through use of SecretService. An instance of this class can be injected for use within the Task.

Source Examples:

1. Crypto Task — Exports any secret type in a flow
2. Ansible Task — Exports key pairs for ssh connections and file secrets for WinRM connections
3. Git Task — Exports key pairs to clone git repositories

Leverage Forms

Most Task implementations should be as quiet as possible and accept data through parameters; however, there is a use case for tasks to build and/or call a particularly complex dynamic form.

NOTE: Tasks cannot read values from a Form they execute. This is because the form call is executed after the task finishes.

Call Other Tasks (or Let other Tasks Call Your Task)

Tasks can be made even more useful when they are available to use by other Tasks. Most tasks can be called without any extra work to accommodate being called from another task. Just set the scope to provided in the Task’s pom.xml and configure the class to be injected at runtime. The big upside to relying on injection is that the Concord runtime will handle further dependency injection that the other Task may require.

There are some downsides to this approach. The other task may not be available as provided (most aren’t) or the code injected at runtime may be from a very different version which may have unintended behavior due to method refactoring or deprecation. A more reliable way to handle dependency versioning is to include them as transient dependencies (instead of provided) in the Task’s pom.xml and instantiate the class directly (instead of relying on injection at runtime). Be aware that if the other task relies on variable injection, it will fail.

The most considerate way to design a plugin for re-use with other plugins is to implement a client layer for the Task. That is, the Task itself calls a client class to handle the actual work, and other Tasks can call the same client class rather than the Task class which may have undocumented @InjectVariable usage. Further, it will likely be advantageous to provide a utility method or class that handles gathering the required parameters.