Shifting Attention From CI Pipelines to DevOps Assembly Lines

By Akhil Yerrapragada and Nagasudeep Vemula

Published in

KTH Distributed Systems

13 min readApr 20, 2020

DevOps Assembly Lines will enable the future with more efficient Continuous Delivery

The crux of conversation revolving around DevOps from its inception has primarily been focused on continuous integration tools and practices and how to increase their efficiency. But ceilings have been reached and even at their peak it would seem that the upgrades in CI have just not been enough to deal with ever increasing demands regarding both speed and quality, a pairing the industry has been dying to attain for years. To solve this very problem things had to be seen from a different perspective and this has led in recent times to the rise of DevOps Assembly Lines.

A brief overview of what will be covered in this article:

Limitations of CI Pipeline
Assembly Lines overview
Challenges that can be solved using Assembly Lines
Benefits of Assembly Lines
Using Assembly Lines with Shippable tool
A vision to Assembly Line future

We hope to convince the reader by the end to inculcate assembly line practices in their own organization. Increased efficiency after all helps us all to focus on more important and value adding tasks and thereby live more fulfilling lives😄

Limitations of the CI Pipeline

We will not be delving into too much explanation regarding how continuous integration (CI) works in this article, instead we will take a look at its shortcomings as a result of which trends are starting to shift away from this approach and look towards newer methods like DevOps Assembly Lines.

As CI became mainstream, developers wanted to define more complex workflows (A sequence of automated and connected pipelines) with decision trees and parallelism, this led to increased implementation of CI pipelines which support such degrees of complexity. Some of the ways in which this is done are [2]:

Defining the CI workflow in stages, so you can get results from a stage faster without having to wait for the entire workflow to finish.
Running stages in parallel, so you can split tests or test your code against different environments or language versions, and get faster feedback.
Forking your CI logic based on the results of a stage.

However, there are certain limitations. Despite being a huge improvement over traditional CI, CI pipelines are mainly developer-focused workflows. Their very design and objective is to ensure developers get to push their code regularly and integrate it without any issues, however, other stakeholders and teams involved in the project have not been given their due consideration in this flow. It is still a very arcane process for members from other teams to weigh in on configuration management.

Various teams involved for instance such as test, implementation, security, infrastructure etc have to coordinate with developers on the project and to do this and be part of the configuration management, they need to use tools and platforms which are heterogeneous in nature. Such tools are typically from different organizations each with their own type of setup and frameworks as a result of which synchronicity between teams tends to be very difficult and merely burdens members with further problems to be solved.

To add insult to injury, such lack of orchestration between teams leads to lack of visibility on key metrics and manual integration i.e. scripting stages of pipeline and writing another script to make each of these pipeline scripts work together will be time consuming, error prone and very costly.

DevOps Assembly Lines ensure that distinct operations are automated in a connected, orchestrated and quick way. They enable different teams to inhabit a common workflow and automate the entire software development life cycle [3]. A DevOps Assembly Line standardizes the workflow across different types of tools, isolated systems(silos), and applications. Further they will connect these standardized elements to form an end-to-end process that ensures absolute visibility, traceability, and auditability from which all the teams are benefited. In the following sections we will give more background on assembly lines.

Assembly Lines overview

Merriam-Webster defines an assembly line as “ an arrangement of machines, equipment, and workers in which work passes from operation to operation in direct line until the product is assembled”. Let us see how we can we extend this definition for the context of DevOps.

Consider a factory where the finished product would be assembly of a vehicle. In the case of the DevOps assembly line the application is basically this vehicle. Just as separate parts are fixed in by varying machinery and processes here there are separate stages for the software development process and this enables the code to be moved to the production environment easily without any roadblocks.

The below image is a suitable representation of the above analogy, it is in fact even shaped like a car being assembled!! Each part of this line is an entire pipeline having specific needs with regards to the tools required to carry out operations, runtime environments, configuration management etc. They all work in collaboration with dependencies getting resolved by the automated nature of one pipeline in response to another and finally deliver the end product.

Figure 1 - An assembly line for a single application or service. Image courtesy: devops.com

Each of the boxes is a pipeline just for different activities. A DevOps assembly line is therefore a pipeline of pipelines, its Pipeception(hoping to make pipeception a thing, Merriam-Webster what do you think?)

Features Supported by the Assembly Line

According to XENONSTACK a digital transformation services provider, the following are features must be supported by an Assembly line. We will go more in depth into these in the following sections.

It should be considerably easy to define the workflow across multiple pipelines.
Scope for monitoring with the help of metrics to track performance among the different pipelines.
Integration with popular source control systems should be supported.
Take into account the rapid feature changes and scalability of requirements brought about by microservice implementations.
Visibility into each pipeline and stage, including logs, status, and versioned data.[4]

Tool Integrations

Some examples of tools that can be integrated among pipelines are:

Ansible for helping teams manage the configurations and also security patches.
Terraform for infrastructure management.
Vault for secrecy management with roles and authorizations provided.
Docker and Kubernetes for managing build and deployment related activities.

Figure 2 - An example of tools that can be integrated. Image courtesy: slideshare.net

Note that the above tools are well used but not well orchestrated among CI Pipelines. This is where the importance of assembly lines comes in. Assembly lines provides a simple and comfortable way of using and orchestrating pipelines which makes it super compatible with architectures such as microservices. How? the tool, shippable provides shippable.yml [7] file(Assembly line) to configure Jobs, Services, Integrations, Languages, Build, Matrix, Environment, Branches where everything can be orchestrated in one go. We shall discuss more on shippable in upcoming sections.

Challenges that can be solved using Assembly Lines

Over the years, many organizations have seen drastic changes in their delivery capabilities of quality products in an efficient manner as a result of pipeline automation and embracing the DevOps culture. However, for the current demands the existing models are found to have plateaued and it is proving difficult to measure for success due to the lack of a proper metric to evaluate for the same.

We will delve a bit into what these challenges are in the following portion and give a brief on how assembly lines can be harnessed to solve them [6]: -

Islands of Automation

The term islands are suitable for most of the tools provided by vendors as similar to an island these tools are isolated in their use case for a specific team. For instance, development, Test and Ops teams each have their own toolkit. As a result of this in order to automate the entire pipeline, teams are forced to work with multiple tools across disconnected pipelines. Automating these in isolation is creating ‘Islands of Automation’.

In these islands due to lack of visibility across the toolchain, finding bottlenecks is time consuming and challenging.

Figure 3 - DevOps Islands of Automation. Image Courtesy: dzone.com

True continuous deployment or delivery is a distant dream for these islands until a transition is made to new paradigm, mainly that offered by assembly lines.

How Assembly Lines can solve this 💻 💻

Assembly lines provide an approach that enables automated pipeline to pipeline collaboration which extends beyond a single team. The dependencies between the pipelines that need to collaborate is configured prior to operations and as a result any change made in a pipeline is easily propagated to all the dependent pipelines without the need for any human interaction.

Assembly lines enable tool collaboration by connecting the islands of automation into end to end workflows which enables paths between these islands which in turn leads to sharing and coordination.

Example: Let us consider an instance where the security team triggers a pipeline to fix a critical security hole in the operating system, the changes made by this fix must be reflected for all the teams working with the same. Assembly lines enable the entire IT portfolio to be updated and managed in a streamlined fashion.

2. Rapidly Evolving Application Architecture

The increasing speeds being offered is resulting in application architecture evolving rapidly namely from N-Tier to Service Oriented (SOA) to Microservices.

Microservices architecture has gotten increasingly pervasive. The applications are broken down into very small fragments and this creates an intricate web of dependencies within the different fragments. To account for this complexity the number of pipelines required to deliver such applications exponentially rises and the traditional methods such as Application Release Automation (ARA) are not nearly enough to manage the dependencies.

The approach needed to solve this issue would need to support non-linear and interconnected workflows.

How Assembly Lines can solve this 💻 💻

Assembly lines consist of features far more suited for the shipping of modern applications with their fragmented dependencies as compared to the ARA method. The primary difference being the provision of holistic and interconnected workflows which are akin to envisioning an interconnected network or ecosystem of pipelines that can collaborate as and when required.

The assembly lines maintain the information subjected to change- state information such as IP addresses of provisioned systems, public keys belonging to an encryption service, backup logs for databases etc and this information is made available to all the pipelines part of the network. As discussed for the challenge of islands of automation the connection of these islands to end to end workflows also helps in dealing in parallel with challenges arising from rapidly evolving application architecture.

Manual activities at endpoints which is what takes place in traditional methods like ARA are just not enough to keep up with the speed at which delivery is required and collaboration thus needs to be automated and streamlined between the teams and this is done with the help of the pipeline connection enabled in Assembly lines.

3. Heterogeneous Apps and Tools

The current technology landscape is subjected to rapid rates of change, virtual machines that once replaced physical servers are now being replaced with Docker Containers and in addition to this the programming languages and tools of choice are also changing from C++ to C# and Java and from Java to Node.js and Go. This rapid evolution has led to organizations having a mix of heterogeneous applications.

Legacy applications that are not containerized are also present in tandem with recently added applications that are fully on Docker. Add to this a mix of applications written in Node.js and consisting of different databases, queuing mechanisms and deployment endpoints and you have recipe for an impossible task of untangling of a complex web. This causes extreme difficulties in committing to a toolchain in the current setting (As if we didn’t have enough commitment issues to deal with amirite? 😅 😅)

We need to manage these heterogeneous applications in a consistent fashion without needing to rip and replace our workflows constantly. Who knows, if we manage to solve this, maybe government bodies can take a feather from this solution for preventing policies from changing every time a new leader is elected. For now, we will satisfy ourselves with solving the optimization of the workflows in our context of IT organizations.

How Assembly Lines can solve this 💻 💻

Assembly lines use a declarative yml based language which is easily readable and easy to write and author in. This lends a level of homogeneity in the flow which reduces the difficulties related to deciphering and general monitoring. This enables the sequencing of the DevOps activities into interconnected assembly lines, we discussed the benefits of interconnected lines for the solutions to challenges above and yet again in this context. Workflows are versioned and ready for reuse in assembly lines unlike traditional methods where reuse is not supported and the flow must start from scratch.

Native integrations for all tools and languages are made available and can be put into use easily with the help of assembly lines. Adding the new tool to the network can be done by simply editing a few lines of the YML code discussed above.

Benefits of DevOps Assembly Lines

Workflow Integration- As discussed before, the entire workflow is streamlined end to end with the help of automation between pipeline to pipeline without any human intervention. This makes it very easy for multiple teams to collaborate on a project and time is not lost unnecessarily on configuration or version management and monitoring [1].
True Continuous Delivery- Assembly lines connect the various tools and pipelines required into an interconnected workflow. This enables easy continuous deployment without the need for any human intervention.
Visibility- Teams in organizations currently exchange information through human mediums of interaction that are dependent on people such as emails, meetings, spreadsheets and slack conversations. This makes information retrieval and monitoring difficult as all these records must be parsed manually. Furthermore, for teams performing different functions, they need to explicitly raise a request for obtaining the information they need if it is with another team’s pipeline. With the help of assembly lines, the teams are aware of activities and have access to information across all the pipelines.
Business Intelligence and Analytics- Assembly lines contain information pertaining to all the teams thus making it easier to carry out business intelligence related analyses on this data and key metrics can be tracked for all teams. This data can thus be used to create a benchmark for optimizing the growth and improving performance of each pipeline.

Using Assembly Lines with Shippable tool

The primary first adopter(at least from its visibility and range of products) is Shippable, a company based in Seattle with roughly 50 employees. Available as a hosted SaaS offering as well as a server, it is a platform for DevOps Assembly Lines that enables DevOps teams make software releases frequent, error free and reliable. Other services include CI, CD and ITOps.

Some of the core features provided by Shippable to implement assembly lines are:

Workflow automation- YAML declarative based language is used as discussed in challenges above. shippable.yml eases the job of configuring all dependent pipelines by providing access to info they need at one place.
Integrations- Integrations in Shippable are used for the management of secrets like keys, tokens, passwords that are required for the applications. With the help of these users can come up with roles and allocations enabling the integrations to be available in a custom manner to whom its specified.
Runtime- The devops activities which are configured in advance need to be executed. This is where provision of a runtime comes into play, the platform provides runtime with installed packages, dependencies and configured command line interfaces as per our automation scripts or requirements.
Visibility- One of the main benefits of assembly lines is the increased visibility of information they provide to all the teams involved. In Shippable, views can be customized to focus on specific metrics that need to be measured for a particular assembly line. In addition to these reports, analytics and audits on performance of the software workflows are provided.

Let us now look at some key areas in shippable tool that help us to configure assembly lines.

“Manage Assembly Line” window

Figure 4 depicts the column (Manage Assembly Line) where shippable allows to configure assembly lines for any project available in version control. This column will be made available when a version control is integrated to subscriptions.

Figure 4 - Assembly lines with shippable

“Resources” tab

A project that is managed with an assembly line need to have shippable.yml file and needs to be committed at the root of repository. As explained earlier, it is the one and only place that will be configured with all the dependencies of project. It will be provided as a resource [8], namely “syncRepo” [9], which helps define assembly lines. syncRepo is pivotal and is the only part configured from the user interface. Figure 5 depicts Type (syncRepo) in resources.

Figure 5- syncRepo in Resources. Image Courtesy:

The “SPOG”

Single Pane of Glass (SPOG) [10] is the area that displays all the pipelines in shippable.yml. This is the view of workflow. It portrays building end to end continuous delivery that is event driven and stateful. Figure 6 depicts multiple islands (source control, test and deployment) connecting each other in SPOG view.

Figure 6 - Automating the Flow. Image Courtesy: docs.shippable.com

Since Jfrog’s acquisition of shippable, assembly lines usage has been extended further such as integration with JFrog Artifactory [11]. Some intriguing tutorials to implement assembly lines can be found here [12].

A vision to Assembly Line future

Starting from the year 2020 itself there will be transitions from simple CI pipelines to modular DevOps Assembly Lines [5]. This will help in giving a consistent and scalable workflow which in turn leads to ease of organizing and integrating activities.

If enterprises are truly focused on speeding up the delivery process, their entire software development cycle must be automated. Upcoming trends will tend to look more towards Continuous Delivery rather than Continuous Integration and we greatly believe that this will lead to a significant attention shift towards DevOps Assembly lines. We hope our article helps more of you out there to understand the need and ease of this transition and we are very excited to see the advancements in DevOps implementation strategies in the next few years.