Using automation to power a next-gen development workflow

2020…what a ride! My team and I have been quite fortunate to have been kept busy with lots of new projects this year. We’re finally maturing out of the the start up infancy stage, and are now embarking on a scary but exciting journey towards the next level of scale as more and more businesses find out how our technology services can best help them serve their customers.

Growing pains

One of the many challenges that this new level of scale brings us as a tech company, is retaining our ability to deliver at pace whilst extending the number of contexts, dimensions and channels the different teams within the company must operate on. Richard Hackman’s famous team communication diagram quite effectively displays how this manifests itself within growing teams by clearly surfacing the exponential growth of communication channels as numbers within an org increase.

More people, more problems

This growth in complexity can be even more pronounced when coupled with a software delivery process that heavily relies on effective and clear communication happening between the right parties at the right time.

Breaking down the development workflow

One of the key requirements to creating an effective delivery process is having a thorough and clear understanding of how the team delivering the work are likely to structure their development. A combination of rich literature on well defined (and refined) design principles made available through various mediums, a good amount of proliferation in cloud services and an industry wide embrace for open source software has thankfully brought about some level of convergence in the components that together form a robust software delivery workflow.

• Structured change management and auditing capabilities

Software development teams need a convenient way which enables them to quickly and reliably see how the code has changed over time, in particular what changed and at what point. This is especially crucial for enabling reliable rollbacks if (when) things go wrong.

• Facilitating the generation of common code segments with internal modules, harnesses and templates

One of, if not, the only guaranteed strategy known for being effective when it comes to preventing software defects is to not writing any software at all. With libraries and frameworks being at the helm of most production apps, a lot of code nowadays is actually setup of boilerplate code or configuration. Creating easy to use code generating harnesses, using easily duplicated code templates or creating internal libraries to allow developers to not to have to worry about this at all are all common and well understood practices within software development teams.

• A Technical design and documentation process

A good way to measure the developer user experience of a software system itself is to judge how easy it is for a new member of the team to own and deliver a code change to production. The key differentiator here is almost always the levels and forms of up to date documentation covering the system and how to build against it.

• Continuous Integration and Continuous Deployments

Frequent releases should be the norm, which almost requires a high degree of automation for modern technology stacks due to the distributed nature of most newly built systems. Thankfully, CI/CD is now fairly common practice, with most technology service providers offering a range of tools, frameworks and integrations for your build technology and deployment methodology of choice.

• A robust peer review process

People often overstate the purpose of pull requests by focusing on approving, or declining and critiquing code changes and sure, that’s certainly a large part of the deal. But there’s also something about general team awareness of how the code is being developed and a familiarity in reading how the code is being constructed that even a passive contribution to a PR helps build and maintain across the team.

• A robust QA process

We come fully equipped with a test driven development process, automated testing, dedicated testing environments, user acceptance testing sessions with stakeholders and still we manage to release software with bugs. There is no right or wrong amount of testing, and it all depends on the specifics of your org, project and team but a QA process as a whole is basically a non-negotiable for most modern tech companies.

• Monitoring and Debugging tools

A common yet often fatal flaw I’ve observed in software teams is overly siloed team functions with a hard handover or “throw it over the fence” approach to delivery. Most commonly, you see this in the form of a development team producing code for a product feature to then have this sent to a testing team which hopefully passes some QA bar to then finally be deployed by a release team. What this often turns into is a complete lack of wider ownership and teams narrowly focusing on their individual functions causing issues to fall by the cracks that software bugs often love to creep into.

One of the ways to combat this is to allow system health and system monitoring to be a team wide activity, hopefully detached from hard deliverables as to allow this to be something any engineer is empowered to engage in.

• Integrations with task management and tracking tools

Save some of the video call exhaustion by allowing your stakeholders, project managers and anyone else who might be interested, to easily understand the status of the project and how it’s most likely to pan out going forward - instead of having another meeting.

• Unopinionated code formatting tools

Another time saver! This is isn’t so much about sticking to the “right” way to write and structure code but more about leveraging standardisation to allow developers to rapidly parse code that has a familiar structure.

Putting it all together using automation within GitHub’s toolset

GitHub have spent the last few years expanding their developer tools to nicely accommodate more and more of the software development process. This feels like a sensible direction given that so much of the developer workflow is entrenched in version control and the code itself which naturally sits quite nicely in GitHub’s favour!

Let’s breakdown a few of the key features to demonstrate how they can empower a realistic use case when assembled together with a few other powerful tools.

Our example use case: Building a web application with our 🔥 workflow

Let’s take the common example of a business who needs to create a simple user data capture form in the style of a single page web application.

A fictional company (Koo2 ©) needs to build an application, which their users can access and use to submit their details. This web application will present the user with a series of questions, to then capture the user’s submitted answers and then finally securely store this user’s information in some fashion for later usage.

Our fake tech company obviously needs branding

• Breaking down and detailing the work via issues and using labels to keep things tidy 🕋

Issues are a great way to define and categorise granular tasks. Project managers and/or engineers can easily create formatted documents using markdown, categorise and label tasks with things like priority and complexity, assign ownership, raise discussion points and tie these tasks back to a wider set of deliverables via milestones and projects.

Labels are also great for detailed search, as you can essentially filter down task boards and progress trackers by any category that you choose to define.

• Organising the Project using GitHub’s automation tools to allow tasks to track themselves 🚀

This is where things start to get a bit more interesting. There are two key features about GitHub projects in particular that I really like.

1. Linking repositories to projects means they automatically get added to your progress board.
2. You can use project settings to allow these to automagically update the status of a task upon key events like pull requests being approved or branches being merged.

• Using Milestones to structure progress 📈

Milestones are a great way to “bundle” related tasks together to form broader deliverables and to track the progression rate of high level features.

Simply link the repository to the milestone and GitHub updates the status of the task in the milestone for you. 🤖

This gives us a really solid template to work from as the development team can now place their core focus on technical delivery by allowing their tools to facilitate project management and coordination with other team members.

• Using GitHub Actions to automate the build and release process 📦

In order to fulfil our ambitions of creating a complete developer workflow, we’ll need to bring in a few more tools and tricks to round out our build process. We’ll then use GitHub actions to string this all together.

Workflow components

1. Conventional commits to formally version code changes. This is a really useful tool which facilitates the use of predefined commit message conventions in order to better communicate expectations to potential contributing team members.
2. Jest to run unit tests
3. QAWolf to run automated UI tests
4. Submitting changes for peer review
5. Chromatic and Storybook to help the team review and visualise in-progress visual changes
6. Lighthouse in order to audit the app for performance, accessibility, progressive web app friendliness, and SEO suitability
7. ZAP Scan to ensure the app is keeping in line with OWASP
8. Release 🎉

Putting our workflow into action

Let’s start with the task of adding a telephone number question input to our Koo2 app.

Most development teams use some variation of a branch based workflow to manage multiple and frequent changes to the code base. A great way to ensure that you’re getting the most out of this is to use a tool like commitizen to help us structure each commit in this way without having to think too hard about it.

Once the code required to service our new telephone question screen has been added and corresponding unit or component tests have been written, the branch can be pushed up to the build server where we let our GitHub actions automatically kick in. Here’s exactly what’ll happen in sequence once we submit our pull request for peer review.

1. The project board automatically picks up the change and updates itself with the task status

2. Unit tests and automated tests are ran to check everything works as expected

The build server installs the dependencies before running the build and test commands

3. Our UI components are rendered in Storybook to allow the team to visually review the changes and easily inspect what’s new at an interface level

Storybook provides a nice interface for inspecting your UI components and how they’ve changed

4. A ZAP scan is executed to ensure no major security vulnerabilities are compromising our system

The output as produced on the build server

5. A lighthouse report is produced to ensure that our app is as performant and as accessible as possible. You also get a nice diff of how these latest changes impact the app’s lighthouse metrics.

It’s completely up to your team what the bar is but this should give you a rough flavour of what a set of criteria for a good workflow could look like.

The final piece of our workflow is to allow the team to review and approve/decline the changes and for the pull request to be merged. Once this is done, the project board will once again automatically update the status of this task as completed and slot it in the right column. 🏁

All of this — automated via GitHub action scripts and commonly provided jobs to allow you to quickly compose the right workflow for your project! 🙏

Here’s a (simplified) sample of what these actions can actually look like at a code level.

name: Run Security Scan
on: push
jobs:
  zap_scan:
    runs-on: ubuntu-latest
    name: Scan the webapplication
    steps:
      - name: ZAP Scan
      uses: zaproxy/action-full-scan@v0.2.0
      with:
      target: "https://uri.to.your.app/"
name: Deploy Story Book
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
       - uses: actions/checkout@v1
       - run: npm install
       - uses: chromaui/action@v1
name: Test and Build
on: push
jobs:
  test:
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@v2
     - run: npm install && npm install -g @lhci/cli@0.5.x
     - name: Create production build
     run: npm run build
     - name: Run lighthouse audit
     run: lhci autorun