Re-evaluating process and workflow in software development
You work really hard at releasing a product and spend months executing a well thought out plan. As deployment progresses, a number of critical bugs start to emerge. The customer is screaming at you because some of the use cases seem completely different from his/her expectations. It is an all hands on deck situation.
Once the dust settles, you call for a post-mortem. All the issues trace back to either a lack of process (in requirements gathering, scoping, development and quality control) or non-adherence to the process if one did exist. A blame game ensues on why no process was followed. You eventually decide to double down on filling process gaps and enforcing adherence to it.
Does this sound familiar? Have you ever felt like no matter how much enforcement you attempt, you never get the compliance or outcomes you expect?
Maybe it’s time to acknowledge that the root of the issue lies elsewhere and not in the process
The Genesis of processes and workflows
The industrial revolution and assembly lines brought structure to manufacturing. The desire to constantly improve both the quantity and quality of the end product gave rise to the study and implementation of processes and workflows. Standardisation and repeatability of tasks became the foundation of manufacturing processes. As large volumes started flowing through these workflows, line managers were now, for the first time, able to look at inventory and throughput at each station to assess bottlenecks in the manufacturing process. Standardisation of tasks also allowed for the setup of quality control stations all along the assembly line. If the end product didn’t turn out as expected, it was easy to go through the workflow and attribute the defect to a specific process and station. The process and/or workflow would get tweaked to eliminate the defect or inefficiency.
Process in software development
As the demand for software started growing exponentially, developers started borrowing tools and techniques from the manufacturing industry in the hope that it would bring discipline and predictability to software development. At first glance, the issue seems to be that of indisciplined developers — they never seem to follow any process. The real problem, unfortunately, is not with the process, but the foundations on which we developed them
Processes and workflows rely on the following to be true:
- Standardisation of tasks — the end product is the output of a sequence of standardized and well-defined tasks.
- The specialization of tasks — Each person performs only one kind of standardized task
- The output of each standardized task is the same
- Volume — Workflows need large volumes to flow through it to be effective and efficient.
- Workflows move in a single direction.
Unfortunately, in software development, none of these are true.
Most software teams vary between 5 and 15 developers and their growth beyond that number is constrained by communication and management overhead. Each developer performs a variety of different tasks in a given day. These vary from design to bug fixing to feature development and even testing. There is absolutely no standardization of tasks and therefore no specialization. Each feature or bug fix is treated uniquely because no two bugs or features are the same and therefore neither are the steps involved in reproducing, fixing or testing them.
For small teams, there really isn’t any real volume to identify and evolve common patterns that could be used to establish a true industrial process.
The implication of having small teams is that you have limited resources. These teams are therefore more focused on solving problems as they come along. Processes and workflows are shackles that are counterproductive and inhibit this kind of creative problem-solving process.
The Overhead of Process
Let’s imagine that you were a developer on a small team of 5 that has adopted the Scrum methodology for a new software development project. Jira seems like a good choice to make for process and workflow management. The most basic workflow available in the Atlassian marketplace for bug fixes looks like this:
You adopt it and get started with your backlog of bug fixes. Consider the following scenarios:
Scenario 1
A critical issue lands on your plate and you see that the customer has requested a staged environment to validate the fix internally before rolling it out to their production environment (their customers). You have no state to reflect or satisfy that request.
Scenario 2
While you were in the “In QA” state with a particular issue, the customer comes to you and tells you that you missed an edge case in the review and need to re-review the fix. The workflow doesn’t let you change state back to the review state. You don’t want to move the issue to a failed state because it might make your delivery metrics look bad. The customer won’t let you move the issue to the deploy or done state because they don’t want to pay you till you fix the issue.
In each of the above scenarios, you have two options. 1) You can spend time working with the project manager to redesign the workflow to incorporate potential new states and paths for a staging environment deployment. This is a non-trivial activity and can take days if not weeks to implement and test before deploying as part of the normal flow. 2) Alternatively, you can ignore the workflow and just deliver what the customer needs, which would be a case of non-adherence to a defined workflow.
For a small team, the time and expertise expended in adhering to a rigid workflow for a bug fix are significant and one that they can ill afford, especially given that the adherence to the process doesn’t deliver any business value to the customer. Secondly, a bug fix is only one of many kinds of activity or flows that a developer needs to adhere to.
The balanced approach
If you are a small team, adopt a process that plays to your strengths and overcomes your limitations. As a small team, your strengths are:
- Low latency communication
- High degree of context distribution
Once you recognize that you don’t have specialized expertise in non-core (non-engineering) functions and tools like project management, HR management, communication systems etc., you want to pick something simple, but custom built for the job.
Slack for example does a great job on the communication front. Channels offer a simple structure with very low cognitive overhead. A powerful search and a plethora of add-ons give your teams all the tools they need to focus on their core value proposition and strengths.
Similarly, when it comes to case management, Manuscript’s custom-built structure for software development gets out of your way and lets you focus on being productive with your value generation. Manuscript has some nifty simplifications that just make sense:
- Every case has one assignee. While that decision seems like a simple one, it is incredibly insightful. Having a single assignee avoids finger pointing or worse, the “It’s not my problem!!!” syndrome.
- The simple states in Manuscript are tried and tested for software development and you will probably want to adopt them as is. They are custom designed to be the right balance between effectiveness and structure.
- No workflows and just state — tracking where you are with your cases, is important. Strict workflows though are not the answer.
Will I ever need a workflow?
The answer is “Yes”. Process and workflows are absolute musts in the right context.
- If you have lots of repeatable tasks, you will want to leverage process.
- If you have a huge volume of tasks (tens of thousands), then you want to work towards a process so that you can break down the task into smaller standard structured operations.
Once you have used the states and assignment with thousands of tasks/cases, you have enough data and insight to see patterns and design a good workflow. Workflows are one of the best tools in that scenario. Until then, a simple structure is your best friend.
Conclusions
- If you are a small team, you want to keep it simple and not add the overhead of process and workflows.
- Use simple structures to organize and manage your tasks.
- While a tool like Jira can offer simple structures, it is overkill — It was designed to be a workflow engine for everyone. It would be akin to using a cannon to swat a fly. Tools like manuscript offer well-tested structures tuned to the software development domain.
