How the development workflow affects task decomposition
One of the most important factors affecting the pace of the development and the success of a project launch is proper decomposition of the product manager’s idea into tasks for direct programming. What’s the right way of going about it? Do we just take the script of the new feature from the product manager and get right down to coding? Or do we first write acceptance tests and then the code that will ensure their successful execution? Or maybe we transfer it all to the developers’ plate and let them shoulder all the decisions over the course of the Scrum-poker?
Let’s think this through and identify the problems that may arise in the process of distribution of tasks and ways to solve them. In this post, we will examine the basic principles of task decomposition when working in a team. My name is Ilya Ageev, and I’m the head of QA at Badoo. Today I’ll tell you how workflow affects decomposition, how testing of tasks differs from laying out of tasks arising as a result of the decomposition, and what rules should be followed to ensure that the development process goes smoothly for everyone involved.
Why is it important?
We should bear in mind that the development process is more than just a code writing session. When we talk about development, I urge you to consider the whole process, starting from the setting of a task and ending with the stable operation of the feature in the hands of our users. If we fail to take into account all the steps that come before and after coding, it’s very easy to get into a situation where everyone is busy doing something, executing their KPIs and receiving bonuses, and yet, the outcome is decidedly poor. The business is going down the tubes, increasingly smothered by the competition, but at the same time, individually, everyone is doing a great job.
Why does this happen? It’s quite simple: it’s basic human nature to approach situations from the point of view of one’s own comfort. The developer is not always inclined to think about what will happen to the code after it’s been written. He/she has executed the task — and that’s all he/she cares about. There is so much uncertainty in human relations. Many developers feel much more in their own element sitting at the computer, focused on solving their own interesting task — blockchains with neural networks — and they are reluctant to be distracted by having to think about product managers, deadlines, and users who will eventually be utilising their ingenious creation (and maybe even daring to criticise it!)
This is neither good nor bad: we value developers precisely for their ability to tackle technical problems in a thoughtful and competent manner. But too narrow an angle of view in approaching a problem often brings the development process to a halt. This applies not only to the development of specific individuals, but of the company as a whole. After all, the growth of the company and the improvement of its corporate culture hinge on the growth of each of its employees. Therefore, it is vital for us to venture out of our bubble now and then, forcing ourselves to get a broader perspective on the problems, in order to stimulate this growth.
And, of course, if such an important stage of the process as decomposition is entrusted to a person who regards everything solely from the standpoint of his/her own convenience, there is a serious risk of running into a heap of problems at subsequent stages: when the fruit of his/her labour is merged with that of others, as well as during code review, testing, laying out into production, and so on.
Thus, when determining for yourself how to properly break down this or that task, figuring out where to start and what to arrive at in the end, it’s important to consider as many factors as possible, and not look at the problem only from one’s own selfish perspective. Sometimes, in order for everything to work faster and more efficiently in the next stages, you may need to handle things in a more time-consuming and labour-intensive way at the stage for which you are responsible.
The writing of unit tests serves as a good example of this. Why should I waste my precious time writing tests if we have testers who will test everything afterwards? This is because unit tests are needed not only to facilitate the coding process — they are also required at subsequent stages. And they are truly indispensable: they speed up the process of integration and regression tenfold and beyond; they lie at the foundation of the automation pyramid. And this is without even taking into account the acceleration of your own work: after all, having “felt around” the code in this or that area of it, you yourself need to make sure that you haven’t inadvertently broken anything. And one of the fastest ways of doing this is by running unit tests.
Workflow
In order to somehow formalise the relationship between the participants in the process, many teams agree on the rules of working together as a team: they agree on the coding standards and the overall workflow in the version control system, and setting the release schedule, and so on.
Needless to say, if you initially agree on the process without taking into account the entire life cycle of the feature, you may subsequently run into slowdowns and the proverbial “rakes” — especially taking into account the growth of the project and the company. Even though one should be mindful of premature optimization, if there’s a process that works well on different scales, why not use it right off the bat?
Speaking of workflow development, many who use Git immediately bring up some kind of “standard git-flow”, maintaining that it’s correct and flawless and often implementing it in their work. Even at conferences where I gave presentations on workflow at Badoo I’ve been asked more than once: “Why did you choose to invent your own? Why not use a standard git-flow?” Let’s consider this question in more depth.
Firstly, when talking about this workflow, they are referring to something along the lines of this diagram. I took it from Vincent Driessen’s article “A successful Git branching model”, which describes a scheme that worked quite well on several of his projects (way back in 2010).
Today, a few major players in the code hosting market generally offer their own flow, while criticising the “standard git-flow” and pointing out its shortcomings; as an alternative, they promote their own schemes, recommendations, and techniques.
If you do a search on git-scm.com (or, better yet, just google it), you may be surprised to find out that there is no such thing as a recommended (let alone “standard”) workflow. This is because Git is, essentially, a framework for a code version storehouse, and how you organize this storehouse and collaboration is entirely up to you. You should bear in mind that just because a certain flow really “took off” on some projects, this doesn’t at all mean that it will also perfectly suit your purposes on other projects.
Secondly, even within our company, different teams have different flows. Distinctly different workflows are used for the development of server code in PHP, daemons in C/C ++ and Go, and mobile teams. And we didn’t arrive at these solutions all at once: we tried out a range of options before settling on something concrete. By the way, these teams not only use different workflows — they also use different testing and task setting methods, as well as setting their own releases and using their own distinct delivery principle: that which is delivered to your personal servers and end-user’s computers (smartphones) by definition can’t be developed in an identical fashion.
Thirdly, even an officially adopted workflow constitutes more of a recommendation than an uncontested rule. The tasks handled by the business may vary, and it’s good if you managed to choose a process covering 95% of the cases. If your adopted workflow, however, doesn’t fit the task at hand, then you should look at the situation from a pragmatic point of view: if the established rules get in the way of efficiency,
then to hell with such rules! But be sure to check with your manager before making a final decision, just in case you accidentally cause a mess. You may overlook some important aspects readily apparent to your supervisor. On the other hand, everything might end up working like clockwork — and you might be able to modify the existing rules in a way that evolves the internal processes and triggers growth across the board.
If everything is so complicated, and the workflow is not a dogma set in stone, but rather a recommendation, then why not use one branch for everything: Master for Git or Trunk for SVN? Why complicate matters?
Looking at a problem from one side only, this approach with one branch may seem quite sensible. Why deal with the headache of different branches and worry about stabilising the code in each one of them when you can just write a code, push it to a shared storehouse — and make your life easy? Indeed, if there’s only a handful of people working together as a team, this can be convenient, as it eliminates the need to merge branches and organise them for the release. However, this approach has one very significant shortcoming: code in a shared storehouse may prove unstable. Vasya, who is working on Task no. 1, may easily break the code of other tasks in the shared storehouse by inputting his changes. And until he corrects them/rolls them back, the code can’t be posted even if all other tasks are all set and working properly.
Of course, you can use tags in the version control system as well as code freeze, but it’s obvious that the approach involving tags does not differ much from the approach involving branches, and that is because it complicates the initially simple scheme. And code freeze, especially, slows down the process even more, forcing all the participants to stop the development until things are stabilised and the release is posted.
So the first rule of good task decomposition goes like this: you need to break down the tasks in such a way that they are merged into the shared storehouse in the form of logically completed pieces that work by themselves and do not upset the logic around them.
Feature branches
We have a great variety of workflow options in our company, but they all share one common trait: they are all based on individual branches designated for features. This model allows us to work independently at various stages, and develop various features without interfering with each other. Besides, we can test them and dump them into the shared storehouse only after making sure that they work and don’t break anything.
But this approach also has its drawbacks, which stem from the very nature of feature branches. In the end, after isolation, the result of your work will need to be merged into the common space shared by everyone. At this stage, you may run into a host of problems, ranging from merging conflicts to overly lengthy testing/bug-fixing procedures. After all, by resorting to your own separate code branch, you isolate not only the shared storehouse from your changes, but also your own code from changes made by other developers. As a result, when it’s time to merge your task into the common code, even after it’s been checked out and proved to be working, major headaches may arise, because Vasya and Petya affected the same lines of code in the same files in their respective branches — resulting in a conflict.
Modern systems of code version storehouse come equipped with a bunch of convenient tools, merging strategies, and so on. Nevertheless, avoiding conflicts altogether is simply not a possibility. And the more changes there are and the more intricate they are, the more difficult and time-consuming it is to resolve such conflicts.
More dangerous yet are conflicts associated with code logic, when SCM merges the code without a hitch (because there are no conflicts in the files), but due to the isolation of the development some common methods and functions in the code have altered their behaviour or have even been removed from the code. In compiled languages, the problem, as it may seem, is less acute: the compiler validates the code. Yet, the situation where the method signatures have not changed, but the logic has may still come up. Such problems are difficult to detect, and they deter the successful release even further and necessitate
testing and re-testing the code many times after each merger. And when there are a lot of developers, a lot of code, lots of files, and lots of conflicts, the situation devolves into an infernal mess, because while we were fixing the code and rechecking it, the main version of the code has progressed far ahead, and everything must be repeated. Do you still not see the need for unit tests?
To avoid this, many try to merge the results of common work into their branch as frequently as possible. But in cases where the feature branch is large enough, even adhering to this practice will not ward off problems, no matter how hard we try. This is because even though you integrate other people’s changes into your code, nobody sees your changes. Therefore, you need to not only frequently merge other developers’ code into your own branch, but also merge your code into the shared storehouse.
Hence the second rule of good decomposition: feature branches should contain as few changes as possible, so as to be integrated into the common code as promptly as possible.
Parallel operation
Okay, but then how can I work in separate branches if several programmers are working on the same task broken up into parts? Or if they need changes made to parts of the code shared by various tasks? Say, both Petya and Vasya are using a common method that is supposed to operate according to one script within the framework of Petya’s task and in a different way under Vasya’s task. What should they do?
In this kind of situation, a lot depends on your release cycle, because it’s the instance of the task being laid out into production that is considered to be the moment of completion of that task. After all, only at that point can we know for sure that the code is stable and working properly — that is, if you didn’t have to roll back the changes from production, of course.
In cases of rapid release cycles (where you post things on your servers several times a day), it’s quite feasible to make the features dependent on each other in stages
of readiness. In the above example with Petya and Vasya, we would create not two tasks, but three. The first one would be as follows: “We change the general method so that it works in the two variants” (or we implement a new method to accommodate Petya), and the other two tasks are those of Vasya and Petya, respectively, who can start work after the first task is completed, without crossing paths and interfering with each other.
If the release cycle does not allow for frequent layout, the above example would become prohibitively expensive, because in that case Vasya and Petya would have to wait days and weeks (and in some development cycles, years) until they can get to work on their respective tasks.
In this case, you can use an intermediate branch shared by several developers, but not yet stable enough to be laid out for production (Master or Trunk). In our flow for mobile applications, such a branch is called “Dev”; in the Vincent Driessen scheme it is called “develop”.
It’s important to bear in mind that any change in the code — even mutual merging of branches, merging of common branches into a stable Master, etc. — must be tested without fail (remember those code and logic conflicts, right?) Therefore, if you’ve come to the conclusion that you need a common code branch, then you need to be ready for another stage of testing: after the merger, you need to test how the feature integrates with the other code, even if it has already been tested in a separate branch.
Here, you may notice that you can test just once — after the merger. So why test before it, in a separate branch? It’s true, you can. However, if the task in the branch doesn’t work or breaks the logic, this abortive code will land in the general storehouse and not only interfere with your colleagues’ work on their respective tasks, breaking some sections of the product, but it can plant a veritable time bomb if someone decides to base a new logic on the faulty changes. And in cases where there are dozens of such tasks, looking for the source of the problem and fixing the bugs is very difficult.
It is also important to understand that even if we use an intermediate development branch of code that may not be the most stable, the tasks or their pieces in that branch should be more or less complete. After all, at some point, we need to release it. And if the feature code in this branch keeps breaking, then we can’t lay it out — our product won’t work. Therefore, having tested the integration of features, we should correct the bugs as soon as possible. Otherwise, we’ll wind up in a situation not unlike the one where one branch is used for everyone.
This allows us to formulate the third rule of good decomposition: tasks should be divided so that they can be developed and released parallel to each other.
Feature flags
But what should we do when there’s a major new change in business logic? Programming this task alone can take several days (weeks, months). We won’t be dumping unfinished feature pieces into the common storehouse, will we?
As a matter of fact, yes, we will! And there’s nothing to worry about. The approach that can be used in this situation is feature flags. It is based on the introduction of “switches” (or “flags”) into the code that enable/disable the behaviour of a particular feature. By the way, the approach does not depend on your branching model and can be used in any and all possible models.
A good example of a simple and understandable equivalent to this is an item in the menu for a new page in an app. While the new page is being developed in pieces, the item is not added to the menu. But as soon as we have finished and assembled the whole thing, we add the menu item. The same goes for the feature flag: we wrap the new logic with the condition of the flag being enabled and change the behaviour of the code depending on this condition.
In this case, the last task in the process of developing a big new feature will be the task of “enabling the feature flag” (or “adding a menu item” in the example with a new page).
The only thing to keep in mind when using feature flags is an increase in the time spent testing the feature. This is because the product needs to be tested twice: with the feature flag on and off. It’s possible to save money here, but you should tread carefully: for example, testing only the state of the flag that is laid out to the user. Then, in the development of the task (and laying it out piecemeal), it will not be tested at all; instead, it will be tested only during the last task of “enabling the feature flag”. Here, however, one should be prepared for the eventuality that the integration of feature pieces after the flag is enabled can turn up problems: bugs overlooked in the early stages may be detected at this stage, in which case finding the source of the problem and eliminating errors may prove quite costly.
Conclusion
To sum it up, then, when decomposing tasks, it’s important to bear in mind these three simple rules:
1. Tasks should be in the form of logically complete pieces of code.
2. These pieces of code should be small and should be merged with the common code as promptly as possible.
3. These pieces should be developed in parallel with each other and released independently of each other.
It hardly gets any simpler. By the way, in my opinion, the most important criterion is independent release, and everything else follows on from that, in one way or another.
I wish all of you best of luck in developing new features!
Ilya Ageev, Head of QA
