Deadlines (in software development) do not give you any predictability

Deadlines do not give you any predictability over your software delivery

deadline (n):
a date or time when something must be finished.

Time and time again I end up in a discussion (sometimes an endless debate) with people who believe that deadline will give them predictability when they can get the software in their hand. I want to use this opportunity to spill out what is in my head about the illusion that people have around deadlines. I am specifically focusing in the context of software development. What people have in their head about deadlines is they can get a specified amount of scope by the specified date. In software industry many people still live under the false belief that they can get all of the scope they want on the date they want — as if software development is like a construction project. That may be true for simple software projects but rarely the case for complex projects. Some people working in the software industry also still believe that to make the software delivery to be more predictable is to put tighter control in the project.

By imposing deadline you may get a promise that the software can be delivered on the specified date. But a promise may not necessarily be delivered. Not because people lie about their promise or because people are not committed with their promise but often times unexpected things that is beyond the promiser’s (the person that made a promise) control can happen in the future.

In the context of software development, many software developers until today make promises based on fear rather than empirical data. Because of this a promise does not give us predictability and it only give us a hope. A promise that is made based on fear can lead us to false sense of security. Predictability does not come from hope but from regularity. Regularity and consistency remove anxiety and build trust with the stakeholders.

Okay up to this point you may not believe me that a promise does not give you any predictability. You may think I am just a crazy guy with a bunch of nonsense. Okay let’s take an example of the airline flight schedule. Airline flight schedule is one example where I often get a false sense of security. I’ve flewn on some of the best airlines in the world but even the best have the worst delay. These days I have learnt to cope with flight delays so that I don’t end up with dissapointment.

On my boarding pass there is a deadline written clearly when I should reach the gate to go on board and when the plane is expected to depart. However eventhough as passengers we try our best to be on time at the airport to check-in and be at the gate to go on board to the plane, it is still possible for a flight schedule to be delayed. Why is that? That is unfair isn’t it? Didn’t the airline made a promise to passengers when the plane is going to depart? And this is clearly written on the boarding pass. Why is it still possible for a flight schedule to be unpredictable even though there’s already a deadline? That is because in the airline industry, passenger’s safety is more important than meeting deadlines that is why deadlines does not give you predictability.

In the context of software development, imposing deadline to software development team will also give us a false sense of security — we think we will get everything we want on time but in reality we actually get less than we actually want. Even though software developers will promise that the specified scope can be delivered by the date that has been set by the stakeholders, they will find a way to cut corners under the hood without the business people knowing just so that they can meet the deadline. When software developers deliberately create technical debt it can be bad for the business in the long run as the software will be flaky and hard to maintain hence the business will not be sustainable. The software will be like a time-bomb that is waiting to explode to make the company go bankrupt. In the long run, delivery of new features on the existing code base will be more and more unpredictable because of the technical debt that lives inside the software.

Predictability does not come from imposing a deadline to the team. So if deadline does not give us any predictability, in the context of software development where does predictability come from? Don’t we all want predictability so that we can manage our business better? Let’s see what actually creates predictability in software delivery.

1. Professional Team

So what is a professional team? How do we differentiate a professional team to an unprofessional team? Why does professionalism give us predictability?

professional (adj):
characterised by or conforming to the technical or ethical standards of a profession

professionalism (n):
the conduct, aims, or qualities that characterize or mark a profession or a professional person.

Professionalism in software development is about developing the software right according to the standards — not cutting corners even under pressure, it is about the courage to tell the truth that might harm the customer if the software is still delivered on the date that has been set. Unprofessionalism is the main reason why software delivery is unpredictable. A professional pilot do not fly a plane when the weather does not permit them to fly the plane. A professional doctor do not cut corners just because the patients is putting them to the corner. A professional software developer do not create technical debt just because their boss told them they need to meet the deadline.

But what standards software developers need to comply to? Which institution defines the professional software development standards? What authority does this institution has over software developers that do misconducts during software development? Answering this question is quite challenging because software development is a very unique industry, at least until this article is written there aren’t any professional body that standardise the profession of software development like in the accounting, law or medical industry. There aren’t any institution that will ban or blacklist software developers from the profession if they do not create any test harnesses, do not create automated deployment script and do not write clean code. No institution out there that will sue software developers for shipping undone software to production.

Agility without professionalism is going to end up in chaos.

You don’t need to be a rockstar developer to be a professional software developer, you just need the willingness to develop software right and to continuously learn professional software development techniques & practices. These software developers should also be working in a safe-to-f
ail work environment. A rockstar development team who is not professional may be fast in the beginning but will be slow in the future because they will be fighting against technical debts in the future. A professional team will be predictable both in software development but also software delivery.

Our role as managers:

• Create a safe environment where people can be professionals without fear.
• Create a path and coach the development team to be professionals.
• Create a community of software professionals in the company where people coach each other on how to be software development professionals.

Also read: Who is the Professional Scrum Developer.

2. Stable Environment

What does stable environment means? Let’s take an example of bus and train arrival schedule. Which one is more predictable? In general, especially in developed cities, train arrival is more predictable than bus arrival. I was quite amazed when I visited Hong Kong because the train can arrive consistently every 45 seconds during peak hours without any delay and accidents. Now let’s look at the environment where bus is operating. Why is bus arrival considered less predictable than train arrival? That is because the environment where bus is operating is unstable, because a bus has to deal with traffic jams, other drivers, pedestrians, bicycles, accidents, speed limit, road works and red lights.

If the team is using an Agile framework like Scrum or eXtreme Programming (XP), stable means there aren’t any interruptions or “urgent” work from stakeholders during the Sprint or the iteration. Yes I do agree that in reality interruptions are very difficult to avoid at workplace, even if there aren’t any “urgent” requests from the stakeholders we still have to answer phone calls, attend meetings and assist other team mates with their technical issues. But we are currently discussing what makes software delivery predictable right? If the Sprint or iteration is unstable it is hard to expect delivery predictability from the team. So is it still possible to get predictability in an unstable environment because this is the current reality most people have to live with? Yes you can, only if the Sprint is consistently interrupted at predictable time and duration. In this case the environment is predictably unstable. A scheme to show you the differences is as followings:

Work Environment Stability

But what if the interruptions does not come from “urgent” work but from production defects? Does this happen often? If it does happen often, is it because the team is not professional? If yes, then go back to the previous point, coach the team to be professional software developers. If the interruptions are coming from the stakeholders who needs “urgent” requests then stabilise the work environment or make the “urgent” request come at predictable time.

Our role as managers:

Coach the stakeholders about the importance on having stable environment and not using political power to push “urgent” requests irregularly.

Crazy idea: Wouldn’t it be awesome if there is a dashboard that will alert the stakeholders when the software delivery will be unpredictable everytime software developers is interrupted just like a dashboard at train station that informs the passengers if the train line is in good service?

3. Regularity in delivery

I’ve been saying to people that making a promise does not create predictability only regularity in software delivery does. The same goes with estimates does not create any predictability, only regular delivery in short cycle does.

I always like to compare the train arrival in Hong Kong with the train arrival where I currently live. Where I live, trains arrive every 10 minutes that is why it is very important to look at the time schedule because missing the train and waiting for 10 minutes is not comfortable (and sometimes it can make you go crazy). Whenever I go to Hong Kong I never need to look at the time schedule because I know what whenever I miss a train another train will arrive before I blink my eyes. From this example you can say that the train system in Hong Kong is more predictable than the train system where I live. We do not need to give the trains any deadline, it will just arrive consistently every 45 seconds (during peak hours)!

In software development, teams who do not have any cadence are very unpredictable while teams who have a cadence are predictable. Teams who sometimes deliver in 1-month, sometimes in 3-month are very unpredictable because they do not have any cadence. Teams who regularly delivers production ready software every 3-months are predictably slow. If the team have a shorter cadence and delivers regularly we do not need to wait long and be anxious about the software delivery. Irregularity in delivery creates unpredictability while regularity in delivery gives us sense of security and it also builds trust with the customers. In that kind of team, you may also find that deadline is irrelevant because before you even think about a new feature the team already got something delivered to you. In software development it is regular delivery in short cadence that creates predictability NOT deadlines.

Agile teams who are using Scrum or XP are predictable when they regularly deliver production ready software at least once every Sprint or every iteration. For Scrum teams, it is expected that they will deliver a production ready software at least once in 30 days or less. If the Scrum team is using a 2-weeks Sprint we know that at least every 2-weeks there is something functional that can be pushed to production environment. For XP team that is using single piece flow, we can be very sure there is a production ready software every week. If the team can not deliver production ready software every Sprint or every XP iteration then that Agile team is considered unpredictable.

If you think that having production ready software every week sounds too impossible (or even ridiculous), these days you will find more and more teams who are pushing software delivery to the limit by shortening their delivery cycle to 1-day and deliver multiple times to production environment every day. For teams who ship production ready software consistently every day, that team is more predictable than teams who ship production ready software every 2-weeks. This is the kind of team that made deadlines and estimates irrelevant in their organisation.

Our role as managers:

• Invest in infrastructure for continuous delivery and release on demand, i.e: Visual Studio Release Management, XL Release, Go Continuous Delivery
• Coach the software development team to increase their capability in continuous delivery techniques.

4. Low variability in the work items

Agile teams who delivers regularly and have backlog items that are in roughly equal size are more predictable than teams who does not slice their backlog items. Teams that does not slice their work into roughly equal size have a high variability backlog items which makes them more unpredictable than teams who have low variability backlog items. Agile teams who slice their backlog items into equal size is predictable because customers or Product Owners can make a forecast based on how many backlog items the development team delivered over time or make a forecast on how long the project can finish.

Agile teams should start looking into backlog items slicing into roughly equal size techniques. There are many slicing heuristics hence there are no silver bullets. Agile teams need to continuously discuss it during retrospectives what are the most suitable slicing heuristics for their context.

Variabilities in Backlog Items

To reach this state, not only it requires a professional team but also collaboration and continuous learning attitude.

Our role as managers:

• Create a safe working environment so that the team feel safe about improving their backlog items slicing heuristics.
• Facilitate retrospectives and use it as an opportunity to reflect on the backlog items the team has delivered from iteration to iteration.
• Coach the team on backlog items slicing as such it has low variability in the future.
• Rather than imposing deadline, apply Little’s Law for probabilistic forecasting based on empirical data: the number of roughly equal sized backlog items that development team deliver from iteration to iteration.

5. Consistent team composition

team (n):
a group of people who work together

Why does consistent team composition throughout the project create predictability? Because a change in team composition means:

• Extra effort for the team to teach the new team members.
• Learning curve for new team member.
• New working and coordination agreement.
• New dynamics in team work.

Which means a change in team composition will increase the unpredictability in the delivery. High turnovers can be one of the reason why the team composition is inconsistent. If turnover is the reason why the organisation can not have consistent team composition, the organisation should find out why team members do not stay long enough in the company.

Non-Agile projects are naturally unpredictable because the whole project members are inconsistent from phases to phases — you can even argue there is no notion of team in a non-Agile projects. Agile teams that does not have consistent team composition from iteration to iteration is also unpredictable.

Our role as managers:

• Create an environment so that people can stay longer in the company and proud being in the company.
• Create an environment where the team composition can be consistent throughout the project.

So what does this mean? Does this mean we as managers can not set deadlines for software developers and they can do whatever they wish? Does it mean as a manager we should let software developers procrastinate? Of course that is not what this article’s is all about. This does not mean we let software developers do whatever they want just because deadlines do not create predictability in software delivery. What this mean is managers should shift their focus, they should be more proactive to increase predictability in software delivery. Managers should shift their focus from forcing the development team to work extra hard to meet the project deadline to:

• increasing the development team capability and professionalism so that the delivery of software will increasingly become predictable over time;
• becoming the servant leader for the development team — improving the fluidity of their work by removing any interruptions out of the way.

Forcing the development team to work extra hard to meet the deadline will end up in overburdenning (無理 muri). Overburdenning will only make the organisation fragile rather than sustainably agile. Overburdenning in software development, as in manufacturing, will end up in low quality product. But unlike in manufacturing, low quality software are not easily noticeable to managers and even to customers because you have to dig through the codes to find out the rot. And because of this reason deadline limits organisation agility. Organisation should shift its focus and start investing in infrastructure for release on demand so that their customers or Product Owners can release the software to production environment anytime they need it with a single click of a button.

All of this sounds like an utter nonsense to you? Don’t believe that the idea of #NoDeadlines would ever work? Or maybe you are afraid to lose control when you don’t impose deadlines to software developers? Shifting mindset is not easy, you may even deny everything that is written here. Don’t worry I quite understand that, I don’t expect you to believe everything that is wrriten here either.

Many companies have already successfully implemented removing deadlines before this article was written. #NoDeadlines requires a leap of faith —challenging and unlearning what we already know — just like when companies start jumping on board with agile software development 20 years ago when it was considered as a laughing stock. Today we are seeing software developers shipping software to production environment several times in a day considered as a normal thing. Where is your organisation at the moment? At what level is your organisation in terms of software delivery at the moment?