Learning Agile in Reverse: My Experience and What It Ought to Have Been

Published in

The Startup

15 min readFeb 27, 2021

I’ve been working in software development since 2015, and have worked with lots of teams, and all of them have gone through phases of what they called “agile” at one point. But the experiences of this process have varied wildly, and I’ve rarely seen teams that would consider themselves hyper-productive, as some agile teams describe. I recently read Learning Agile, and got a whole different worldview from what I had experienced. Here’s how I was introduced, and why a lot of it wasn’t done right.

First experience with agile, scrum, kanban
key takeaways from the book “Learning Agile”
a lot of hate for agile on forums
why teams don’t become hyper productive
main things I’d focus on when trying to implement agile

How are new developers introduced to Agile?

Working in an agile fashion isn’t a requirement for new developers, but it’s often an nice-to-have while interview for mid-senior developers and above. In my experience, the new developers get dropped into a team already working in what’s called a scrum fashion, they’re told to find and organise their work in Jira, and get to it. The rest is monkey see, monkey do — they attend stand ups, sprint planning, and follow what everyone else is doing.

This was the first mistake in my training — the word agile was thrown around a little, but nobody made clear what the agile manifesto was, or why it was important — the focus was simply on working in a way that could be described as scrum by an outsider. I wish I had been sat down at the time and told “these are the agile principles, and we try to base our activities to help us stick to these principles”. In reality, I was told to be at the whiteboard for 9:30 every morning to stand up and demonstrate the progress I’ve made.

The agile manifesto is great. If you haven’t actually read it, do your self a favour, it’s extremely short, less than the first paragraph of this article in fact. But how do you ensure that your team abides by values? That’s a bit lofty. The scrum framework provides a structure to help implement the values of agile, by recommending certain ceremonies, like a time-boxed sprint, a daily stand-up, sprint planning and a sprint review.

In my experience, I’ve seen teams adopt scrum because someone mandated it, and the team performs “scrum” to the letter, but not in the spirit of agile. It’s better than having no process at all, because this results in chaotic development, but it doesn’t allow for hyper-productivity. The teams simply have some analogy of chickens and pigs and some extra meetings to attend, and they get back to work as they know it, with a few extra boxes to tick.

Much like software estimation, doing agile effectively isn’t enormously hard, but it’s not necessarily intuitive, and reading more about this topic can only help your standpoint.

Key takeaways from Learning Agile

You can implement all the practices of scrum and get slightly better results, but this isn’t optimal.

Just because you do stand-ups, sprints, have a product backlog, and a scrum master, doesn’t mean you’re smashing scrum. There are good and bad ways to do all of these practices:

I’m sure you’ve seen plenty of these practices in the wild. For example, have you ever been part of a standup that consisted of everyone taking turns to tell your manager what they did yesterday, what they’ll do today, then disengaging once they’ve provided their status report, yet the meeting still takes 40 minutes? I’ve definitely experienced this before, and it’s not how things should work.

Nobody would do a given practice without a principle behind it, and “someone else is doing it” is a rubbish principle to follow. Effective scrum teams recognise that constructs such as sprint planning and standups exist to facilitate the values of the agile manifesto — we have standups to ensure good communication saturation. We meet regularly with stakeholders to allow us to incorporate change easily.

There are Agile Principles to go with the manifesto

This was news to me — to a developer with hands-on experience, they won’t come as much of a surprise, but a couple of them stood out to me, and they were all worth reading about.

“The most effective communication is face to face” — that’s been difficult during covid, especially if you’re thrown into a meeting with strangers you can’t see. I’ve seen 2 polar opposites of this in the last year, where I’ve worked with people I’ve never seen day-in-day out for months on end, but also spent significant time on teams meetings using vs-code liveshare to write software and debug issues, while building personal relationships with colleagues. This one definitely rings home, and I’ve endeavoured to show my face more lately.

“Working software is the primary measure of success” — if your team has committed to deliver something and it’s not quite finished, it might be embarrassing, but it should be impossible to hide. Customers need to be able to see the software running and delivering value, otherwise you haven’t delivered value.

Effective Scrum teams understand “collective commitment” and are “self organising”

My first thoughts were, “what do you mean by a self organising team?”. This stems from collective commitment — the team, as a whole, agrees to their stakeholders to deliver a set of features within a sprint. Each member feels committed to that goal and responsible for making it happen. Either the whole team succeeds, or the whole team fails.

However, I’ve more often seen sprint planning more along the lines of:

take a look in the backlog
select enough items for the sprint to equate to the approximate team velocity
distribute the stories to developers so that each dev has an approximately equal share, which they’re individually responsible for

Ideally, we’d be presenting what we can commit to the stakeholders, and getting the whole team to agree that “Yes, we can promise to deliver this set of features within the sprint” or not. Rather than immediately assigning all the stories and making devs responsible for them, the entire team is responsible for making sure all the tasks get completed. In this way, the team is able to decide among itself, which stories should be completed in which order, so that they can complete the highest priority items of the sprint. When a developer finishes with one story, they can select the next most appropriate one from the sprint backlog — this is more effective than assigning at the start, because at this later point, you now have more information to use when making decisions.

This concept was a really interesting way of working, that I really liked, and I’m please to say I’ve seen it working well recently. Effective scrum teams use language that shows they function as a unit — “in this sprint, we delivered …”, “we are currently blocked by …”, “our goals for this sprint are …”. Teams that don’t get collective commitment, don’t get scrum.

Kanban is a way of working, not the name of UI to show progress.

Ever used a kanban board at work with swimlanes? They do look nice and give a nice overview of where things are, but I’ve seen this vary wildly from what it actually means to work in a kanban fashion. For example, in several teams, I’ve seen swimlanes called “blocked” — this literally makes it impossible to view where in your process tasks are piling up, if you’re moving them to a different column. How will you gain the relevant info to know how to set Work-In-Progress limits? Often, teams don’t use WIP limits, or mis-use them, I’ve seen teams say “We’ve got 10 people, therefore we can’t have more than 10 tasks in development at the same time”. Sounds reasonable, but it’s got nothing to do with kanban, or optimising processes.

I’ve also seen teams blindly copy a kanban board format from another team and use that for their team — which blatantly goes against the ideas of kanban. Instead, they should model what the team currently definitely does, and model that process as stages in a pipeline of delivering work, to uncover blocking points.

The concepts around constraints and limiting work in progress are not naturally intuitive, but the literature on this topic is really interesting, especially in the context of manufacturing, where kanban found its roots.

Decide as late as possible

Make every important decision for your project when you have the most information about it — the last responsible moment. This was a strange takeaway for me to understand, but it effectively takes the complete opposite stance of waterfall, where everything must be planned up front and documented, and instead says “we value being able to incorporate change easily”, and that’s easiest done when you make the decisions closer to when they’re needed. Practically, this means assigning a dev to a user story right before it needs done, not at the start of a sprint. Make architectural decisions just before they’re to be implemented, not long in advance.

Seven Wastes of Software Development

This provided a more interesting twist than I was expecting, the answer was not just “installing npm modules / compiling my code”. This essentially highlighted that anything that doesn’t deliver value to your users is waste. This includes extra process for status updates and project tracking etc., unfinished work, extra features that weren’t requested, task switching, waiting on reviews and approvals, and late-found defects that could have been found earlier by Test driven development and quality measures.

Xtreme Programming

This could fill a lot of books, but it was nice to read around this topic. I’m a big believer in pair programming, I think working with someone in person is a great way to get more value out of your efforts, putting two heads together seems to create a sum greater than it’s parts. And if you haven’t experienced the totally weird sensation of writing your tests before your code and seeing them pass just as you’ve written enough code while practicing TDD, I strongly recommend taking some time to see how it feels.

A lot of hate online for scrum

I’ve seen a few posts lately on reddit about developers with a passionate hatred for scrum, and I collated their reasons:

Daily standups limit hours you can work and are anxiety-inducing: this one is absolutely a case of following scrum to the letter or spirit. The goal of a standup is to ensure communication saturation among the team, to make sure everyone is aware of the state of their commitment to the sprint goals. However, combining a standard “3 questions” format along with managers who want a status update, some people have described this as a daily re-interviewing for your own job. This is exactly as horrible as it sounds. This is a difficult feeling to break, and requires a cultural change and perhaps some coaching, to ensure this practice is actually evoking the value it was aimed to expose.
Engineers are interchangeable: the idea that you should only ever take the next item in the backlog, regardless of your specialty or domain knowledge. Personally I think this rule ought to be taken with a pinch of salt, devs are obviously not interchangeable, especially in a cross functional team, and different people will have different passions and motivations. I think this could easily be adapted to “take the highest priority task suited to you” in most cases, and being pragmatic about ensure sprint goals are met.
Velocity being used to demand more from developers:

“If the team points 20 for sprint 0, and only completes 10 points at the end of the sprint, the rulebook for Scrum says you clearly overestimated your capacity. But come retro it’s always ‘what happened’ and thinly veiled ‘guys, we have to do better’ ”

There are a lot of valid criticisms of some devs experience with scrum, but I think this rebuttal was some of the best discussion:

Scrum is not the problem here. Agile is not the problem here. Corporate culture is the problem. Leadership is the problem. You think these same managers somehow wouldn’t f**k up a Kanban team?

That has certainly rung true before. I hope that if I’m in the position of running a team one day, I’ll have the experience and empathy to understand how to get this well balanced.

Why teams don’t become hyper-productive

I recently read a great paper about metrics for hyper-productive teams here. There were certainly what I felt were some outlandish claims in here, like teams commonly achieving 300–400% increase in velocity with agile, vs waterfall processes. One stat I didn’t have any difficulty believing, was that over 90% of teams implementing scrum don’t become hyper-productive.

Daily Standup Modifications

If you’ve ever witnessed the whole team justify their existence one by one, you might beenfit from a change of scenery — these rituals can fade over time if everyone isn’t engaged. Instead of asking the 3 questions of what you did yesterday, will do today, and any blockers, you can shift the focus from the individuals on the team, to a discussion around the sprint backlog’s highest priority stories. This keeps the team focused on their collective commitment, rather than the individual focused on completing “their” work — it’s everyones work:

What did we achieve yesterday towards our top priority?
What was our contribution worth in story points?
What is our plan for completing this top priority today?
Is anything stopping us from getting closer to this goal?

Remember, the goal of daily standups are to maximise the communication saturation — both of these methods can achieve that, but this way helps bring out the values of collective commitment from the team, and discussions around how to organise and which tasks to tackle next. This format allows the team to become self-policing, and keep the meeting efficient.

Time-boxing this effort to under 15 minutes is definitely a valuable idea, to only discuss the most important items we’re committed to. Allowing your daily meeting to stretch out to 45 minutes to make sure everyone has a chance to be heard and discuss problems can become an enormous time-sink, and developers can begin to hate this ritual. A longer meeting means context switching before and after, and that’s an hour of your most productive time every morning gone.

Measurable and Comparable Metrics

If you’ve done scrum for a while, you might have heard the adage “but story points are not comparable across teams! You can’t compare one team velocity to another, they’re arbitrary to the team!”. Yeah, that’s totally true, but it doesn’t stop anyone wanting, even needing, to compare teams somehow! But we need new metrics in order to do this, since the intuitive metric is actually a poor idea.

If we know the following:

Velocity = sum of original estimates of all completed work
Work Capacity = sum of all work reported during the sprint, whether finished or not

then we can start to get closer to comparable metrics. Note the velocity includes only original estimates, not new items brought into the sprint midway through, and it only accounts for work that is actually satisfactorily delivered. We can use the following metrics:

team focus = velocity ÷ work capacity. The percentage of work done leading to delivered value.
percentage of adopted work = sum of original estimates for adopted work ÷ original forecast of sprint. How much extra work have you taken into the sprint midway through, because you completed the sprint work early? Did you underestimate the capacity?
percentage of found work = original estimates of found work ÷ original forecast of sprint. How much extra work gets discovered that needs done to complete commitments throughout the sprint? Are you being overly optimistic with your initial estimates and story uncovering?
accuracy of estimation = 1 - (sum of estimation deltas ÷ total forecast). Measuring each tasks effort against its estimate to see if you’re doing well in this area, or if this is the cause of sprints underachieving or overachieving.
Forecast accuracy = sum of original estimates ÷ (original estimated + adopted work + found work). When you forecast, how well is this translating into work done in the sprint? Is the actual work done in the sprint coming from other places, or are you accurately uncovering and sizing the tasks?
Win/Loss record. This one I find really useful and simple. Count each sprint as a win only if a minimum of 80% of original forecast is accepted, and adopted + found work constitutes 20% or less.

These are a lot of metrics, that can freely be compared across teams, without directly comparing story points. They help measure how well the team is functioning in an agile fashion, and these metrics can each help highlight problems with how the team is operating. Aside from that, these metrics provide so much more value than saying “Team A is taking 40 points per sprint but team B is only taking 24”. These metrics can tell you useful things about your teams dynamics.

Forecast Accuracy should be around 80%. If it’s close to 100%, the team might be facing external pressure to deliver work, and is therefore “underestimating work because of fear of reprisal”. If it’s too low, the team might be working with too much chaos being injected to their routine. This metric can help you uncover dysfunctional dynamics in the teams mode of operation.

Another one is Accuracy of Estimation — if you’re estimates are 100% right all the time, then they’re not really estimates are they? You’re bending reality to suit your prior guess, or you’re spending far too much effort working out estimates. If they’re too low — your team may not understand the technology or domain they’re working in. Either way, this metric being way out can indicate underlying problems that stop your team running optimally.

Actually recording and looking at the metrics

One of the most neglected aspects of working in a scrum fashion is principle #12: regularly reflect on how things are going and tune things to become more effective. Without enforcement, it’s easy to skip retrospective meetings, as the team is “too busy” or working under tight deadlines. But in order to reach hyper performance, you need a good awareness of all the things stopping you.

Professional athletes or musicians monitor their performance, reflect on their weak points, and improve them — the same can be done with a software team, if you record data about the different dynamics you’re facing, actually look at your team’s trends, and implement changes to improve the issues discovered.

Main things I’d focus on when trying to implement agile

There are obviously a lot of components and metrics that going into managing a high performance team.

Empathy and Human Connection

One of the things I think would help me do this well in future, is experience of being in the role of developer and witnessing a range of team dynamics, and empathy for the team — software development is a human activity, not robotic, and building good relationships and working well together is far more important than squeezing every ounce from a developer.

Two points come to mind to nicely summise this:

You can buy a person’s time and effort with money, but you can’t buy their passion or ingenuity — those are only given willingly if you have a person come to think their work is valuable and important and they’ll be recognised for their achievements
The difference between hyper performing agile teams vs. traditional waterfall teams can be 2000:1, but the difference between a the best and worst developers on a team is just 10:1. More than 2 orders of magnitude greater improvement can be made, by working well together, than by turning everyone into rockstar developers. That means good communication, good empathy, and passion for our craft.

Collecting data and reading widely

Being an individual contributor, it can sometimes feel easy to spot what’s going wrong, because you’re forced to deal with it. It’s easy to develop an attitude of “I hate Jira, it’s just boring data entry and updates” — a humourous quote I seen as a younger developer was something along the lines of “If I were Voldemort, I’d have hidden my horcruxes in the Jira UI — Harry’s never finding that!”. But leading a team requires you to know what’s sub-optimal without actually being at the coal face as much, and what’s going wrong between humans — you can’t gauge these high level metrics without data. As an individual contributor, you can gauge what you feel is wrong, but it’s not your job to spend time looking at the data and uncovering systematic problems. You need this valuable data from Jira or other tools, in order to do that well.

I’d actually recommend reading through Atlassian’s Agile guide. Yes, I can appreciate the irony of buying a product from a company for the purpose of implementing values, but Jira is a very widely used tool, and their Agile guide really hows how you can make the most of understanding your team’s working processes and metrics.

Take the agile principles to heart

The founding agile principles are a great statement, and if you’ve ever worked really effectively in a team with your users, these should ring true for when things go well — you communicate clearly and often, you build things that work, and you make changes when you find something better.

I’ve seen teams blindly follow scrum practices, but this works so much more effectively when everyone understands and embodies the reason for doing those things. I would definitely advocate for everyone have a proper introduction to agile, rather than simply dictate practices to the team.

Conclusion

I’ve learned a lot by reading about agile and putting what I now know into the context of the places I’ve seen it, and it’s now helped me spot a lot of things I missed before. I’d throughly recommending diving deeper into what agile means to understand and practice it.

I’ve taken some chunks from Learning Agile and the Scrum Metrics for Hyperproductive teams paper, but would definitely recommend this reading to anyone interested in these areas.

If you want to read my handwritten notes and see how well reMarkable converted them to text, take a look here: https://github.com/Mark-McCracken/handwritten-notes/blob/main/Learning%20Agile%20notes.pdf

Enjoyed this?

This article was a little late, but my next one will be a more technical article around BigQuery and the practical applications of nested and repeated fields, and should be published within the next week. Follow me to get notified.