Agile Does Not Mean Unstructured
Keep the Cowboy Developers Back at the Ranch
Agile does not mean unstructured, it means embracing the inevitability of structural change.
This concept is extremely important in today’s development environment where everyone is seeking the holy grail — to minimize MTTD (Mean Time to Deployment) and maximize the number of new features delivered.
I build distributed systems. I build a lot of them. I build them for work, I’ve built them for side projects and startups, and I even build them for hobby projects and books. Whether these proclivities are healthy for me is a topic for another blog post. I’ve also been doing this for a very long time. I remember the days of the waterfall process where everyone spent months (or worse) designing, documenting, detailing, and planning a system before fingers met keyboard. I’ve seen user manuals and developer guides proofed and printed before any code was written.
When things went well, it was magnificent to behold — but things rarely went well. The waterfall analogy should have clued people in to its biggest weakness: once you start paddling down the rapids, it’s nearly impossible to get out of the water until you reach the end. We all routinely missed deadlines, and we frequently told our business stakeholders that we couldn’t get a new feature released until all the water drained from the current Class VI whitewater rapids.
We wanted to get around these unforgiving, sisyphean timelines and we needed the ability to change direction or easily create forks in development to release an unexpected patch or feature. And so Agile with a capital ‘A’ was born.
At the core of the agile philosophy is the ability to deftly react to changing markets, consumer demands, and the blindingly fast pace of technological innovation. Unfortunately, I’ve seen this principle lauded as the reason behind some really poor development practices.
One negative pattern I want to talk about in this blog post is the lack of design or formal architecture inherent in an agile workflow and the impact not implementing one can have on development.
Agility will not rescue you from bad system design.
Tell me if this scene sounds familiar: A talented development team is asked to build a product. They have enough requirements and customer stories to get started. Embracing modern concepts like agile or even extreme programming, the team pops open a few GitHub repositories and they start building. They deliver daily stand-up updates, perhaps they’re adhering to scrum (a topic for yet another blog post…), there are weekly demos that show measurable achievement. Concussive course corrections happen often, though these setbacks are rationalized by comments like “We’re agile!” and “We’re learning!”, so it’s okay. It’s all part of the plan…or anti-plan as the case may be.
What’s missing from the above process? A scout; A map, or any sort of direction determined by a rigorous process.
Over the past few decades (I said I’d been doing this for a while), I have noticed that as the formal waterfall process is eschewed, so too are many of its other formalities, namely documentation of key system components and concepts. These misguided teams (often derisively referred to as “cowboys”) have stopped trying to find a good balance between what I call scouting — planning just far enough ahead not to lead your team into a ditch — and waterfall. Cowboy teams simply embrace the extremes.
Rather than course correcting based on reasonable scouting intelligence and some rough maps or maybe a compass, these “cowboy teams” course correct only when they rebound off of an obstacle. Like a Roomba that picks a new direction once it bumps into a wall, these teams bounce and deflect their way to production, rarely stopping to question if there’s a better way.
Do we really want to teach new developers that the best way to build an entire ecosystem of microservices is to simply open up an IDE and start coding? To answer that question, let’s compare typical paths of two teams using different methodologies.
One team is scouting; documenting the path ahead with architectural diagrams, interaction diagrams, sequence diagrams, and rough milestones that all lead to team consensus. Another team is bouncing — they forge ahead until they hit a wall, then course correct by applying what they learned from slamming into said wall.
There’s a pervasive attitude among misguided purists that if you take the time to stop and draw a diagram of your system, or you stop to mark up a sequence diagram for how systems interact, then you’re wasting time; you’re not being agile enough. This perspective ignores the bigger picture.
Before I had formed any kind of definition of scouting as it pertained to software development, I was on a number of teams where I started drawing diagrams and writing documentation because I felt there was a need. At each pivotal point, I would draw out a vision of where I thought we were going and socialize it with the team. Not only did this force us to come to consensus and agree to ubiquitous language, but the simple act of drawing what we were building almost always forced us to confront deferred aspects of the design, resulting in adjustments — course corrections made to smooth our journey around obstacles. We didn’t have to sacrifice any of our agility to make this happen.
Distributed systems are complicated. Microservice-based systems have hundreds upon hundreds of moving parts. You’ve got to deal with authentication and ingress and egress and inter-service comms and intra-service comms and observability and resilience and … the list goes on and on. In an environment as deep and complex as this, should we blindly fumble around in the dark until we eventually arrive in production through luck and serendipity?
I’m not advocating for waterfall, nor do I think it’s realistic to have a permanently fixed view of our end-state architecture. I’m advocating for an agile practice where, periodically, we draw what we’re going to build, come to an agreement on that drawing, and start building toward it — scouting just far enough ahead to get moving. Assuming that documentation is fluid will help with this practice and encourage us to make the act of documenting systems enjoyable.
True agile methodology means embracing the inevitable change to these artifacts based on new information; the results of spikes and prototypes and new technology. But blindly building with no real waypoints makes no sense. Perhaps even more importantly, the mere act of getting people to achieve consensus on a diagram generates incredibly meaningful questions and usually gives you more insight about your system than you can glean by pinballing around impediments. This consensus is often more beneficial than the drawing itself.
So, in conclusion, go forth and be agile — but scout ahead first.
And leave the cowboys at the ranch.
These opinions are those of the author. Unless noted otherwise in this post, Capital One is not affiliated with, nor is it endorsed by any of the companies mentioned. All trademarks and other intellectual property used or displayed are the ownership of their respective owners. This article is © 2019 Capital One.
