Today, if you work in software development, you use Scrum. Or Kanban. You are Agile. But does it work? Does it help you toachieve the goals you set? What goals did you set when you deciced to use Scrum and be agile? What is the purpose of Scrum and Agile anyway? What is the purpose of any method for software development?
Goal
The goal of a software development project is to deliver software with the following features:
- high quality
- minimal cost
- within a pre-defined time period
Cost and time are obvious, quality needs more specification. Quality is
- product implements all required features in the way they were specified
- the product is good, in the sense that the user likes it and likes to work with it, or at least accepts it
- the product has no bugs or flaws — this includes security issues, performance issues, errors, crashes, etc.
You want to specify all kinds of things in advance so you can monitor progress and verify that you received what you ordered. Because you don’t want to specify in too much detail at the start of the project, you divide your project in smalls steps (sprints) and you specify and verify every two weeks. Requirements in consecutive iterations are not necessarily on a smooth path towards a goal. Stakeholders (users and customers, represented by product managers and product owners) may change their focus and require features and functions that are not in line with what you have been working on so far. Fortunately, product owners stop stakeholders from changing their mind, they make sure that requirements remain focused towards the original goal.
A different approach
I like to work in large projects in the same way I work in small projects. The smallest projects I know are one person projects where you are the owner, the developer, the user and everything. One of my projects is www.schwartze-ansingh.com. It started with me getting two boxes of historical letters. I had no plan when I started to scan the letters. My long term goal was to write a book. When scanning the letters, I realized I needed a list or table to keep track of them. After I scanned the letters, I decided to transcribe them into ascii. I added data to the table, which I had created in postgres. When making notes for the book, I converted the single table into a structured database so I could store the notes. To access the database I wrote an API in Spring Boot. Then I realized that now I had everything in a database, I could make it available via a web site. Then I thought a web site would be more useful than a book, so I dropped the book and added more functions to the web site. The web site is work in progress, and it will be so for many years to come.
If I would create a development method from this project, it would go something like this:
- set long term goal without being too specific (“write book”)
- set short term goal (“create inventory of letters”)
- start working (“start scanning the letters”)
- reflect (“I need a table, letters are hard to read so I need to transcribe them”)
- start working (“add notes, transcribe letters”)
- reflect (“I need a UI, I need an API”)
- start working (“build API in Spring Boot”)
- etc.
Working and reflection are not alternating processes. They are simultaneous: while working on something you realize you need something additional or you need to change things. Traditionally, you decide on architectural issues and tooling at the start of a project. These decisions then limit choices you make further down the road.
What you really want to do is to postpone all choices that may limit you in what you do. If you need a table, use a text file or a spreadsheet. When the data gets more complex later, you can decide on a database. When you have the database, you use psql or pgadmin to enter simple data. Later, when you need a UI, you switch to Python or Spring Boot. When you decide to publish some of the data, you add Apache Tapestry or Wicket to your project and you create some simple pages. At the same time, you are still adding functions to the back end, you are still scanning and transcribing letters. You spend an increasing amount of time using the system your are building, in my case, finding information about the senders and recipients, about the events and people described in the letters, etc. You get annoyed by the way your site works, so you replace Tapestry by React. When the body of information you have collected grows, you add Apache Lucene to the project so you can find your stuff more easily. And this goes on and on, you scan letters, you do research, you add functions, you replace tooling and functions. In my project, this is still going on.
Working this way makes me happy. I call it Nimble Software Development: it feels like dancing along a path that zigzags around obstacles towards beautiful views, while your goal is to get to the other end of the wood. When there’s a fork in the path, you sit down and think, when there is a cliff, you retrace your steps back to the last fork or crosroads. Nimble means move when you need to move, where you need to move, at minimum energy and maximum result. Nimble means don’t worry to much about things that are in the future, all you need is a vague idea where you want to go. The only thing that you need to be absolutely sure of is where you put your foot next.
Top-down or bottom-up
Does design and development of a software system work top-down or bottom-up? What works best? Let’s imagine how you plan and design a web system.
You start with a general idea on what you need. “You” is a customer, or users, or the management. They decide you need an archive for the antique letter collection of your museum. Secondary requirement is that you want to make the collection open to the public.
Scrum would proceed to create user stories that describe in detail actions that users, visitors and conservators can take. The system is created bottom-up: you create the first sceen in all detail, then you proceed to the next screen. It’s like painting the Night Watch, starting in the upper left and working to the lower right. I doubt you’d get a painting that is worth watching.
Rembrandt used a different approach. I envision he did the following. He started from the assignment “paint the militia of Captain Frans Banninck Cocq” and worked from there. Before putting the first stroke of paint on the canvas, he thought about the composition. He knew he needed something more innovative than a static image of the militia. Rather than locking himself up and think about the composition, he just gathered his models (militia members, or maybe replacements, I don’t know) to experiment with the composition. While they were taking their places, he saw what he needed to do: he should paint them as if they were preparing for a parade.In his minds eye he saw the painting, with all figures in place, with the proper colors and lighting, with the arquebuses and drums and banners. Only then he picked up a brush. He drew, with big strokes, not Frans Banninck Cock, not a drum or banner, but the outline of the group. A circle where Banninck Cock and Ruytenburgh will be, a few doodles on the left for the arquebusiers, a few lines on the right for the drum and the lancers. A spotlight on the lady in white.
Had the Night Watch been a software system, Rembrandt would now have the tooling and architecture. Next, he can start to actually create code — the details of the figures in the painting, changes to the composition.
Like Rembrandt working top-down, first creating the composition and then proceding to add detail, I think software systems should be built top-down too. In Nimble Development, you first create a top level abstract goal, subgoals that are a bit more specific and a little closer in time, and even closer goals in more detail.
Nimble Development
For a team, in a larger project, Nimble Development would look like this:
- Assemble a cross-functional team of developers, users, testers, designers, devops, etc.
- Sketch a long term goal (“long” can be anything, from a month to ten years)
- Make sure all team members understand the goal and agree with it.
- Set short term goals, the shorter term the more specific.
- Make sure communication is in place. Do not plan meetings. Instead, create multiple communication channels connecting all team members. Channels can be online (Slack) or off line (speech). Avoid meeting rooms.
- Start “work” and “reflect” as parallel and recursive processes.
- “Work” means design, build, test, deploy, or anything that’s needed to deliver a product.
- “Reflect” means evaluate what you’re building, set short term goals, think about the long term goal and change it if needed, write new requirements, etc.
- While working and reflecting, adjust goals. Make goals more specific, maybe shift focus or direction.
The Team
It’s the team that makes the project, not the method. The team needs to be
- cross functional
- empowered
- determined
- communicative
‘Cross functional’ means you need developers and designers for their skills, you need users and stake holders for the information and experience, you need testers and devops.
‘Empowered’ means the team can make its own decisions, no meetings with management required, no procedures for approval. The team knows what needs to be done, it wastes no time doing it.
‘Determined’ says the team has a drive to get things done. With no detailed long term goals and no management setting deadlines, the team needs to have a strong motivation to deliver the right product asap.
Because you don’t have meetings, team members need to be communicative: you want to exchange information about the product and about everything you are working on whenever the information is available and needed. This is the hard part of Nimble Development: knowing when to share information and knowledge with other team members and when to keep quiet and work.
I like to think of a development team as a special forces unit that knows what it needs to do and just does it, solving issues when and where they occur. They talk when they need to talk, they keep quiet when they need to keep quiet.
Conclusion
Nimble Sofware Development means
- Your team is cross-functional
- Your team is empowered
- Start designing top-down
- State a long term goal without much detail
- Add features as you go
- Work in short cycles of “reflect/build”
- Don’t do meetings
In Nimble Development, your team members are dancers who dance along a path, picking flowers when they see fit, planting trees in open spots, creating the product by typing their code.
