Software Development at Airtime
In this post I am going to describe the process we use to develop software at Airtime. I’ll be focusing on the day-to-day developer flow and how a feature goes from ticket to deployed software.
This post is for anyone interested in how Airtime builds software, or to help you think about your own software development process.
It may seem misguided to talk about “Software Development Process” in the context of a startup like Airtime. After all the term “startup” for many people conjures up images of a handful of developers huddled over their laptops cranking out product features as fast as they can type and deploying them to production so that customers and investors can see how fast the product is evolving.
But what you may not think of immediately is that group of developers is using a process:
- Decide what the next big feature is
- Sketch it out on a whiteboard (optional)
- Write code
- Ship it
- Go to (1)
What is important for your company is to pick a software development process that enables you to accomplish your business goals as fast as possible.
You probably would not use the above process for a company that is developing medical or industrial control software since there are real world dangers involved which present an existential thread to the company, whereas that process is probably fine for a company making an online editor for dog memes.
Also, the example process above does not scale to larger teams due to the growing communication and coordination complexity.
Focusing On What Matters
Like any startup, Airtime’s primary goal as a company is product exploration and finding product market fit. To do this as fast as possible we want to spend the majority of our time developing product features that matter to our users.
For us this means minimizing the time we spend on bookkeeping and outsourcing what we can.
We employ automation and tooling where we can and strongly prefer 3rd party solutions we do not have to maintain.
Why do the heavy lifting when someone else will?
At Airtime we track tickets with Jira, host our code on GitHub, build and deploy via Jenkins and Circle, distribute apps and collect crash reports via Crashlytics/Fabric, and use Fastlane for the iOS client for handling provisioning profiles and upload to the App Store.
We use a variety of other 3rd party tools but those are just a sample to give you an idea.
Airtime’s Software Development Process
Before talking about the solution, let’s talk about the goals we wanted our process to achieve.
- Clear Ownership. Progress on work should have a well defined state and owner. Make it obvious what happens next.
- Visibility. State of development is easily visible to other team members.
- Minimal complexity. Again, focus on what matters. We want to spend our time building, not bookkeeping.
Clear Ownership
It is critical to have clear ownership of work. Think of the real world scenario where someone living by themself has trash that needs taking out. It is clear that it is that person’s responsibility take out the trash. As soon as you introduce roommates it becomes much less clear and typically fighting ensues unless you create some sort of schedule or system.
In our case, tickets start in an Open state and are assigned by the engineering manager to a developer. That developer moves that ticket to In Development at which point it belongs to that developer.
When that developer has implemented the ticket, they move it to Branch Testing and the ticket is now owned by QA. QA can either fail the Branch Testing and send the ticket back to the developer or indicate Branch Testing Passed at which point the developer owns merging the feature branch.
Once merged QA once again owns performing integration testing and moving the ticket from On Staging to Ready for Release.
Note that in the diagram above Ready for Release transitions to Resolved once a new version of the app has been released or the service deployed.
Visibility
As soon as an organization grows to more than a few people, communication and interruption become a real problem and slow down your organization’s progress.
The secret to combatting this is to make sure tickets are in the right state and up to date. This allows any team member to asynchronously make the right choice as to what to do without needing to wait on or interrupt anyone else.
Minimal Complexity
The easier it is for a developer to have mental model of your development process the better since it will feel like a tool rather than a constant puzzle to be solved. Even if you are able to automate much of the bookkeeping process it is good to understand what is happening and why.
Save your focus and concentration for developing software that matters, not for trying to remember how your software development process works.
The worst situation to be in is having a manual complex system since that will surely burn out your team.
Working With Git
Similar to our software development process goals above we also have specific goals we wanted to achieve with how we used Git.
- Stable Development Branch. An up-to-date, stable branch as a starting point for features so that each developer is not distracted by bugs and build issues that aren’t theirs.
- Feature branches. So that engineers can share in progress work with the product team and QA that are only merged once everyone is satisfied with the changes.
- Release branches. Allows us to know what changes went into a release and also allow for an easy starting point to make hot fix releases with targeted changes.
These goals could be achieved with the much-referenced “A Successful Git Branching Model” (diagram below) but for us it violated our high level goal of minimal complexity.
Instead what we opted for is a simpler model in which we have a develop branch that has the latest features where other developers can start branches for their feature work from, and where periodic release branches can be branched from. The two models are largely similar except we do not use master and our hotfixes are done directly on the release branches.
Tooling and DevOps
Previously I stated that you should focus on what matters and this means also knowing when to automate so you can use the saved time to build product instead of manually distributing a test build for example.
It can be fun to write scripts and tools but if they are only used infrequently or only by a small subset of people of your organization then it is probably a net loss.
Use your head or a convenient table.
Here’s a few DevOps projects we thought were worth it at Airtime.
Automating Builds
At Airtime anytime code is pushed to a feature branch, develop, or a release branch on GitHub a build is kicked off.
If you have more than one developer on a team you should always automate the building of branches that are part of your developer flow.
Here’s a couple of the reasons:
Minimizing downtime. Per our branching model above, every developer starts a new feature from the develop branch. If develop does not build then no developers can start a new feature without fixing it first. This means multiple developers may spend time fixing the same build failure. Takeaway: Always build develop every time it is updated.
Checking all targets and tests. In our experience developers rarely check that all targets/configurations build and tests pass when pushing code to a feature branch. And frankly it’s probably better that they don’t since rather than eat up the CPU cycles of their development machine they can let a build server check.
The best thing about automating your builds is that once you set it up, it is easy to add it everywhere. Given how cheap build services are these days and the benefit it provides, there is little reason not to do it.
Distributing Builds
We also thought it made sense to make it as easy as possible to distribute builds. This means that it would not only be easy for developers but for anyone to send themselves a build they needed.
Specifically we wanted to make it easy for
- Product managers to see in progress work
- QA to be able to get a build for a feature branch they are testing,
- Broader Airtime team to get an early preview of what is about to be released
- For developers to upload builds to the App and Play store (mobile teams only)
For the first three goals, we built a system where if someone comments qabuild or stagebuild on a pull request, a build is triggered on Jenkins (needed for keyword detection) which starts a CircleCI job which uploads the build to Fabric and distributes it to the QA team or everyone respectively.
We also build develop once a day if there have been any changes that day and distribute the build to the entire company so everyone can see the latest and greatest changes. They also feel confident that if they see a bug that it is probably still a bug and should report it.
The fourth goal is currently only relevant for the iOS team. Any time a build is pushed to a release branch it is automatically uploaded to the AppStore. We do not release automatically but it simplifies the process of getting release builds uploaded.
Integrating Development Tools
At Airtime we decided to integrate our development tools as much as possible to reduce time wasted checking multiple UIs and keeping them in sync. This has become much easier over the years so there is little reason not to do it.
There are some suites of tools such as GitLab that come integrated out of the box. Chances are though that your team is using at least some tools that require some level of integration.
Here are a few examples of what we’ve done:
Slack and Everything.
At this point most organizations that run Slack have integrated it with their development flow and there are plenty of articles about it. We try to use it as our information center for everything.
We use Slack integrations for notification of build failures on CircleCI and Jenkins, backend service deployment status from AWS, creating JIRA tickets in the flow of a discussion, service interruption alerts from 3rd party services we rely on, notifications on ticket flow on jira, and the list goes on.
Atflow
In order to automate the bookkeeping tasks that should be done in Jira when working on a ticket we created a tool called Atflow (short for Airtime Flow) based on hubflow and go-jira.
Atflow allows us to start development work on a ticket from the command line and the tool takes care of naming the branch and updating the ticket state in JIRA and making sure it is in the current sprint.
By ensuring that you are naming your branches and pull requests correctly you get the little magic in Jira like this on every ticket.
A Few Final Notes
Product Development
In the interest of brevity and staying on topic I did not go into much detail about roadmapping, feedback, and task breakdown.
Suffice to say that we have product managers that take input from a variety of sources, create a feature roadmap, collaborate with engineers and designers on the best way to break up the roadmap into high level feature epics which engineers can then break down into individual tickets.
In the future we will have an article from the product team on how we do product development.
Process Variations Between Teams
Note that building an app like Airtime requires multiple engineering teams and each team has slight variations in the process necessitated by the challenges they face but large in part the process is the same across teams.
A few examples of major differences I did not mentioned above:
- Our backend team relies on test driven development (TDD) rather than manual QA
- Our media team has a combination of automated and manual QA test suites for validation of new media server and client library releases.
Manual QA Versus Automation
For app testing we rely on manual QA as opposed to automated UI testing since our product spec changes frequently as we adapt the product to what users want. We felt it was more actually more effort to maintain the tests until we got to a certain level of product market fit.
Final Words
Good luck developing and improving your own software development process for your organization.
If you like what you see and what we are working on then come join us!
