The state of software development. Are we really increasing development teams productivity?

Over the last 10 to 15 years we have witnessed a complete transformation of how software development teams operate and build products. I have been building software products and leading product and engineering teams for over 25 years, but the relatively recent innovations in tools, platforms and processes has probably been the most dramatic over the last few decades. Several catalysts played a key role in how we got to today’s typical processes and disciplines in software development. In this post I will examine two of the most noteworthy and discuss what life is currently like to build & ship software and what could possibly be done to improve the productivity of development teams.

1. The move to off the shelf tooling.

The disappearance of “internal tools teams” for software development and the move to off the shelf 3rd party tools is a clear example of the wave change in this direction. If you’ve been around for a while, I am sure you had either worked with an internal bug tracking system or had a deployment of bugzilla (shudder) that you used, before teams moved to Jira and Atlassian tools for issue tracking and overall project management. Either your teams hacked together their own source code control systems or you had an on premise installation of Mercurial, Subversion, Perforce, Clearcase, etc before like most of the industry software development teams moved to some variation of git, and most likely GitHub. This same trend happened for CI/CD as the industry moved from home grown tools and scripts to Jenkins or a similar SaaS offering like CircleCI or TravisCI. Last but not least Google Docs or latest versions of Office 365 have replaced personal installs of MS Office as a tool for product managers to define products and requirements, but that shift has, for the most part, been a shift from local installations of office productivity to SaaS tools with additional collaboration capabilities.

2. New disciplines and platform technologies.

Agile and scrum tore down the old archaic and slow processes to help increase productivity and speed up velocity and rate of delivery. As use of SaaS, cloud platforms and collaboration tools took hold working with remote and distributed teams became the everyday norm. Terms like DevOps, continuous integration and continuous deployment (CI/CD) became the common day language of product teams and several tools and technology platforms, most famously Docker and container orchestrations like Kubernetes, rapidly revolutionized the CI/CD space with a focus on ease of use and speed of development, deployment and delivery. Having said that, there are several industry debates regarding the “ease of use” of orchestration platforms, but that is the subject of another blog.

So where are we now?

It is worth noting that the new software development tools should get full credit for delivering real value in their respective verticals. They each have done a pretty fantastic job of solving the problems in their space. If you’re developing software today, you would not even remotely consider building your own tooling for project management, developing products, testing and deployments. While it’s common to pick the best of breed tools, what is difficult, and thus missing, is the custom automation and integrations to create natural workflows and a set of life cycles that would connect these tools and steps together to create a seamless end to end and mostly automated process.

The reason I say these integrations are custom is because product development orgs and even teams within a single product organization have slight variations to the process and procedures they follow. Moreover software development in general unlike general manufacturing is not an assembly line process, which is why it makes it difficult for general and holistic “one-size-fits-all” integrations to solve the customization problems.

To clarify what I mean by custom processes, just consider how software development teams define agile “story points”, or the process and quality of their code reviews, code merge or testing and deployment procedures. It’s very common that these procedures and many others have different variations within a single product development organization let alone an entire industry. To solve for these variations in process, policy and thus the ensuing workflows, organizations have become accustomed to inserting human driven and controlled ceremonies strictly designed to gather related data and moving it along to ensure a smooth flow. A clear example of this is the scrum ceremonies of daily standup where literally thousands of hours are spent every year by a single medium sized software development team providing daily status and updating information manually and periodically spending more time pulling reports together in slide decks which are soon forgotten, and yet releases are often met with delays and missed deadlines. In some extremes there are roles whose sole job is to gather the information from all these tools, sometimes by consistently interrupting busy developers, to ensure information is kept relevant and reduce the chance of error and delay. In the best case scenario, which has become more common, the job of keeping the information in all these systems up to date and flowing has fallen on software developers who also have to code, test, deploy and monitor their services. There is plenty of data that suggests developers spend at most 2–3 hours a day on code and the rest of their time is spent on manual tasks, updating data, status meetings, chasing production fires, and other non productive activities related to keeping up with the manual processes and running their services. Can this really be the most effective and productive way to build, deploy software and manage services in production?

While tool and platform companies are hard at work making it easier to work within their respective vertical, for their users it does often feel like moving between siloed systems with their own sources of truth, separated by brick walls, with the recent salvation of catapulting everything into a collaboration messaging tools like Slack in the name of transparency and productivity. And it’s quickly becoming a ticking time bomb of productivity loss. To be clear, Slack is a fine tool for group collaboration and even systems updates to groups or individuals. However, simply dumping information into overloaded Slack channels is a far cry from a system where information flow and data synchronization is fully automated. Furthermore we have quickly reached information overload in tools like slack to the extent valuable data, critical to decision making and proactive actions, is getting ignored and muted because there is a nonstop stream of information flow, which requires its own significant amount of time and effort to sift through.

So what’s next?

If we are really interested in the next level of software development productivity we need a mechanism that enables full custom automation by giving product development teams the power to automate their manual and time consuming data gathering, data synchronization and decision making processes. Such system has several characteristics:

• It seamlessly integrates with existing tools and enables teams to programmatically and with custom business logic, implemented in code, manage development lifecycles, use data to recognize patterns and to make decisions.
• It has built in mechanism and customizable logic and flows to keep information up to date and in sync between project management tools, code and CI/CD processes.
• It should handle all the authentication, authorization management and upkeep overhead for these integrations and simply enable teams to focus on what processes they need to automate and what proactive steps they need or want to take to keep their services running. This is an extremely important point so as to not overburden product teams with yet another service to manage. Furthermore it should enable many custom automations quickly and out of the box with the implementations fully available in code for customization as needed.
• Users should simply be able to focus on running micro apps (and the respective business logic) that performs their specific process automation or action that is necessary to increase productivity.
• A simple example of this is scrum automation tasks like keeping issue tracking and project management boards in sync with the status of code, providing personalized and relevant data to teams and individuals as needed.
• More complex examples are being able to recognize patterns and proactively taking actions to resolve or prevent failures before they occur or automatically recover from failures once they’ve occurred.

At DevScore we are focused on building a low code automation platform for software development teams to address these needs. Our goal is to ensure software development teams can quickly deploy automation functions that help them save time and improve their personal and team productivity. We provide out of box integration with project management, software development and production monitoring tools. Additionally we will continue to build custom functions and templates so developers can quickly and easily automate their project management tasks, scrum ceremony updates, coding activities and the CI/CD lifecycles.

In future posts we will dive deeper into specific use cases of automation and how development teams can leverage the power of DevScore’s automation platform to rapidly automate tasks and increase their overall productivity with better visibility and control.

Find us on Github and Youtube.

Feel free to reach out and give us your feedback at info@devscore.com