Measuring Engineering Productivity and Advancing Software Delivery

8 min readJan 29, 2024

The rise of Agile methodologies, DevOps practices, and cloud infrastructure, organizations are able to build and release software faster than ever before. However, with the acceleration of delivery speed, measuring productivity and improvements can become more challenging. Traditional metrics like lines of code or number of features NO longer apply in an environment optimized for speed and flexibility.

New methodologies require new productivity metrics focused on business value over output volume. Cycle time, deployment frequency, time to restore service, and change fail rate paint a clearer picture of an engineering organization’s effectiveness than outdated vanity metrics. An ideal set of metrics highlights software delivery performance, quality, and predictability across the entire organization. Each metric provides insights into a key business objective:

Delivery Performance

Lead Time: The time it takes to make a proposed change available in production to users. It starts when code is committed and ends when code is successfully running in the production environment. Tracking lead time reveals bottlenecks in the process. As teams remove obstacles, lead time shortens.
Time to Restore Service: The minutes or hours between identifying an outage or service disruption to restoring service. Mission-critical services and infrastructure should have a low time to restore. Any minutes or hours of downtime directly impact the business and customers.
Change Fail Rate: The percentage of changes to production that fail. This includes rolled back changes, changes that require hotfixes, or changes that impair users. A high change fail rate signals low quality, gaps in testing, and a lack of guardrails around changes.

Release Frequency

Deployment Frequency: How often an organization releases changes to production per day, week, or month. The more frequently you can package and deploy changes safely, the more you accelerate value to customers.
Mean Time to Recover (MTTR): The average time it takes to recover from a failed change or incident. A lower MTTR indicates teams can quickly roll back and restore service when failures occur.

Predictability

Change Lead Time: The average time for all changes from code commit to production, giving visibility into the throughput of the entire system. Consistent lead times signal predictable, reliable processes.
Cycle Time: The time it takes to complete a work item, feature, or user story from start to finish. Cycle time measures throughput for individual units of work and can highlight bottlenecks.

Productivity Enablers The pillars of software delivery productivity consist of architecture, process, automation, and people. Thoughtful attention to each dimension unlocks compounding results.

Architecture A domain-driven, modular architecture reduces friction during downstream development. When the codebase contains clear domains with explicit boundaries, teams can make changes faster. Architectural decisions balance several technical dimensions:

Modularity: Components have strongly defined responsibilities and interfaces. Different modules can be maintained by different teams with limited coordination.
Testability: The design enables the right testing strategies and assertions around business risks. Legacy codebases often lack automated test coverage.
Deployability: The system can release small changes to production frequently in a safe manner. New services can deploy independently from legacy code.

Key metrics to track architecture fitness:

Time to add new feature: On average, how long it takes an engineer to build, test and deploy a new backend feature. As architecture improves, adding new well-defined features accelerates.
Scaling cost: The cloud costs or hardware resources required to safely operate at peak demand. Costs should scale linearly, not exponentially, with increasing traffic volume.

Process The deployment pipeline, branching strategy, code reviews, and development process directly impact productivity. Thoughtfully designed processes remove wasted time and effort, while poorly defined ones inject friction. Key process principles include:

Small batch sizes: Committing smaller units of work more frequently yields faster feedback loops.
Peer code reviews: Thorough human inspection catches bugs and spreads knowledge, while self-service frameworks remove bottlenecks.
Trunk-based development: All engineers commit to a shared mainline branch, avoiding long-lived feature branches and integration debt.
Test automation: Fast, reliable unit and integration tests shift testing left and prevent defects from escaping to downstream environments.
Tight inner loops: Local development environments mirror production giving fast feedback without waiting for pipelines.

Important process metrics:

Pull request cycle time: The critical path time it takes to iterate on a proposed change. Lower cycle times enable faster collaboration.
Test coverage %: What portion of the codebase is exercised by automated test suites on each commit, signaling quality.
Mainline commits per day: More commits directly to trunk indicates reliable processes for small, frequent changes.

Automation Automating manual tasks provides outsized returns as engineering orgs scale. Test suites, deployment pipelines, and infrastructure provisioning are prime candidates. Automation also standardizes and de-risks mission-critical processes. Key focus areas include:

Deployment pipeline: Automates build, test, inspection, and release steps for fast, reliable delivery.
Testing: Runs fast, broad regression suites against every commit to prevent defects and technical debt.
Infrastructure: Treats infrastructure-as-code and fully automates provisioning and config management.
Incident response: Playbooks codify incident workflows, empowering teams to restore service rapidly.

Metrics for automation maturity:

Deployments per day: How often the average service deploys to production daily or weekly. Frequent, automated releases compound productivity.
Failed deployments: The change fail rate specifically for the deployment process. Should trend to zero with increased automation.

People Skilled, aligned engineering teams create organizational leverage. Too often, technical training and practices fail to keep pace with innovation. Structure, incentives and culture profoundly impact talent productivity.

Small, independent teams: Align ownership to modular domains fostering autonomy and innovation.
T-shaped skills: Balance specialized experts with engineers having both depth and breadth to increase flexibility.
Aligned incentives: Share goals across functions connecting technical decisions to business impact.

Key people metrics:

Engineer turnover: Losing existing team members slows productivity until replacements skill up. Track turnover rates over time.
Average tenure: Longer tenured team members master code, tools and processes to operate more effectively.

Measuring Software Delivery Performance Many organizations still rely on vanity metrics like story points or utilization rate. These local optimizations often distract from broader business goals. Mature engineering teams instead focus on end-to-end metrics capturing the flow of value to customers:

Cycle Time: Cycle time measures throughput and flow through the entire system. It starts when a task enters the backlog queue and tracks each subsequent state change until its completion. Cycle time averages variation out across work items, revealing constrained process steps. Engineering teams can influence cycle time through technical initiatives like test automation, trunk-based development, and modular architecture.

Lead Time: Lead time starts from code commit and measures throughput from development into production. It exposes deployment pipeline bottlenecks and reliability challenges failing over to production environments. Engineering teams directly own lead time improvement via CI/CD automation, infrastructure as code, and release procedures.

Time to Restore Service: The minutes or hours between an identified production failure or outage to full restoration directly fuels business continuity. Rapid recovery from incidents requires both technical capabilities and organizational alignment. Mature teams elevate monitoring, incident response processes, and resilience practices as first-class concerns.

Identifying Key Metrics The most influential metrics provide insight into the constrained steps in a system. Monitoring cycle times across development stages highlights opportunities to amplify throughput and team productivity through focused improvements.

Common constraints include:

Lengthy test and release pipelines causing slow feedback and inhibition of innovation
Unreliable production changes leading to failed deployments and impaired users
Time-intensive manual tasks like server provisioning and configuration management
Poor modularization creating external dependencies and coordination costs

Improvements targeting identified constraints unlock massive productivity wins and accelerations in business value. For example, test automation, continuous delivery, infrastructure as code, and bounded contexts each address core bottlenecks.

Measurement Challenges In practice, measuring software development productivity presents several inherent challenges:

Difficulty defining units of output — Lines of code and story points fail to capture business value created and conflate effort with results.
Variability across projects and teams — Teams, technologies, system age and technical debt skew comparisons.
Lagging and incomplete data — Code commits and deployments provide the most insightful signal but require instrumentation.
Incentive misalignment on metrics — Local optimizations like utilization rate and story point velocity distract from system throughput.

Addressing Measurement Challenges

Despite challenges, insightful metrics exist to gauge productivity, quality, and performance. The keys include:

Define units of work based on customer or business value like features delivered rather than output volume.
Instrument critical process steps like commit, test, review, deploy to benchmark cycle times.
Aggregate measures across projects and teams to dampen variability and reveal constraints.
Incentivize group outcomes like lead time and production change success over individual metrics like utilization.

Even imperfect metrics aligned to business value provide better guidance than vanity metrics. Treat measurements as a compass, not a scorecard, directing attention to bottlenecks. As engineering investments target constraints, more meaningful metrics naturally emerge.

Driving Productivity Within Organizations

Improving software development productivity requires both technical practices and cultural principles.

Cultural Transformation

Ultimately, accelerating software delivery hinges on people and process more than technology. Changing ingrained behaviors across large organizations represents the hardest challenge and most overlooked lever.

Leadership Principles

Customer obsession: Relentlessly prioritize end user value over technical elegance. Ask “why” 5 times more often.
Data-driven decisions: Form hypotheses, run experiments and learn from evidence, not just intuition.
Innovation culture: Make “change” the only constant not merely an occasional project. Default to openness over controls.
Mission focus: Clearly communicate how every engineering team ladders up to business outcomes over heads-down execution.

Team Principles

End-to-end ownership: Avoid hand-offs between functional silos. Keep teams focused on delivering customer value.
Continual learning: Dedicate time for upskilling, reading, and internal mobility. Don’t let daily execution dominate available hours.
Balance autonomy with alignment: Enable teams to choose solutions while connecting decisions to business impact.

Developer Principles

Deploy to production weekly: Commit code in SMALL batches without branching or long-lived feature work.
Fix forward, not back: When bugs occur, add preventative tests first before debugging. Prevention > treatment.
Optimize for flow, not utilization: Take on batched work rather than 100% allocated time. Flow efficiency beats resource efficiency.

As leaders instill these principles, engineering productivity flourishes bottom-up through ownership, mastery and purpose.

Overcoming Resistance

Attempting any degree of transformation inevitably encounters resistance. Long tenured team members enjoy and defend the status quo. Middle managers worry about losing control or decision authority. Heated technology debates distract from business outcomes.

Successfully nurturing change requires transparency, inclusion and empathy combined with relentless persistence highlighting the mission. Common hurdles include:

“We don’t have time for that”: Daily execution fully consumes available schedules. Block learning and exploration opportunities.
“Our context is too complex”: Legislating “one size fits all” solutions often fails, but neither does complete relativism. Seek nuance.
“We’ve always done it this way”: Change upends existing social structures and norms. Appealing to “how it’s always been” excludes new voices and ideas.

Get Ahead of Objections:

Celebrate small early successes to demonstrate benefits skeletal and incremental over wholesale “boil the ocean” attempts.
Involve sympathetic team members to tailor messaging and demonstrate peer advocacy, not top-down mandate.
Outline a compelling vision for HOW change positively impacts people and their day-to-day work, not just the business hopes to gain.
Accept lasting company-wide transformation takes years of sustained focus beyond any initial rollout. Prepare for setbacks during the journey.

With careful persistence and inclusive participation, engineering teams transform from rigid and siloed to flexible and aligned. Resulting speed and stability unlock innovations satisfying both customer needs and business performance sooner.

Summary

As software increasingly powers competitive differentiation, measuring and elevating development productivity sparks a positive feedback loop. Thoughtful metrics expose delivery constraints while cultural principles empower teams to invent solutions. Distinctions between technical capability and business value fade through shared mission context. Architectural initiatives accelerate system throughput. Automation scales reliable processes. With aligned insights into end-to-end value flow, engineering leaders can multiply returns on precious people hours putting innovative ideas into customer hands orders of magnitude faster.