The treacherously high cost of software

The idea that businesses should write, own, and operate software for any undifferentiated part of the business is an asymmetrically poor value proposition. By undifferentiated I mean a component of code that does not need to be reinvented to support your business or serve your customers. Your support ticketing system, CRM, and payments stack are undifferentiated.

The asymmetry mentioned above comes from the fact that an undifferentiated software project will contain all the costs and risks of any unique or strategic software project but a capped upside: At best you get a perfect piece of ancillary software, which by its very nature has a limited opportunity for impact. At worst you get a seemingly bottomless pit of money and time that slows your pace of innovation.

As your business builds software you should be cautious — the initial implementation isn’t the end of your investment it’s the beginning. Each line of code you write is like a pebble you add to your backpack as you trek along your company’s mission. Those pebbles can add up.

High tech businesses often overestimate the ROI of building custom tools to automate or manage various aspects of their business (analytics, reporting, billing, payments, software infrastructure, development tooling, etc). Indeed, seeing how expensive and difficult so much of this work can be made me bullish on B2B software generally — and in part motivated my interest in joining Stripe. In my role at Stripe I often get a view into how different companies solve these sort of tooling needs — and how it turns out.

To date I’ve seen many firms (especially technically inclined ones) get into a bad habit of building their own solutions for many parts of their business. This is likely because we’ve been living at the dawn of the computing age and suitable software platforms have been scarce. This habit is bad because you start adding pebbles to your pack at a rapid rate without producing much differentiation or advantage to show for it.

Software is perceived to have favorable economics in part because there is essentially zero marginal cost to run it more times or let more people run it (excluding the cost of processing data — which can be eye wateringly expensive). If you’re doing avoidable and undifferentiated software work for your firm and you have to upgrade or change the software often, you will lose out on both of the types of ‘zero marginal cost’ and your software will be quite expensive.

Why is software expensive to build and maintain?

1. Just treading water costs money if you want to be compliant, secure and operable — and it’s about to get much worse.

As noted above, software is only especially cheap when it runs quietly in the background without requiring human intervention. Software engineers are expensive to employ, their work is often hard to estimate and their efforts can take a significant amount of time. The less frequently an engineer is working in a given codebase the less their output will be rapid and predictable (this makes matters worse for ancillary systems).

How likely is it that a firm can simply build and then run-but-not-touch a software system assuming no requirements will change?

While your business’ needs might not change the outside world is a source of constant change some of which will collide with how you do business. Compelled changes can cause cascading increases to the scope of work and cost:

• Security: New exploits and problems are constantly being discovered and these always require immediate remediation. For example, imagine there’s a security issue in the version of Java you’re using, uh oh. As your engineers work to fix it they realize the latest version with the required security patch is far ahead of your current version and various dependencies don’t upgrade smoothly.
• Regulatory: Legal shifts mandate changes on a timeline your company doesn’t control. The problems we discuss here are also about to get *much* worse. Just as historically we’ve been at an immature state of B2B software readiness we’ve also at the beginning of the era of internet regulation. Compelled changes driven by governments around the world are increasing not decreasing. GDPR, PSD2/SCA, data locality requirements and more are on the upswing. and as more and more businesses look to operate globally these set of challenges will become crushing.
• Dependency changes: Some system you depend on could upgrade the way they pass data back to you or take requests and this will break your integration and thus your entire application.

The regulatory changes above are especially notable. GDPR was a particularly salient example since it had such a huge impact radius — essentially every software business with customers in the EU was impacted. If you build an internal applicant tracking system (ATS) and happily run it for 4 years and then GDPR comes out, you are now forced to spend engineering time adapting to the GDPR requirements or your core business will not be able to licitly certify compliance to the government or your customers. If you have an ATS and 41 other internal systems which have arguably personal data in them, you get to do 42 times the work. A company betting its future on being the best ATS in the market has priced in the risk of this maintenance work and is able and willing to pay for it. You likely have not and are not. Trying to cobble together a broad array of scrappy internal tools is going to keep getting worse as regulators continue to become more deeply involved in the internet and computing.

Of course these compelled changes not only force you to do more work but increase the size and complexity of your code base. Yet more pebbles to carry.

2. Your requirements will change

If you don’t build a general purpose platform it will be slow to pivot and make changes to the requirements you have for your software

Sometimes I talk to people who view writing their own software as a way to stay in control of their dependencies and execution. They view outsourcing some important but undifferentiated job as a risk to their ability to react quickly to changing needs.

The problem with this view is that you end up either 1) Writing a specific purpose built solution that will break when you shift your business requirements or 2) Creating a generalized customizable platform that anticipates a broad array of use cases.

1)Takes you down a slow road of increasing cost and decreasing speed. 2) is incredibly expensive — indeed it’s what SaaS providers have to do to build a product to sell — except most companies never realize the scale gains (and that sweet ~0 marginal cost of distribution that helps make up for the upfront investment).

With 1) you’re likely paying more to write your own systems rather than customize an off the shelf solution. With 2) you’re not just adding pebbles to your pack — you’re lifting a boulder.

3. All of these changes will be more expensive than you plan for

Most people don’t realize or think about how much software needs to be cleaned up as it grows and evolves. Engineers spend a lot of time doing things they call refactoring, or paying down tech debt. If you don’t invest in this understanding what is happening in your codebase becomes harder over time and development becomes slower.

Engineering teams that operate live services spend a significant amount of time fixing things when they break and dealing with incidents.

Even simply running services that operate across a large amount of data can be incredibly expensive. Unless you’re investing making systems more efficient and buying infrastructure at scale you’re unlikely to fully realize the efficiencies at SaaS provider can.

When to build vs buy

Of course it doesn’t always make sense to buy a SaaS solution to your business problems. Sometimes building is the right choice. Sometimes building is the only option — there simply exists no suitable software in the market that is designed to serve your needs.

The reasons I would build a system include:

1. Planning to staff a team of 5+ for the next 2+ years
2. Core to my company strategy where I have unique requirements
3. No good off the shelf SaaS solutions

For example, when I was at a large consumer internet ad based company we built a very customized dashboard for the sales team to fit our non-standard sales workflow. In the ads business the sales job wasn’t done when the client signed: Account managers had to work on getting clients to spend through their ad budgets. Our dashboard created a mashup of internal ad system data, salesforce information and management created information like quotas/targets. There was no off the shelf solution and keeping an ~800 person sales team on track was worth a ~perpetual investment of a midsized engineering team.

I would be especially skeptical of building a system if:

1. There exist many exogenous (e.g. regulatory, partner) changes required over time
2. The software requires an area of domain knowledge where our firm doesn’t specialize
3. I don’t plan to have an active development roadmap

I’ve seen many investments turn out poorly with examples of internal built applicant tracking systems (ATS), billing systems, and cloud infrastructure management tooling all of which had to be ripped out and replaced when the complexity and cost of maintaining these undifferentiated systems became too great.

Why is this view non-consensus and non-obvious?

• Availability bias: Because software doesn’t physically degrade or take up space many have developed a poorly calibrated mental model to understand the true cost of ownership of building, maintaining and operating software. We don’t see the cost of maintenance until it’s too late.
• Incentives: I’ve known engineers and engineering managers at a large consumer internet company who pursued projects because they would increase the chance of being promoted rather than solving business needs faster or cheaper than open source or vendor solutions. There’s a series of confounding incentives that lead engineers to want to build more software and engineering leaders to hire more engineers.

SaaS is a great business not because it’s so easy to do but because running software flawlessly is so hard — and actually what the customer wants. Every line of code is like a pebble you have to carry in your backpack. As companies grow these piles of pebbles turn into mountains.

My hope is that if you’ve read this far you’ve gained an appreciation for the true total cost of every line of code you write. If you weigh yourself down the nondescript little pebbles you won’t have the strength to carry the diamonds — those bits of code that really differentiate your business. Every pebble you haul along your company’s journey should be furthering the unique value you can provide to your customers.

I’d love to hear what you think, you can find me on Twitter: @noahmp