One of my favorite podcasts is “The Pitch” — a weekly show where founders looking to raise money pitch their companies to a group of investors. While the turn of events on the show can be unpredictable, one thing is guaranteed: if it’s a B2C play, investors will ask about customer acquisition cost. And if the founders don’t have an answer, the momentum gets sucked out of the air.

On a much larger stage, Uber has been criticized for not disclosing how much they pay to acquire drivers, riders, couriers, and eaters in their S1.

Surely, any serious company must know these numbers. You’d be crazy to invest in any company that cannot tell you the answer, right?

But here’s the deal: I don’t think any company knows exactly what it costs to acquire a new customer. Moreover, asking for a single number to quantify this is way too simplistic for such a multi-dimensional problem.

To see why, let’s start by digging into the prevailing metric when it comes to evaluating customer acquisition cost.

CAC is not CAC

When people talk about the cost of acquiring customers, they’re typically referring to CAC, which stands for customer acquisition cost. For a given period, it’s defined like this:

CAC = $$ spent on acquisition / acquired customers

Or maybe they’re referring to the more advanced version (let’s call it CAC2) where the denominator only includes customers that clicked on an ad prior to signing up and the numerator only includes spend from trackable channels (here “trackable channels” typically means digital marketing, while offline marketing conveniently gets labeled as “brand spend”).

Both metrics are easy to calculate and they’re valid in their own right – for example, it’s always bad if CAC or CAC2 increase and it makes sense to monitor that.

But none of these metrics actually tell us how much it costs to acquire a new customer.

Let’s start with CAC. This is not the cost of acquiring a new customer! This is just marketing spend divided by acquisition. Simply dividing two aggregated numbers does not imply causality or even correlation. In fact, this metric is only valid if acquisition is completely non-existent in the absence of marketing spend.

CAC2 is only slightly better. First of all, it only works for ads that can be tracked through clicks and URLs . But even then, click-based attribution is just plain bad; you just can’t give a marketing ad 100% credit just because someone clicked on it.

Consider the following (admittedly) extreme example:

• A company spends $1M on marketing in a quarter and acquires 10k new customers.
• Everyone knows about the company. It’s been all the rage for some time! 100% of new customers joined because because of self-selection and word-of-mouth.

Let’s so the math: CAC is $1M / 10k = $100. According to this metric, we paid $100 on average to acquire a customer.

But in reality we paid $0 on average to acquire a customer and we wasted $1M on marketing. In other words, CAC is not CAC.

OK, so what should we really be looking at?

First of all, we should think of customer acquisition cost as a known unknown. This is a complex problem and we need to treat it as such.

Here’s my advice: if you’re spending non-trivial amounts of money on marketing, you need to hire data scientists and performance marketers who are fully dedicated to measurement of marketing spend and experimentation.

Here’s why: Rather than relying on overly simplifying metrics, companies need to understand the cost curve for every marketing lever:

A cost curve tells us what the true organic inflow of customers is — i.e., the level of acquisition we’ll get if we don’t spend any money on a given marketing lever. This allows us to exclude the organic inflow average cost calculations. More importantly, it also tells us what the marginal cost of acquisition is — i.e., much we have to spend to acquire future cohorts of customers.

And, of course, these curves are agnostic to click-thru attribution.

But how do we do this?

We’ll never know the ground truth. But rather than run with simplistic and potentially misleading metrics, the goal should be to come up with the best approximations and continuously refine the results.

The only way to approximate these curves is through continuous experimentation — i.e., randomized A/B tests and matched market testing —along with statistical modeling.

Experimentation is the only way to estimate how much marketing is affecting acquisition above and beyond the natural inflow of new customers. In order to isolate the impact of marketing, we need to run experiments where the “control group” is exposed to business-as-usual marketing and the “test group” gets no marketing. Experimentation is also the only way to measure the marginal impact of incremental spending,

These tests can be hard to execute, and let’s not forget the synergy between these marketing levers. As I said, this is complex.

Putting it all together

Let’s rename CAC to what it is: spend divided by acquisition. I’m not saying we should abolish this metric, but let’s not pretend it’s something that it’s not.

Let’s stop treating click-based attribution as gospel. An ad should not necessarily get full credit just because someone clicked on it prior to conversion.

Let’s realize that acquisition cost is a known unknown that needs to be learned through a scientific approach, over time. This is hard, but those who are most informed here have a competitive advantage when it comes to growth. And those who rely solely on CAC and CAC2 might be wasting money or leaving opportunities on the table.

And, last but not least, let’s ask the right questions when it comes to customer acquisition cost. This is a multi-dimensional problem and we should ask a wide array of questions, such as:

• What experiments have you run, and across which levers?
• How does the marginal cost compare to the average? Are you saturating your audiences?
• Do the marginal costs vary across levers — i.e., will re-allocation of money provide a lift in acquisition?

Surely, spend divided by acquisition does not answer these questions.

For more details on experimentation and modeling for acquisition, see here.

📝 Read this story later in Journal.

👩‍💻 Wake up every Sunday morning to the week’s most noteworthy stories in Tech waiting in your inbox. Read the Noteworthy in Tech newsletter.