Judging VC Skill: The Hardest Open Problem in Finance?

Byrne Hobart
Feb 18 · 12 min read

One life skill working in finance teaches people is how to act disappointed, even insulted, at being offered an absurd amount of money for one year’s work. Every bonus season, analysts throughout the industry psych themselves up to be performatively bummed that The Number wasn’t even higher. This is not just about greed; it’s also a cultural performance that gives you camaraderie and job security. “If you’re not fighting me over thousands of dollars,” you boss might wonder, “are you really going to fight for me over millions?”

A negotiation like that doesn’t just happen because two people are fighting over a finite bonus pool. It also happens because nobody really knows what they’re worth. And this is in an industry where your only job is to make as much money you can without violating certain constraints (those constraints vary from hard rules like “Don’t break the law” or “Don’t violate your risk limits” to softer constraints like “If our firm’s founder gave a talk at Davos about the importance of beating climate change, don’t make us file a 13-F with a bunch of oil stocks.”)

In theory, it should be easy to figure out what to pay a portfolio manager: give them some capital, give them limits on how they can trade it, and pay them a percentage of what they make compared to some benchmark. Very simple. The benchmark represents beta, and everything not explained by the benchmark is alpha. But “give them capital and pay them a percentage” is an interesting construction. To the PM, it means their compensation is a call option. If your book is $100m and you’re paid 10% of your profits, you basically own a call option on the performance of a $10m portfolio. And everybody knows that you can raise the value of an option by raising either the expected value of the underlying asset or the volatility.

If you hire someone and one of their KPIs is “Surprise me!” then expect to be surprised.

Finance is full of arms races, and one of the fun ones is between a) compensation schemes, and b) clever ways to game these schemes. A smart PM who didn’t want to work too hard might realize that there are some well-known quantitative signals, like momentum and carry, that can be implemented pretty easily and that lead to decent long-term results. Or a PM might engage in factor timing. Whether you’re making the bet directly (as a quant) or indirectly (due to your personality type), if you were systematically betting on momentum and betting against value, you’ve had a good couple years, although you had a rough September. Or if the risk team decides to judge you based on returns compared to volatility, that suggests another easy trade: invest in companies like DaVita or Stamps.com (RIP), where there’s one big regulatory risk that won’t necessarily hit in any given year — or find companies that are growing in a mature industry due to increasingly clever financial engineering and lever-pulling. Everyone knows these will be painfully volatile eventually, but in the short term they’re pretty boring.

Someone who is smart enough to design a perfect, un-gameable compensation scheme has also basically built a good options-trading strategy, so the people who are naturally good at that sort of thing become front-office quants rather than risk managers. The world just abounds in Malthusianisms.

In a sense, passive investing and factor-based quant strategies are responses to the basic problem of compensating investment managers. That problem is: you only want to pay fund managers for alpha, but alpha is hard to produce, and in the aggregate it sums to zero before commissions and taxes. “Beta” used to just refer to the performance of a broad market index like the S&P 500, but now it’s a continuum; something close to the Kolmogorov Complexity of whatever program produces the same investing decisions. So purely statistical value investing is “beta,” since “buy the cheapest 20% of the S&P 500 by price to book value” is about as easy as “buy the S&P 500. A discretionary macro or concentrated stock-picking strategy has to be close to pure alpha, since there’s so much idiosyncratic opinion behind every trade.

In an efficient market for investing talent, the only edge a hedge fund’s LPs have is in identifying superstars early. Over the long term, fees should approximate how much value a manager uniquely adds to a portfolio. And within each fund, analysts should get paid based on how much of that value they added.[1]

And this, remember, is how you design compensation schemes in equities, a liquid market where everything has a price, volatility is easy to measure, and anyone at your firm can check your work.

The game is close to impossible when you’re playing on Easy Mode. Let’s look at what happens when you raise the difficulty level.

Can You Evaluate VC Returns Ex-Post?

One of my favorite blog posts on venture returns is Jerry Neumann’s power laws in venture. His key point is that if venture returns follow a power-law distribution, average returns rise indefinitely as you get a bigger sample set. There is no well-defined mean! If you measure adult height, you quickly converge on 5’9” for American men and 5’4” for American women. You will find outliers, but they’re equally common at both ends of the distribution. But if you measure startup investing returns, you’ll keep getting tripped up: flop, failure, failure, flop, Google, fad, fraud, freaky scandal, Facebook…

Does this imply that the ideal strategy for venture is to invest in as many companies as possible? If you’re sampling from a power-law distribution, that’s what you should do. But can you sample from that distribution? The “distribution” doesn’t really exist, it’s a function that provides a good fit with historical returns data, but those returns were not produced by some scalable, random process.

You can argue that there are two broad approaches to venture, define by which problem is the hard problem. If the problem is maximizing N, you “spray and pray,” and have a widely-diversified portfolio like YC. If you think the problem is ensuring that N consists only of samples from the right distribution, you concentrate.

Once you’re lucky, twice you’re good, also N is usually zero.

This dichotomy means there are two arguments against using any given fund’s returns as a measure of either a) manager skill, or b) the industry’s returns:

  • For diversified funds, any sufficiently bad fund could arguably be not-really-VC; by aiming for a high sample size, they sampled outside of the correct distribution. If you want to invest in five startups, you can probably choose five tech companies. If you invest in five hundred, you’re probably going to end up with “tech-enabled” pizza delivery or something.
  • Any concentrated fund can plausibly argue that they invested in ten different moonshots which each had a 10% chance of a 1,000x return, but they picked the wrong ten. It happens!

Financial Characteristics of a Venture Portfolio

Traditional venture funds invest at multiple stages: Series A through Series N, for any N you want. If you only invest early, you get brutally diluted later on; the people who got into Uber’s first round are happy, but not “I bought 1% of a $70bn company for $50k” happy, because they’ve gotten diluted down to a small fraction of that. For the math to work, a venture fund has to be a single-stock momentum fund; they have to add money in every round, and add more in each round, too.

Since venture success is defined by dealflow, i.e. by whether or not you have a chance to invest in the hottest companies, the main function of the Series A investment is to get a chance to invest in Series B and Series C and so on. Arguably, the better the fund, the more of its real value today consists of pro-rata rights rather than the investments themselves.

That’s a general case of positive convexity: the better the situation, the higher your exposure. There’s another level of convexity, though. If a fund backed a company a long time ago, when the founding team was unknown, they have a lot more data on how good the CEO is at predicting the future and reacting to crises. Many companies get a sonic boom of bad press once they get big enough: Uber in 2019 was finally a big enough company that Uber company parties in 2012 were worth writing about. But board members heard about that stuff earlier, so they know how effective a CEO is at either jettisoning bad employees or sweeping problems under the rug.

This leads to the dynamic where there’s a U-shaped relationship between how long an investor has known about a growth company and how positive they are on it. New investors are just star-struck, and then they learn more and it’s mostly bad news (the people who heard the bad news first were the ones who decided not to invest). But longtime investors know that every company has skeletons in its closet, and they have a sense for how good the company is at either cleaning the closet or keeping it locked shut.

All this adds up to a good explanation for Peter Thiel’s observation that “Whenever a tech startup has a strong up round led by a top tier investor… it is generally still undervalued. The steeper the up round, the greater the undervaluation.”

This artificially smooths out venture returns. Over time, a successful company’s valuation rises faster than its value, because every company is born when a small group of founders and investors realize that something that the consensus views as having a 1% chance of success actually has more like a 10% or 50% chance.[2] Over time, as the scenarios that could hypothetically kill the company either don’t materialize or don’t turn out to be fatal, the uncertainty discount diminishes, so the valuation steadily approaches the intrinsic value. Think of this futurism arbitrage component of venture returns as a carry trade, with the same return distributions. On any given day in 2018, it was more likely that Juul would never get banned and that we’d eventually live in a world where combustible tobacco is widely illegal because vapes are so common and safe. On particular days in 2019, this looked a whole lot less like the most probable outcome.

This skews investor returns in two ways. First, the right deals are enormously different from the almost-right ones. And second, absolute dollar profits are more a function of continuously reinvesting at the right time, not just making the first good call.

Is “VC Alpha” a Meaningful Concept?

Discretionary investors talk about “alpha” as a unitary thing: the intercept on a graph that plots your returns versus the expected returns of having the same exposure to your benchmark. Quants do it a little differently, and talk about alphas, plural: the individual signals that lead to excess return.

This is the right way to do it, if you’re a quant: you have lots of different signals, all of which have to be profitable ignoring transaction costs; then once enough signals fire at once, a given trade is profitable even including those costs. (For example, if it costs you 20 basis points of your position value to put on a trade, and your best signal only produces 10 bps of excess returns per trade, it sounds worthless. But if you trade off of five signals at once and their collective expected return is 30 bps, you’re in business.)

Discretionary investors have alphas-plural, too, they’re just harder to measure. Maybe one smart portfolio manager is good at 1) paying attention to footnotes in SEC filings, 2) reading body language, and 3) not panicking when the market pukes. Perhaps any one of those skills would be good enough for an analyst, but all three are necessary to add value as a PM. The alpha-singular is the sum of several alphas, but unless the PM in question has insane levels of self-awareness and takes extraordinarily good notes, that won’t be obvious.[3]

For venture investors, the main alphas are dealflow and judgment. Typically outsiders overweight judgment (“did you know it was going to be big?”) and underweight dealflow (there are lots of companies that everyone thinks will be big, but only Sequoia gets to say so with a check).

And judgment is easy to misunderstand, too; the hard question is not just whether or not a company could succeed, but whether or not it’s important to keep investing more in later rounds. “Judgement” is also not just one think; growing an investment alongside a company is a different process for every company, and the more successful a company gets the closer the sample space approaches 1. The more successful a venture investment is, the less it’s an example of investing and the more it’s a case study in Uberology, Facebookalytics, or Airbnb Studies.

Since judgment has to develop alongside each investment, it’s necessarily impossible to measure in advance, and hard to measure at the time. Which leaves the other component of VC alpha: sourcing. Y Combinator is the world’s best example of a venture investor that built a comparative advantage in sourcing; YC has become the Silicon Valley equivalent of a BA, and like everything SV produces it’s more efficient and cheaper upfront than the legacy alternative, but sometimes has colossal long-term costs.

But other successful investors constitute a sort of hand-crafted version of YC: they don’t interview every single smart and ambitious person who wants to start a company, but they do get introduced to lots of friends-of-friends. Having an abundant supply of loose ties in a tech-adjacent space is a source of persistent VC alpha — but this, too, is hard to quantify in advance. The best investing strategy is to know everyone in a space before it appears investable, so you get lots of rock-bottom seed and Series A valuations. But that looks suspiciously like a big waste of time. In 2011, Bitcoin was only well-known to crypto nerds, anarchists, and drug users. Not exactly the network that will impress the typical LP.

In a way, all this aligns with an efficient-market view of the world. Excess return in venture comes from either knowing when to double-down (which you can’t measure in advance) or from knowing the right people (which outsiders will tend to mismeasure in advance). There are a handful of companies with a durable, visible advantage at both of these — and they get to pick and choose their LPs, either through invite-only funds or by charging higher fees.

From a strong efficient market hypothesis perspective, “alpha” is an error term; it’s the extent to which reality doesn’t line up with a model. And that, too, is a good baseline: if it were easy to measure investor skill, the market would automatically reflect this, and the skill premium would instantly vanish. The only difference between the difficulty of measuring VC skill and other investor skill is the width of the error bars.


Don’t miss the next story. Sign up for my occasional email newsletter. Or check the About/FAQ page for more.


[1] This model explains a useful stylized fact about making money. If you don’t intend to become super-rich, you will generally make more of your money at companies where the company-wide Gini Coefficient of income is higher. All else being equal, the people who make 1/10th of what their boss makes are doing better than the people who make half what their boss makes. Programmers at Facebook are poorer relative to Zuck than programmers at Accenture are relative to its CEO, but they’re not complaining. This applies across other restaurants, too: if you work at McDonald’s, your average customer might earn 50% more than you do; if you’re a waiter at Daniel, your average customer earns 10x what you do, but you probably do ok.

In the case of a fund, part of the value a fund founder/manager adds is being good enough at sales to raise enough capital to get to breakeven, after which the returns do all the marketing necessary. Eventually, every fund run by a greedy capitalist should expand its assets under management past the point of diminishing marginal returns to investors, because the way you hit those diminishing marginal returns is when your fees on the next $100m in AUM are higher than the value you can add with an additional $100m. There is justice in the world; every hedge fund manager who is overpaid for running a huge fund was once terribly underpaid for running a tiny one.

[2] It’s important to distinguish between the bet on the future and the bet on a company here. Many companies get the future right and fail for other reasons. So when a founder implicitly assumes that their company has one in three odds of becoming a multi-billion dollar firm, it takes the form of “The future will definitely be different in some particular way, and the company that figures this out will certainly create a lot of value, but there’s a nonzero chance that one of our first couple hires will be such a disaster that we tank the entire enterprise and somebody else wins.”

[3] Many big firms invest a lot in PM R&D, trying to reverse-engineer which kinds of companies and scenarios a particular manager excels at. And maybe this is part of what Bridgewater is going for: running a principal component analysis on every employee’s contributions.

Byrne Hobart

Written by

I write about technology (more logos than techne) and economics. Newsletter: https://diff.substack.com/

More From Medium

More from Byrne Hobart

More from Byrne Hobart

The Pirahã Are Off By One

Byrne Hobart
Jan 24 · 4 min read

134

Related reads

Related reads

Related reads

Mind Control

2.5K

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade