Why is Real-Time Market Data so Expensive?
Because you need it, duh.
When I’ve worked on previous apps that used stock market data, I was quickly lost in the complex landscape. I understood so little about the actors in the space that I didn’t even know where to start.
Although I haven’t written a Yahoo Finance scraper myself (which feels like it might be a rite of passage), I think a big reason people choose it is that it is so much simpler than some of the alternatives! It is easy to see how someone might accept a little technical complexity to avoid 5 rounds of sales calls, multiple NDAs and contracts, a one year commitment, and (of course) a giant price tag.
Now that I’m working on yuzu.dev, I’m happy to be in the position to be able to unmask some of the complexity. Like, why on earth is this data so freaking expensive?
I mean, isn't everything supposed to be free on the internet!?
Well, the data is valuable
At the risk of stating the obvious, lots of traders need this data to plan and execute their trades. Whether you’re an advisor, manager, planner, broker, dealer, or a gambler, you need the data to be able to perform your function. It’s pretty hard to win at poker if you can’t see your cards.
The rise of arbitrageurs, high-frequency traders, and algorithmic traders has made the need for (and value of) data even more acute. You might want a lot of historical data for training and testing, or you might want extremely low-latency data to compete to be the fastest player in the market. Either way, the data is not an ingredient you can skip.
And exchanges want to make money
Once upon a time, most US stock exchanges were collectives, organized by their trading members. They’d do quaint things, like meet to trade in coffee shops or under a particular tree. The members agreed to pay a small commission on their trades to fund the collective. As the exchanges have evolved and computerized, they learned to make quite a bit of money along the way. Today, market data is a huge source of revenue for every major exchange.
In 2021, for example, Nasdaq’s “Investment Intelligence” segment, which includes market data and analytics, earned over $1B in revenue. In the same year, NYSE earned $838M in revenue on “Data and Connectivity Services”.
As you can see, popular trading venues are great businesses. First, highly-liquid markets with lots of traders are the best place to trade, granting the biggest venues powerful network effects. Second, the data that this generates along the way, both current and historical quotes and trades, are their own cornered resource. That is, if you’re trying to model and execute trades on one exchange, you need their data, not data from somewhere else. As their network effect strengthens, so too does the value of their proprietary data.
In theory, you’d do almost anything you possibly can to attract more volume to your exchange, offering better terms, lower commissions, and anything you can, knowing that the decrease in revenue from commissions can easily be replaced by other sources, like proprietary data… right? Well, this is exactly how it played out, and exchanges have competed with one another for volume, shrinking commissions and branching out into market data and other segments to expand the total revenue pie.
And don’t forget about the monopoly
Okay, if everyone needs the data, it might be nice to make sure everyone can have the data. Efficient markets blah blah whatever, you get the idea. The SEC agrees and makes it happen.
A brief historical tangent
Let’s turn back the clock and get some context. In 1975, the US passed the “Securities Acts Amendments of 1975”, which created the National Market System. This empowered the SEC to require and regulate distribution of market data by the US exchanges. The SEC strengthened requirements in Regulation NMS in 2005, ultimately leading to a highly regulated and rigid landscape.
These rules impose requirements, both on data providers (like exchanges) and data consumers (like brokerage apps). Exchanges have to report trades and top-of-book quotes to the operators of the consolidated tape, and brokers have to consume the consolidated tapes to route customer orders to the exchange with the best quote.
Consolidated Feeds
The consolidated feeds are the way to fulfill your obligation as a broker and route your orders to the exchange with the best quote, because they display every trade and the best bid and offer across all US exchanges in real-time.
The problem with these feeds is the price. They’re run by government mandated and regulated monopolies, so you could say there isn’t a lot of wiggle room. An operator of a feed is called a Securities Information Processor (or “the SIP”). One, the CTA, is operated by the NYSE, and the other, the UTP, is operated by Nasdaq. Since you need the data from both to comply with Reg NMS as a broker, there isn’t real competition between them.
The basic pricing structure is a fixed fee just to have access to the data, maybe $1.5K/month, and a per-user fee to display the data to others, at $3/person/month. Naturally, the per-user fee adds up quickly.
Exchange Feeds
Thankfully, if you aren’t required to get the full consolidated feed, you can get creative. If you don’t need 100% of trades and quotes, you can get cheaper data that only covers activity on a single exchange or a group of affiliated exchanges. The last sale price from these feeds might be a little off sometimes, but they can be way cheaper (or even free).
Every major US exchange offers one or more products for this purpose. Premium products like Nasdaq Basic ($1/user/month) and Cboe One ($0.25/user/month) combine data from multiple exchanges for more breadth. As you continue down the price ladder, individual exchange feeds, like Cboe’s EDGX ($0.1/user/month) or IEX (no per user fee) are the most affordable.
The details of all of those options are a long tangent that is better served by another post, but if you’d like to learn more, I’m always happy to chat. You can hop in the yuzu.dev discord, or book some time to chat with me here.
