Bringing Wall Street and the US Army to bear on public service delivery.
TL;DR — I spent 7 years doing Technology in Wall Street. I have a #zanyidea about how digital infrastructures that serve public needs can be built.
Two things happened on April 1, 2014 that significantly changed how I viewed my career in finance until that point. Michael Lewis’ new book, Flash Boys was just out and 2 senior currency traders left the firm I worked at, on allegations of wrongdoing and collusion in the foreign exchange markets.
I worked closely with one of those traders and was totally “clueless” on what was going on — borrowing a term from Hugh MacLeod’s cartoon and Venkatesh Rao’s Gervais Principle of Organizational dysfunction.
Amongst other things, I functioned as a liaison between the “quants”, crunching financial data and building algorithmic models, and traders on the trading floor, translating math equations to keystrokes with a shared purpose of making gobs of money.
The quants, mostly based out of London were soft-spoken, with thick English or Irish accents far from their trader counterparts who tended to be loud and boisterous. One of the senior quants had a zen-like penchant of explaining the inner-workings of his team’s algorithms using Lord of the Rings metaphors.
This algo is like Gandalf — it watches, observes and when the price is right, grabs as much as it can get.
This algo is like a Hobbit — it goes around snapping up liquidity at good prices unbenounced to the rest of the market.
…then the New-York based Trading floor, the nerve-center of this vast group-mind where traders of all colors, shapes, sizes, and personalities sit glued to their screens waiting to deploy a Gandalf or a Hobbit into the ring where Trillions of dollars, euros, yen, pesos, and, swiss francs are traded literally at the speed of lightning.
Among this menagerie are salespeople who need to sell the abilities of said Gandalf/Hobbit to institutions or “high net worth” individuals who want to put their gobs of money to work.
Let me also add that many of these traders do not have patience for flowery metaphors.
During this time, my job was to support and maintain a good chunk of the digital applications and infrastructure that powered Foreign Exchange Trading out of New York City. I was part of a team distributed across London, Hong Kong and Mumbai and observed a “follow-the-sun” protocol to sustain the 24/7/365 nature of FX trading. Unlike the more glamorous equity markets where the NYSE/NASDAQ have fixed start and close times, currency is constantly changing hands.
To describe the technology that powers this scene will always be an understatement. When banks and hedge funds are ready to cut through mountains to gain an speed-up of a few milliseconds, the bleeding edge is somewhat, an unsatisfactory term to describe the technology and infrastructure. Big Data, bluntly put is a battered eye-lash or a yawn, at best.
A good chunk of work was to analyze reams of data to prove why my trader’s 150 million EUR/USD trade at 1.2234 did not execute in under 500 milliseconds. In about as much time, I had to unpack and explain Gandalf and Frodo algo to a 6'5" trader who did not WIN in that moment and was counting on me to say that the technology was at fault.
More than half the time, he did not like the answer because the tech was not at fault. Even worse, we both knew that the real reason was that his fingers didn’t move fast enough. There was little he could do about it — a dramatic routine ensues.
150,000,000 Euros for Dollars in <0.5 seconds
Wall street tech doesn’t get talked about much in pop-tech media, yet it keeps gobs of money flowing in and out of NYC and the rest of the world. The only major significant news making tech event was the 2010 Flash Crash and that happened because things worked way too fast!
The Financial Information Exchange (FIX) is one of the few data protocols that power the backbone of global finance and banking. From using your ATMs to converting $ to pesos for your next vacation, to your 401ks ticking (higher or lower) ; Trillions of FIX and SWIFT messages are exchanged between servers and desks in NYC and London, and sometimes Kansas City.
FIX is a feat of information engineering but is also largely self-serving.
In a (ridiculously reductionist) nutshell, here is how FIX came to be.
In the early 1990s, a bunch of big banks knew that a great way to make more money was to, in effect, talk or type faster. The simplest way of doing that was to agree to speak a common language and come to agreements or disagreements really really quickly.
A few savvy banks agreed to do it, started making money and everyone fell in line. Done.
In 2012, The UK government published a document titled “Standards in computer based trading: a review” that provides amazing insight into the origins of FIX amongst other standards in electronic finance.
Selected excerpts
- FIX was created when Fidelity Investments asked the notorious Salomon Brothers to find a way for passing information about “pre trade liquidity” (you can glaze over the terms for now) and trades between their offices in Wall Street and Boston.
- Early implementations were driven by an interest to maintain a sweet commission-earning relationship with Fidelity, and other large asset managers. This helped drive adoption of a common language that would ultimately serve the public interest by increasing competition and reducing commission rates from 25–40 basis points to between and less than one basis point today.
- Whilst in this case the public good benefited from private benefits driving standardisation this is often not the case. Either standards that would benefit the public are not associated with private benefits and so do not get implemented or they are fall foul of industry bickering as each non aligned private benefit argues their corner. This reinforces the point about all stakeholders having a voice in standardisation.
- In Financial Services, as in most business endeavours, there is also a cultural deference in that everyone defers to money, particularly to the payers of large fees. In some cases, this deference can serve the public interest, but in many cases, it does not. Fidelity’s pivotal role in establishing FIX illustrates this.
- FIX is maintained by FIX Protocol Limited, FPL, a not for profit company that manages the specification. FPL is funded by market participants voluntarily becoming members and their membership fees are used to promote the protocol.
- Because FIX is an open protocol that can be used by anyone, FIX suffers from the free rider problem. The free rider problem is that firms do not contribute to something can still benefit from it.
Other interesting origin stories include:
I started my Wall Street career in October, 2008 — just as Lehman Brothers was unravelling and global finance was having an existential crisis of faith.
During that time, Big finance and Insurance were the only sectors willing to sponsor the H1-B work visa on the east coast. Joining the financial industry was more, a function of need than choice.
3 years later, in October 2011, I witnessed the Occupy movement from the perches of the trading floor. The memories during those weeks of protest were surreal.
Trading floors are full of screens. The Screen:Person ratio is at least 7:1 in a work area dominated by the lack of natural light and about 500 high-earning individuals under one roof all united by a common purpose — to make gobs of $$$$ or preserving the ability to make gobs of $$$$.
While CNBC was broadcasting scenes of protesters camping out in Zuccotti Park, we were actively normalizing our views of Bryant Park.
Up until that point, working on Wall St. symbolized to me, a great meritocracy of sorts. The frenzy of the Occupy movement and public radio (NPR)coverage of the financial crisis were incredible learning opportunities to better understand and educate myself of the historical context, I was an enabler of. Over the next 3 years, I began honing in on a simple idea:
What if I could re-purpose my tech skills from the efficient movement of financial data to the efficient movement of public data?
Wall street does technology well because the proverbial carrot is cold hard cash (period). Wall street gets stuff done because they throw gobs of money at tech problems. Gobs of money is a talent-magnet. Problems get solved even if it means the brute-force of passing cables through literal mountains or being petty enough to move your servers, a few feet closer to the market exchange. Any solution to such problems leads to 10x-ing gobs of money.
This is the core script. A core function of this script, I’ve realized is Information Exchange protocols, jargon for being able to share standardized data or speak a common language, at some considerable scale.
Their adoption, use and maintenance for local public services has never been more apparent. The same forces that allow Wall Street, Uber & Airbnb operate and make gobs of $$$ can be deployed for the delivery of public services and preserve gobs of public trust in public institutions.
Consider these examples:
- FIX and SWIFT, as described above power Global Finance.
- Automatic dependent surveillance — broadcast (ADS-B) and Aircraft Situation Display governs Global aviation and air traffic control.
- Marine Traffic uses a globally defined Info. exchange protocols to power Global Trade.
- Local number portability (LNP) allows you to easily port your cell number from one carrier to the next.
- Even the used car market has information exchange protocols. (CarFax, BlueBook).
Most of these protocols were developed in the late ’80s and matured in the 90s to facilitate the friction-less and inter-operable movement of data that facilitated communications and locomotion. These protocols form the digital bedrock to reduce the marginal cost of delivering data of economic value.
What exists for Public services then? Consider Energy, Transportation, Healthcare, and Water Management:
For Energy
The Open Access Same-Time Information System (OATI), a product of the Energy Policy Act of 1992 that led to, amongst other things, ENRON.
ENRON treated the flow of energy the same way Goldman Sachs treats the flow of money.
ENRON was great at developing applications using OATI and made billions while also woefully subverting public infrastructure and millions of Californians in the process.
The or California Independent Systems Operator (CAISO), effectively the New York Stock Exchange equivalent to maintaining California’s energy marketplace. took over when ENRON imploded.
CAISO currently utilizes the OATI protocol.
CAISO is a non-profit.
For Healthcare
The Electronic Medical Record (EMR) and Health Information Exchanges aspire to offer unified information standards and reduce the marginal cost of exchanging patient data for better health outcomes. Many local Health Information Exchanges are setup as non-profits and work closely with their state and local governments to adopt unified information exchange protocols.
Vermont Information Technology Leaders (VITL) is a canonical example of a health information exchange.
For Transportation
Google brought us the General Transit Feed Specification (GTFS) which happened to be a side project and many consider to have empowered various open data and civic tech movements.
Autonomous Vehicles and automated transportation are happening right in front of our eyes. The cars, computers, and connectivity industries are effectively dancing around what could be a Heaven or Hell scenario for Urban Transportation.
A clear role and opportunity to get AVs working in the public interest is to operationalize digital protocols around moving people and things from point A to point B, autonomously.
Today Uber, Lyft and bunch of Transportation Network Companies (TNCs) do this individually for their own businesses.
ARGO’s efforts to operationalize a digital metric for citywide street quality include developing open-source algorithms to routing a vehicle across ALL streets in a city.
Vehicle-2-Vehicle (V2V) and Vehicle-2-Infrastructure (V2I) are examples of this in progress but are not keeping up with the current pace of innovation in urban and regional mobility solutions.
…and finally Water.
Interestingly, back in the early ’00s, ENRON executives planned to “ASK WHY?” for water markets and created Azurix to begin privatizing global water markets. Things went thankfully south for them else Frank Herbert’s Dune would not be just another cliquey sci-fi fiction novel.
Also during the California Energy Crisis of the early ‘90s as James Sweeney writes, California’s Department of Water Resources (DWR) was ordered by Governor Gray Davis to effectively bailout California Energy by “purchasing wholesale electricity on behalf of the electric utilities making the State the dominant purchaser of electricity in what should have been a marketplace.”
There does exist an “Open and Transparent” opportunity for California’s Water Data via a recent piece of legislation calling for the creation of an Integrated and Open Water Data platform that amongst other benefits, promotes “openness and interoperability of water data” and “making information accessible, discoverable, and usable by the public can foster entrepreneurship, innovation, and scientific discovery.”
Projects like the Open Water Rate Specification (OWRS) are just being started that would enable water stakeholders exchange data seamlessly and reduce the marginal cost of sharing and delivering water data.
Where glossy investment banks got together and leveraged advances in technology to benefit themselves, local water utilities are starting to do the same to benefit California’s water system.
A small team of purpose-driven public technologists, leveraging advances in low-cost device, data and decision-making and the right kind of support is all it takes to build and maintain public, digital infrastructures.
The coordination and technology talent needed to pull these feats of public information engineering are, in spirit, similar to constructing the bridges, highways that we all pay for via our taxes.
The eventual answer may lie in a digital equivalent to the Army Corps of Engineers . To build, operate and maintain the next wave of digital infrastructures to power 21st century public service delivery.
