Doing enterprise financial data visualization after data journalism

Most of my work experience has had something to do with charts. I spent a couple years as a data journalist at a business magazine, Bloomberg Businessweek, illustrating stories in one way or another. Then I spent a couple years at a financial analytics company, Kensho, working on information software for banks and hedge funds and such. The small tasks and parts were very similar; the big goals and context were very different!

Some of this follows from the users being specialists; some of it follows from that specialty being finance. Specialists make different space-time tradeoffs, in computer science terms; they hold large systems and models in memory, and expect parsing data to take little marginal time, because they see a lot of it. They are more willing to learn a system than to learn interesting and idiosyncratic encodings for every new thing they see. Novel forms repel attention; novel information in familiar forms attracts it.

The work is applied like an ouroboros to itself. A chart captures something salient because we believe it does. Visualizations are fed back into their subjects; the terrain is remade by the maps made of it—as ever, but especially so here. When Emanuel Derman describes the sudden appearance of the volatility smile in options markets after 1987, it is almost as if the smile (the upward curve in the implied volatility of an option as a function of its strike price) appeared because you could see it. Or, as someone else said, “It is the generation by models of a real without origin or reality: a hyperreal. The territory no longer precedes the map, nor does it survive it.” Not that I saw this up close! But it made up much of the background and the horizon.

This is written for someone who, like me, has a background in data journalism and not finance; I’m sure it shows naïveté in both. Each bullet point could be a post, but I had to start with something broad and shallow. The basic headings are classic and essentially conservative truisms: respect the people, the market, the incumbents, the normal forms, the limits of the craft. Insofar as it is too conservative, I hope it helps inform your defiance!

1. Respect the people

• Nobody talks about “finance” in finance. As an engineer or designer, it is natural to start off asking domain experts things like, “You work in finance? In your experience, how do y’all do this or that, in finance?” But this is so vague as to be unanswerable. What holds in one role may be insulting in another. Your eyes adjust to the light and you see the contrasts that remain; “finance” disappears and other terms appear. Investors vs. speculators: the former want to own the thing; the latter want to resell it. Fundamental analysis vs. technical analysis: the former looks at quarterly reports (balance sheets, cash flows, income statements) to decide what the price should be; the latter looks at past prices to predict what the price will be. Buy-side vs. sell-side: the former have skin in the game, because they’ll own the thing and have the exposure; the latter are doing something more like media, content creation, idea generation. Equities vs. fixed income: the latter has another dimension to work with (tenor), which produces curves and spreads and topologies that don’t exist in equities. And I can’t even begin to talk about options. Or, “Delta One” vs., uh, other deltas: I don’t know, but it seems like almost nobody big is ever just buying and selling shares of a stock; there’s almost always some “structuring”, and it’s rarely handled well by outsiders’ software. These distinctions are reified in banks’ desks; they are so few and big that their org charts dictate the ontology of the whole domain.
• Nobody’s a tourist. This is the space-time computational complexity thing. Compared to news readers, there are few dilettantes or spectators; the same people are looking at the same forms over and over; they are looking at more cells, more times. So you overcome all the constant upfront parsing costs and care more about the incremental variable costs and asymptotic behavior for large n. It’s as if your only readers were regular readers, or perhaps more like a trade publication. (Once you recalibrate your terminology to the new context, you can once again discern a spectrum of tourists-to-specialists within any field. To a longtime employee of a company, an investor buying shares is very much a tourist.)
• Sharing content is an adversarial game. It’s much more (explicitly) fraught than in news; the participants want different things. Unlike a friend sending you a story, sharing market-moving information inherently leaks some advantage. Greed (wanting to trade) and vanity (wanting to look good) have something to do with it, but that feels uncharitable and crude; I don’t understand well enough. Sell-side folks may just want interesting trade ideas to call clients and generate transactions; buy-side folks actually need those trades to go in the right direction. The former share content so that your trades are more likely to flow through them than someone else; the latter get paid in rough proportion to how little the view is shared before they trade, and how much it’s shared after.
• Mobile and touch don’t matter. It’s a job; the user is sitting at a PC at a desk. When they’re “on the go” they’re answering emails. The field isn’t “behind” or “slow”, awaiting disruption, as I once thought, fresh from watching mobile and touch disrupt media; rather, the office desk is in the nature of the thing, and it may be pretty close to a reasonably long-term equilibrium. If you’re trying to siphon time and attention, phones are great. And there’s surely room to do better pushing and alerting. But if you’re really actually trying to help people do the bulk of their primary job, they’re at a PC at their desk. (Again, this is relative to working in media, and insofar as mobile does matter, it’s largely in newsy areas.)
• Save more state. The persistence of any customizable view becomes far more important. When you’re following something over time, as opposed to seeing it for the first time, all that matters is what’s changed. If you look at something every day, you want a zoomed-in view you can zoom out; if you’re looking for the first time, you want a zoomed-out view you can zoom in. (@topherchris once pointed out that regular blog readers experience the blog chronologically, but first-time readers read it in reverse.)
• People will tolerate a lot of pain if it works. My friend RJ Assaly likes to say, “If you gave someone a shock collar, where one shock was ‘buy’ and two shocks was ‘sell’, and you proved to them that that made money, every fucking trader on Wall Street would be wearing a shock collar right now.”

2. Respect the market

• Acquaint yourself with the perspective from which a historical price chart is profoundly useless. For a graphics person, this gets at the heart of fundamentals vs. technicals. For a fundamentals-oriented investor, looking at a historical price chart for something you don’t own is like a backtest of “What if I had bought it then?”, and there’s absolutely nothing you can do about that. It doesn’t say anything at all about the value of the business. Insofar as markets are inefficient, fundamental folks look to the ground truth of the business for clues to its current mis-pricing, whereas technical folks look to its and other instruments’ price histories; charts, as a form, have such a natural bias for the latter that there’s a camp of the latter known as “chartists”, and it’s a dogma your users may reject.
• There is no true price — only a latest “tick”, “quote”, “print”. Names dissolve on inspection, like “everyone knows what money is except economists,” and the same happens with prices. A price that might be unambiguous in a news story might be vague or useless to a trader. (What exchange, time, size order? An index level or an instrument tracking it?) And you can never know the price of anything until it trades, by which time it’s out of date. The user is so sensitive to a map’s relationship to the territory that trying to make the seams invisible destroys trust, whereas showing the seams builds it. Make your assumptions (conversions, correspondences, transformations) explicit!
• The market will rigorously test your consistency. Many instruments have formal relationships to each other that probably aren’t captured in your schemata—relationships that ought to hold, but may be excellent trading opportunities if they don’t. (E.g., an ETF tracking an index.) There are strong incentives for users to spot inconsistencies in your data, to ask if a discrepancy is real or an artifact.
• “The market economy, capitalism, is a discovery process,” as Nick Land said: “It’s inherently epistemological. It’s not just the object of philosophical attention — it has its own inherent philosophical activity.” Many efforts to apply artificial intelligence to markets fail to recognize how much real, general, and special intelligence is already priced-in — and that pricing information in is mostly not invertible to getting it out. (There was a good tweet I’ve lost about how ML predictions sometimes fail to see how a price is already a prediction.)

3. Respect the incumbents

• Bloomberg. There are more things in Bloomberg than are dreamt of in your meetings. This was not the consensus when I worked at Bloomberg, but I now believe the Terminal is incredibly well-designed. Folks reply, “I get that it’s useful, but I don’t think that means it’s well-designed,” and I rejoin: No! It is well-designed in every way that matters, even visually! (Such nice high-contrast easy-to-spot color-coded inputs and affordances! So nice that it stretches rather than reflow critical content off the screen!) I want people to look, and recoil, and then remind themselves that Bloomberg is wiser than that disgust. If you haven’t been in the position it’s built for, it’s a deep-set Chesterton’s fence; only after you’ve understood it can you disagree with it. It reminds me of Dan Wang talking about tacit process knowledge. Every screen testifies to a hard-won understanding of how to do a job, encodes a profound understanding of a niche, and rewards careful study like a finch beak. And it’s so fast!
• Excel. One large hedge fund shows every new young software engineer a particular Excel spreadsheet that makes hundreds of millions of dollars a year as part of their orientation, to beat the programmer’s “Excel isn’t serious” hubris out of them upon arrival. At this level, Excel is not interchangeable with Google Sheets or Apple Numbers or even Excel for Mac. We made engineers install it, got PCs so we could run the real thing, ran VisiCalc in an emulator, put its reference card on the wall, put on a looping YouTube playlist of Martin Shkreli (!) and Joel Spolsky screencasting their spreadsheeting.
• Research. Banks have research departments which write about the world for clients, constituting a world of “paywalled” content where the wall is so high that it isn’t even discussed in the same conversations as “media”. These departments’ exposure and incentives may feel more familiar to someone with a media background, more pundit-like, albeit somewhat more academic.
• Style guides. Customizable chart style comes up a lot; it matters more for some users than others. Users’ stylistic elasticity seems to be a function of how much of the money they’re managing is their own. If the users are managing their own money and only have to convince themselves, style matters less. But if it’s going into research to be sent to people with whom they have delicate relationships, then chart styles are proof of work.
• Outlook. If you are producing information software for financial institutions, then your software’s outputs are upstream of an email, doc, or deck. Archie Tse’s “Why We Are Doing Fewer Interactives” presentation finds a slightly different expression here: people are happy to click things to get to the desired view, but the artifact they get to can’t depend on interactivity to reveal information (like hover tooltips), because it will end up as a screengrab in a message. So folks often want annotation controls, which may be fine to offload to other apps — like, people are gonna circle stuff with Office shape tools.

4. Respect the normal forms

• Relocate information from form to label. In news graphics it is common to encode different series in different ways, such that the form shows properties and relationships visually: one is a change in another, or a part of it, or spatially positioned relative to it; they may overlay, transition, or morph. In fact it may be confusing if you don’t present different series in different ways! Here, that’s usually an anti-pattern; those forms should be normalized (to lines, bars, and scatters) and that information should be conveyed in text labels. Those forms dominate partly because of what’s easy to make in Excel, but relaxing that constraint distracts more often than it helps. Here people tolerate longer text labels, and actually read them, because they need to. Unless a novel form is an overwhelming improvement, I’d focus on novel ways to get to (find, search, merge, transform, produce) familiar forms.
• Factor more stuff out of cells and into headers. Like the greatest common unit and thousands. I’d have said this was always true but still found myself realizing I ought to do it more. But consistently, for better caching of parsing assumptions and asymptotic performance at scale. In news, a little table with “$1b” mixed with “$1m” might be OK; in finance, it’d be a “1,000” and “1”, with “($, millions)” or something in the header.
• You will hear fewer complaints about y-axes not starting at zero. People know the instruments; it’s not misleading. They’re like, “Thanks, but I know the price of oil hasn’t doubled in the last day.” See also. (Maybe this storm has passed.)
• Tables are a thick concept. Compared to charts, I feel like there’s not a table discourse or subculture commensurate with the demands of the craft. Maybe it exists more in the data science subculture and I don’t see as much of it. But damn, the tables you’ll see! Layers of nesting, date join logic, inspecting provenance, conditional formatting, bitemporal modeling, checksums. The mere fact that a heavier line denotes summation of the rows above could be a rich subject for anthropology.
• Most things should be a table, not a chart. Here I must say to my past self: Yes, Hestenes said “Geometry without algebra is dumb! Algebra without geometry is blind!”, but no, that doesn’t mean every table needs a complementary chart. Besides, tables are visual too, lol! (But I remain committed to more parity here, to charts as peers.)
• Technical analysis people seem to prefer “-12.34” for negatives; fundamental analysis people seem to prefer “(12.34)”. The former is more common for stock moves, and the latter on balance sheets. Some people say the negative sign is cleaner, and others say parentheses jump out more. If your userbase spans those constituencies, then this is a good example of where no amount of thoughtful good taste can substitute for configurable options: either without the option for the other will infuriate people.
• Right-align your numbers and use tabular numerals. Ensure prefixes, postfixes, infixes, etc. (dollars signs, commas, parentheses) don’t break the alignment of place values. This is no different from journalism, but its importance increases with scale.
• Daily returns charts are great. Like this. It looks noisy until you realize you’re looking for big moves and can easily see them there in their original context (as opposed to cutting them out, sorting them, listing them). It’s a good example of a more full and raw view, where smoothed-out summary statistics and visualizations would destroy important information. The apparent visual noise doesn’t seem to bother the people who care; it works as familiar texture, grounding them.
• Dual axes are sometimes fine. I agree with basically all of this wonderful Lisa Charlotte Rost post on why not to use dual axes — and yet! Some smart, experienced, well-informed, well-intentioned people with skin in the game sometimes really, really want to use them! So I am forced to conclude that they know something we don’t. Like: maybe they are not being misled because they are not looking for what we think they’re looking for. Sometimes they aren’t looking for a statistic or trend or thesis; they are navigating a space. When this happened to this instrument, what happened to that? This isn’t a correlation, it’s: when there was a single big move in one, what did the other do? It’s easier for the eye to leap from one line to another when they’re close. And, for interactive charts, non-normalized values can be more stable under translation, since they don’t continually re-index to the leftmost point’s fluctuations. It reminds me of the appeal of a conformal map in cartography, which keeps shapes recognizable. (Here, angles can change, so I guess I should say “affine”?) Traders seem to know the shape of a dip like lobstermen know the coast of Maine; in a flash they might say “This looks like 1992!”; the mental model demands a recognizable projection. And non-normalized lines preserve references to absolute levels, which are essential landmarks, like “when AAPL was at $100.” And in a world of nonlinearities, a 1% move when rates are low is not the same thing as a 1% move when rates are high. (Lisa also links to this good defense by Peter Ellis.)
• While we’re at it, pie charts are fine too. OK, this is also no different from journalism, and I rarely saw them (it’s more of a PowerPoint thing?), but — what cyclic modular spatial visual encoding can you find to encode your proportions other than angle? (I wish I knew enough abstract algebra to talk about the circle group.) I get that it’s sorta hard to compare across pie charts. But if you see a stacked bar chart, nothing about it signals that it’s out of 100%! No Cartesian encoding is a non-arbitrary 100%! Whereas pie charts need no axis! Everywhere in the universe, 360º ≅ 0º. Pie charts carry their axis with them throughout the universe! That’s very special! Also, absolute rotation exists in a way that absolute position and motion do not, and there’s something sublime in that. If you don’t like it, take it up with the golden compasses prepared in His eternal store! Normal people sometimes like pie charts and they’re right to.

5. Respect the limits of the craft

• Think less about designing the form of the container and more about the choice and labeling of columns and rows and content. A sequence of columns in a table on a screen is like an API for humans: if people are used to a specific set or ordering, take it seriously and don’t break their assumptions. They are processing a lot of these really fast and can’t keep re-reading the headers. The structure of the content is not just “important”, it is the thing! It sometimes seems that designers and users are talking about disjoint conceptual bundles as referents of “table”. Designers sometimes mess with aesthetics and interactions and people literally can’t see it, it’s like something out of The City & the City, it’s totally orthogonal. Users cannot understand or talk about a wireframe or mockup with fake columns. It’s not actually functioning as a prototype. It’s not just like lorem ipsum in the layout — it’s like sending lorem ipsum to the copy desk. It’s vacuous.
• In news, the story is found in the data before publication (by the journalist); in software, it’s found after shipping (by the user). In journalism, to design the visual form before you know the story in the data is to put the cart before the horse. But software is a horseless carriage, lol, and you have to design by inference from specific to general. A good story is often an exception to the rule; if those exceptions are too sparsely strewn about a space you have no control over, then “following the story” results in software that says nothing outside a tiny contrived corner. It’s easy to mock up examples that make a tool look more useful than it is in the wild. So I think the question is: if this is the exception to the rule, would the unaided user have been able to see it on their own? Are the good examples endogenous? Are they viable outside the womb?
• Speed matters more than all of this. A news reader has optionality: they skip the boring stuff. A software user does not: if one part (the front, the back, the API, the interactions, the rendering, the data…) is well-conceived and another part chokes, they are stuck with the conjunction. (Or perhaps: readers and users alike have more horizontal optionality than vertical, and software is a deeper and shakier vertical stack.) Notably, a slow heavy GUI is often worse than some kind of fast minimal or headless solution, which can be a little disheartening for the graphics person. This is compounded on the web; if it’s not all written in highly optimized native C code, then you’d better be saving people a lot of steps!
• “As soon as it’s interactive it’s bullshit.” My friend Sam Shleifer, who worked on the machine learning and data science side, said this to me once. As soon as it’s interactive, you’re letting the user confirm their priors. “Anyone who cares is cross-validating their shit,” which GUIs have historically—perhaps contingently, and not necessarily!—precluded on some level, if only in the meta-analysis. “Doesn’t that make you feel awful?” (See also Bret Victor’s “interactivity considered harmful.”)
• Most things in life don’t need an interactive data visualization. I feel like so many blog posts are like, “Let me tell you how important my job is, and how it’s part of everything.” Of course it’s part of everything, but everything is! I am small and there are many jobs and I love that there are.

6. Do not avert your eyes

For a while in this job I worked out of One World Trade Center. I made a habit of looking up the names on the memorial and read a lot of old news coverage, including some really wild responses I had not heard at the time, as I was 11. Baudrillard said the attacks “respected the symmetry” of the towers, and that whatever was built in their place “would not even be worth destroying”. A composer called the attacks “the biggest work of art there has ever been”. The parent of the company my dad worked for was based in the South Tower; they all got out. The next day, Ward Churchill said some rather harsh things about the supposed innocence of all those people working in those buildings in the finance sector, calling them “little Eichmanns” (after John Zerzan).

I like finance; I find it subtle and compelling, even sublime, transcendent, beyond human comprehension (or maybe just mine). It is cleverer than many criticisms of it. I surely owe to it various properties of the modernity I enjoy. But that doesn’t make it good! Nick Land again (on too many amphetamines, by his own account): “Machinic desire can seem a little inhuman, as it rips up political cultures, deletes traditions, dissolves subjectivities, and hacks through security apparatuses, tracking a soulless tropism to zero control. This is because what appears to humanity as the history of capitalism is an invasion from the future by an artificial intelligent space that must assemble itself entirely from its enemy’s resources.” (Whew!)

Obviously I do not know what that means, nor how to apply it to my work. Just, feebly: the work is very abstract, but there is something concrete underneath, and I think it’s still breathing!

This is just going for a sampler of a baseline. It doesn’t cover money, tooling, teams, etc. It’s mostly stuff I failed at. Doing great work may mean rejecting or violating any of these; in fact, it might require it. And if this is not your context, don’t take these as lessons — just reports from a distant land by which to illuminate your own.

This is only here so I can set it as the preview image