Our goal was to visualize the Google search interest around the German elections end September 2017.

Over the course of the project, we launched a lot of different smaller and bigger visualizations, from daily to yearly views of the top searched terms for the candidates on our project site 2Q17.de over embedded widgets on external sites to special interfaces for live events and debates and images and movies for social media.

→ Find a good overview of all the products we produced here.

After previously discussing our design process in detail, we now want to talk about what you can’t see in the finished project: all the various decisions that went into the actual implementation.

2Q17’s main site is based on a relatively simple feed-forward mechanism: we get data from Google Trends, turn them into our own format and store it in the backend, from which the frontend can grab and display them.

The frontend itself is a static React + mobx-based website built with webpack.

Yet, the devil is — as always — in the details. So let’s dive right in!

Backend

The bigger the project, the simpler you want to keep individual aspects. We’ve tried to stick to this premise when it came to the backend:

Starting out, we got daily snapshots of the data in its raw form directly from Google Trends (and sorry, there’s no public API available for obvious reasons). Since this data still needed some treatment (see “Text data needs gardening” section in the previous Medium post), we had a dedicated Linux virtual machine set up on the Google Compute Engine that ran various python scripts daily to get the data into app-compatible form. The resulting files would land on Google Cloud Storage and be directly accessed from the frontend which itself was served from there.

One of the nice surprises was how well Google Cloud Storage performed as a super-simple webhost. By tweaking the DNS settings right you can use it to very easily host a static website. And the performance is great: even once tens of thousands of people hit the site after the Spiegel Online timeline article went live, there was no noticeable slow down. We ended up hosting everything there, from the data to the site itself and the embeds.

Again in the spirit of keeping things simple we early-on decided against using a database. While databases are useful for manipulating data or retrieving very specific aspects of a data set, we had neither of these things: our backend scripts would create TSVs and the frontend would read them whole and display them.

Even when the complexity of the data grew — from daily snapshots, to weekly and monthly versions, and finally to four hours-old “real-time” data sets — we were still fine.

There’s two types of data files we’re working with: the “latest” versions, which shows the latest daily- or 4h-data, and the “archive” versions, which contain data for every day, week and month in 2017 (up until the election).

Everything is stored as files on Cloud Storage and can be retrieved with a simple fetch operation. While the site loads the latest data automatically, changing the timeframe from days to something else triggers these on-demand requests. Similarly, fixed embeds with a specific date also load one of the archive data. Fortunately, these additional requests are pretty fast, so that changing weeks or months only results in a very subtle delay.

Code support during the design process

When it comes to working with data, we’re always facing the (potential) break between ideas and data. Once you put the actual data into your designs, lofty visions might shrivel quickly. That’s also what makes datavis design closer to designing games than static graphics: it’s about creating rules and seeing what pictures the data draws with them. And in the end, the data always wins.

That is also why code support and quick prototyping is critical during the initial designs — to create shorter feedback loops between ideas and the resulting data-driven graphics. But code can also work as a playground where it becomes easier and faster to explore new ideas.

Working with a much-maligned type of chart definitely increases the motivation to getting it right. That’s why we set up a dedicated prototyping environment just to play around with various types of word clouds and their animations. While word clouds can be usually found as static graphics (thanks Wordle!), they’re arguably much more expressive in an animated form. In our case, the frequent switching between sets of words (e.g., from one candidate to the next or one day to the other) just had to be supported with suitable animations. But that opened up various questions:

Where do the search terms come from? Where do they go? How can we differentiate between search terms that stay on the stage and the new or leaving ones?

Thanks to our custom prototyping environment that let us play around with different layouts and animations in short pieces of code we could relatively quickly explore different ideas and visual metaphors:

This environment took care of layout-independent tasks (mainly transitioning between terms and rendering them) and let us quickly churn out variations manipulating every aspect of the DOM we could think of.

After implementing various ideas, we settled on an internal rule set for those transitions. The final word cloud show the taken decisions and how those transitions can support the visualization:

2Q17.de

Similarly to the last few projects we did, 2Q17’s frontend is built on a combination of React + mobx.

One common problem for complex web applications is how to manage the internal state. Javascript is flexible enough to easily create spaghetti code of the highest order (it definitely feels easier to produce than the properly organized variety).

So, using some state management library can help in constraining the code from delving into pasta-territory. While mobx is less known than its ideological siblings Redux und Flux, it has similar ideas: there’s a centralized state that controls how the application looks. Interaction, new data, etc. change the state which is reflected in a re-rendered application.

Mobx is a lot less dogmatic when it comes to the actual implementation, though, (that’s what mobx-state-tree is for) which it gives it a nice, low-impact feel. It basically provides you with ways to create observable values, automatically computes other values based on them and re-renders React components when they’re affected by value changes. It also works great with React, which anyway favors a “pure function” approach, where components themselves should have as little state as possible and be just dumb containers for feeding from a global state.

In the 2Q17 case, our overall architecture is based on two main mobx “pipelines”: one for DATA, one for STATE. The data pipeline keeps track of data loading and processing (deriving values from the TSV files), while the state manages changes through interaction: selecting a different day or candidate or clicking on one of the pulse bubbles.

Both pipelines are merged in the dataAPI class, which creates props for the React components based on info from both. For example, dataAPI’s “wordcloudTerms” function filters the wordcloud data from the dataStore according to the currently selected candidate and timeframe in uiState and spits out an easily digestible array as prop for the React WordcloudComponent. If the values in uiState change due to clicks or taps, the wordcloudTerms function is re-evaluated and the WordcloudComponent is re-rendered with an updated array.

Mobx automatically keeps track of all variables that a given React component relies on and through this dark magic, all changes in either of the pipelines automatically re-render affected components. So, everything a component sees from a state change is that their respective props have changed. Sticking with the WordcloudComponent as example, it is initially rendered with just an empty array for its “terms” prop which causes it to stay blank. Once the data pipeline is done loading the relevant TSV-files and the dataAPI with processing them into a WordCloud-compatible format, mobx triggers a re-render on the component with the new — now filled — “terms” prop. This results in the new terms floating into view. Further changes to the terms (e.g., by switching candidates), trigger more re-renders with new data and so on.

This automated re-rendering really is the most convenient aspect of mobx, since developers no longer have to keep track of which components are affected by a certain state change. We even went so far as to make as many components as possible stateless (easier to debug, easier to re-use, …), to take advantage of the centralized mobx state.

Performance

Facebook’s React framework is great for building complex web applications since it encourages modularity (Components) and enables a very convenient fire-and-forget rendering approach with its virtual DOM.

In reality though, there are still some pitfalls when it comes to performance. While building a simple, non-animated website with React is very easy, having something that performs well even with hundreds of animated elements floating around is decidedly harder.

Without going too much into details: since React is relatively naive when it comes to re-rendering components, shouldComponentUpdate is a must to tell it explicitly when a re-render is necessary. By defining this function in a component, you as a developer can decide when the component should re-render. This can be crucial when hundreds of components would be re-rendered each frame even though nothing (visible) has actually changed.

Along those lines, minimizing the render functions by splitting everything up into tinier and tinier components might seem petty, but does wonders for the performance (especially when being very decisive in re-rendering them). When you have a look at our word clouds, even single terms are actually two nested React components, to separate the position- and scaling-animations (which are cheap CSS-transforms) from the more costly color- and font-animations:

While working on performance, you sometimes also learn something new about tried-and-true approaches:

Everything usually gets faster when work can be split into separate threads. Since this is only possible through Web Workers in Javascript, at some point I created a version of the word cloud that performed the d3-force calculations in a web worker. But unfortunately, moving the data for hundreds of nodes between the background and foreground threads once a frame actually caused more overhead than performance gain (despite JSON.stringifying everything). So I ended up dumping this approach, with hopes for SharedArrayBuffers in future browsers.

Talking about tried-and-true approaches: one common thing I like to do when trying to make things fast is switching from the DOM/SVG to either Canvas or even WebGL. With browsers becoming better and faster, however, this year it actually made sense to stick with the DOM:

In the mobile word cloud, tags are entering from the left and exiting on the right side of the screen. And if you look closely, you can see that the tags are initially even outside of their container and twisted a bit, thanks to some rotateY- and perspective-CSS magic. These effects would be very hard to recreate with Canvas or WebGL, so we decided against dropping the DOM.

What helped us with the performance in this regard was that word clouds become cluttered and hard to read with less space. That’s why the number of terms (and thus the number of costly to render elements) drops with screen width. You can try it yourself (and feel like a web developer) by resizing the window — the smaller the window gets, the more terms leave the stage and the less new ones get introduced.

Wrapping up

If somebody would have shown us the eventual extent of the 2Q17 project when starting out — who knows if we would have actually done it! Dozens of different data visualizations working across phones, tablets and desktops, varying timeframes with close to real-time data, and half a year of sometimes quite intense efforts led to a project that — looking back — became quite grand in its ambition.

Code was a helpful and sometimes frustrating companion along the way. By keeping things as simple as possible (skipping a complex backend, relying on React + mobx for a straightforward frontend architecture), we could postpone some of the inevitable complexity towards the end. Similarly, focusing optimization efforts only on parts of the code that were never going to change or never being re-used, eased the workload and made us throw away less code.

And so in the end, an abundance of discussions and experiments on both the design- and development-sides let us explore our very varied data source in detail and learn almost more than we ever wanted to know about Germany’s interest in their political candidates.

This article is a joint production by Dominikus Baur, Christian Laesser and Moritz Stefaner.

If you enjoyed this write-up, make sure to check out also:

• Christian’s Behind the scenes article
• Moritz’s reflections on lessons learned and the portfolio page at http://truth-and-beauty.net/projects/wahl-2q17
• http://2Q17.de for an archived version of the site.

I’m currently thinking (and reading) a lot about the implications of a data-driven society. Data collection is taking a foothold in our daily lives, through social media but even more so through the internet of things. All of a sudden we’re split into two beings: our good old physical selves and outsourced digital versions in the cloud.

Pundits like to emphasize the benefits of big data and algorithms to all of us (I assume with economic motivations). We’ll get the end of theory and become better parents, friends and employees. These assurances usually leave the rest of us with a feeling of dread. The big data specter hangs above us all and we’re in fear of old drinking photos showing up in job talks or having our lives ruined by one stupid tweet. Also, we’re facing a new data apartheid, with powerful data lords on the one side and us sheepish content generators on the other.

But what’s interesting for our daily lives is that big data probably means the end of everyday cheating.

In the data-less days of yore we’ve learned to get away with various little transgressions: crossing a red traffic light at night, lying to your boss about the tedious report or secretly having a smoke hidden from your spouse and doctors. All small things without consequences. Till now.

Since everything will be sensorized, it will get substantially harder to create a favorable virtual identity. Various biosensors collect how little we actually move, location data shows how much time we spend at our favorite bar and our financial histories make it already hard to get a credit after wrongdoings. And with smart everything (cities, cars, fridges) nothing will be secret any more. Never-tiring algorithms will catch every outlier in the data.

All of a sudden everyday cheating will be futile. The city captured your car’s ID when crossing that red light, your boss knows that you’d spent three hours on Facebook instead of doing the report and your wearable air quality sensor rats you out to your spouse and doctors.

On the plus side, these micro big brothers could also mean an end to privilege. All charms and good looks won’t help you land a job when the frightfully neutral algorithm deems you unworthy. Talking yourself out of a parking ticket becomes impossible with a machine. Knowing the right folks becomes useless when the data’s against you.

Along the same lines I predict a rise in data tweaking: companies that specialize in adjusting your data to make it more positive by hiding your wrongdoings and creating fictional noise to fool the algorithms. It’s the crypto wars of hackers against security companies all over again just this time about data integrity.

But the end of everyday cheating has a much more important implication for our society’s integrity: we have to reevaluate laws and rules that at the moment only “work” because people can break them without consequence. Maybe that traffic light at that crossing really isn’t necessary, maybe sleeping in means more productivity, maybe New Yorkers are actually quite capable of jaywalking without killing themselves.

In the end, it might make us at least more honest about ourselves.

Originally published at do.minik.us.

If you like this article, please ❤/👏 or share it! For more like this, follow me on Medium or follow me on Twitter for general datavis ranting.

About me

I’m Dr. Dominikus Baur, an award-winning datavis designer and developer. You can find the projects I’m most proud of and more on my website: https://do.minik.us.

You have a fascinating project to work on? You want to turn these ideas into reality? Let me know!

Every time I read some article on Augmented Reality or see it pop up in a movie/TV show, it looks like this:

Nothing screams future like neon colors and flashy animations. And I don’t excuse myself for that: even in my last article on how to bring data visualization to AR, my examples were high in the neon department to make it obvious at first glance that This Is The Future™:

It reminds me somewhat of ‘90s web design, back when blinking text and construction GIFs were all the rage.

Now, while this type of augmentation might be clearly visible and extremely flexible (pack all your data into whatever representation you want), it is also highly distracting:

• Primary colors and animations are geared towards catching our attention. Actually focusing on tasks at hand might become hard in such an environment (imagine reading a scientific paper in a casino).
• Virtual objects overlap physical objects. Which might be clearly a problem when the hidden physical object is a car speeding towards you, but can also be annoying in more benign situations, like searching for your car keys that are hiding behind a virtual bar chart.

So what to do about these issues?

The whole premise of this article and the one before it was that near-future AR setups couple see-through-glasses with automatic object recognition. Which also means that these systems can shape your reality in any way they want.

They don’t have to display neon-blue blinking squares but can create whatever they want and overlay it on your reality.

How about using these powers of augmentation to create silent augmented reality? Augmentation that helps you, but becomes (mostly) seamless, blended into reality. No more constant overwhelm with animations and garish colors, but additional, useful, non-distracting information.

Realistic virtual objects

While it might seem required to reach Cyberpunk cred, Augmented Reality doesn’t have to display anything neon.

Computer graphics have come far enough in the last decades to generate seemingly realistic objects and use them to represent data.

While we could, of course, just throw a neon-blue bar chart into the reader’s face, we could also use their current surroundings for inspiration to show something like this:

Imagine those rocks being created by the AR engine and only visible through glasses, which means they’re just as unreal as any of the neon AR we’ve seen before. Still, they’re fitting much better with their environment, creating none of the jarring reality breaks. And they’re just as good at their job of displaying data.

If the AR system is aware of a person’s surroundings it can react with suitable virtual objects. There’s massive libraries on the web filled with realistic 3D models of rocks, seashells, books, small animals etc etc. — all suitable objects to be used for visualization. Combine those with some realistic looking shadows et voilà. And yes, there are even 3D models of pies to be used for you-know-which-chart.

To uphold the illusion of reality, these virtual objects should appear real at first glance, which makes shadows, occlusion, and perspective important. But other aspects of their behavior also have to be tightly controlled: your beach rock bar chart can’t just appear out of thin air. The virtual rocks might either drop from the sky — or maybe less irritating — slowly grow out of the ground. Physics engines for video games seem like a great fit for these problems.

If we loosen our rules a bit and allow displays that consist of seemingly real objects but are definitely nothing that you would see when you take your glasses off, other interesting things become possible.

Virtual planes demonstrate the business of an airport over the day, just as virtual birds (in conjunction with their originators) show their flight paths.

While these examples were constructed through photo manipulation (either by directly stitching together photos of planes or combining video frames with a filter), one could imagine them happening in real-time and augmented reality.

But we don’t have to stop at adding realistic virtual objects. AR can just as well be used to distort and manipulate the real world itself.

Reality distortion

Given that our Augmented Reality system can recognize real-world objects and create arbitrary graphics to overlay on top of them, something even wilder becomes possible:

Augmented reality is not restricted to creating virtual objects situated in the real world. It can also manipulate real-world objects to make them (seemingly) shrink, grow or even disappear.

Maybe you have an ugly dent in your car: since your AR glasses track where you’re looking, they could display an overlay of perfectly shiny car paint overlaying the dent, thus making it disappear. Similarly, objects such as the dirty laundry on your bedroom floor could be hidden by overlaying a texture of the clean floor on top of it.

And once these real-world objects are (apparently) gone, the system can generate a new representation of them — manipulated in size, shape, color etc. — and display it on top of the (hidden) object to encode data.

In case you want to declare me completely crazy now, watch this short demo by Laan Labs, running on an unmodified iPhone using the new ARKit:

And this is just the current state of this trick. Imagine what it will look like in ten years time.

This reality distortion requires object recognition as well as the ability to fill in occluded (and thus invisible) backgrounds. All things that even today’s image manipulation software is already able to do. Think Photoshop’s Content-Aware fill on your nose:

Once we have such a system capable of recognizing physical objects and manipulating our visual perception of them, we can create data visualizations without garish overlays or virtual objects: the actual world becomes our building material to map information to.

When we look at the basics of data visualization, there’s only a handful of so-called visual variables that let us encode information. They’re all based on our visual perception and which attributes of objects we can distinguish, such as position, size, brightness, area, color, etc.

The concept of visual variables also makes it possible to take apart every type of chart into these basic building blocks: a scatterplot encodes data as horizontal and vertical position. We can encode an additional attribute for each item as size, thus turning it into a bubble chart and so on.

The process of data visualization is at its most basic just mapping numbers from the data to these visual variables.

With the idea of visual variables and a powerful reality distortion AR, we can create data visualizations from everyday objects:

Color + Brightness + Texture

These are some obvious ones — just recolor real-world objects to encode data. The plant on your window sill which you’ve ignored for days might already show hints of becoming brown, but AR datavis can really emphasize its sorry state, hopefully spurring you into action.

To improve its chances, the AR could also enhance its brightness, making it stand out more among all the other objects by your window.

Finally, AR could also change its texture, making it look more stripey or spiky, whatever might grab your attention —maybe even displaying snake-like animations slowly flowing along the plant as a last resort.

All these visual manipulations do not require getting rid of the actual object by hiding it behind some simulated background. It’s enough to create an overlay in the right size and shape on top of the actual object. The data we visualize (e.g., the dryness of the plant’s soil) determines the amount of color, brightness or texture.

To make the most of a data visualization, you’d probably also want to compare multiple objects — maybe see which of your plants actually needs water the most — something which should work well even with just digitally changing their color, brightness or texture.

Size + Orientation + Shape

These visual variables are where we’re moving from subtle augmentations to more drastic ones. By hiding the actual objects and distorting a virtual representation of them, AR can modify an object’s size, orientation or shape for data visualization.

Visualizing data in a supermarket doesn’t have to happen with a heat map as in the example above. The objects themselves could also be visually distorted to reflect data.

Maybe the neat rows of products become bar charts themselves, by distorting the height of a stack of butter.

They could also shrink in size to represent how far they had to travel to get there, borrowing a perspective metaphor (and making selecting local products much easier).

And in addition to that, they could also slightly rotate or even change their shape to encode information — be it nutritional or about their prize.

Position

The most powerful and easy to read visual variable — position — is the one which should probably be used the most sparingly in AR datavis.

All of the above reality distortions make interaction with those (actual, physical) objects already somewhat harder. It becomes difficult to take a product out of the supermarket shelf if its size or orientation has been manipulated.

But what’s arguably worse is having these products floating around in space, or moving unpredictably across the shelf. What helps with readability in a datavis chart might not necessarily help with interacting with the real world.

Combine that with an additional array of the virtual realistic objects from above, and we’re deep in confusion country.

Reality Hacking

What all of these ideas have in common is that they’re messing with the basics of our perception. Things we’ve learned since our earliest days — that most of what we see is real, that objects can be grabbed and interacted with — no longer applies, when half of the objects are virtually generated and the other half is thoroughly distorted.

These extended Augmented Reality techniques are reality hacking. Our own version of reality becomes not only machine readable but also writable — with all consequences. Which should lead designers and technologists to tread very very carefully and ask themselves the right questions before every decision.

Being the technology optimist that I am, I hope that early reflections such as this one can push us and our technology to a good outcome. The more we reflect now on the implications of this technology, the less mistakes we’ll make once AR glasses with perfect object recognition and manipulation are on everyone’s noses.

So that our AR future may look more like this:

than this:

If you like this article, please ❤/👏 or share it! For more like this, follow me on Medium or follow me on Twitter for general datavis ranting.

I’m deeply grateful to Alice Thudt for her brilliant suggestions and comments on the article.

About me

I’m Dr. Dominikus Baur, an award-winning datavis designer and developer. You can find the projects I’m most proud of and more on my website: https://do.minik.us.

You have a fascinating project to work on? You want to turn these ideas into reality? Let me know!

Silent Augmented Reality was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

Data visualization on mobile devices seemed promising, since the time of the first iPhone: very capable portable computers! Innovative touch interaction! Highly localized content! Hundreds of visualizations for mobile devices exist, both as apps and part of daily news content. But there’s one major problem that mobile visualizations couldn’t shake yet:

There’s just never enough space.

Mobile displays are necessarily small to be portable, and then there’s also fingers in the way. Usually with data visualization, more screen space means better analysis: Data can be shown at a higher resolution, uncovering smaller relationships and parts of the data. It also becomes possible to show multiple charts side-by-side and having coordinated views, quickly flicking to and fro from one lens on the data to the other.

It’s the difference between having your chart on a sticky note or across two side-by-side posters.

I think that Augmented Reality (AR) could solve this problem.

AR has mainly been a research direction for the last thirty-odd years, but is now slowly entering the tech mainstream. AR overlays reality with virtual information, which appears to be part of the environment (depending on your company affiliation, the principle could also be called Mixed Reality). Think: Snapchat Lenses with face tracking, Microsoft’s HoloLens letting you play Minecraft on your couch table or Apple’s recent ARKit.

When AR was initially proposed, headsets were usually bulky with low resolutions and low refresh rates. Hickups between head movement and on-screen content made it hard to keep up the illusion of actual virtual elements in your physical world, maybe even leading to cyber sickness.

Smartphone production and improvements in technology leading to lower prices for better components have both revolutionized Virtual Reality (see Oculus) and also Augmented Reality — since they’re both building on similar technologies (small, high-resolution portable displays, head tracking, etc).

So while our current AR is mostly based on the peephole metaphor (you’re looking through your phone to see face filters or Pokémon), future AR - the one we’re interested in for this article - should work with a headset only — hopefully very non-intrusive glasses — leaving your hands free to interact with (augmented) reality.

So how could this future AR solve the lack of screen space for mobile visualizations?

By augmenting your reality, AR puts screens everywhere and nowhere at the same time. Screens become fully virtual — with all advantages. Lack of screen space no longer exists, since these virtual screens can potentially fill your whole field-of-view (and beyond).

Plus, just like other mobile devices, AR devices know where you are in the world (thanks to geolocation) but even more: where and what your are currently looking at! Combine that with automated object recognition and you have all kinds of fascinating applications for datavis opening up.

To be more specific, I can see three promising directions for these future Augmented Reality visualizations:

Situated personal visualizations

One of the promises of mobile visualizations has always been creating a personalized experience of data. Various apps make use of your location to e.g. center a map on your current position (Google Maps), show restaurants around you (Yelp) or keep track of your running routes (Runkeeper).

AR visualization has the potential to become even more personal: knowing about your preferences and goals, it can display the right data at the right time. But what’s even more interesting is that everything can be situated in the right place:

AR Visualizations become part of your environment, augmenting real-world objects (and people) with relevant information, placed at just the right positions. Visualizations are no longer images displayed in little glowing boxes, but augmented textures to the world.

Imagine trying to make your way across Boston in a snow storm. Your app knows exactly where you want to go and can draw from relevant information from the internet. It also knows that you’re looking at a bus right now and helps you in making decisions: should I get on the bus or take an Uber? Do I have to hurry? Where do I have to transfer? How long will it take? And will this snow ever end?

All this information is right where you need it and completely private — no one else can see what you see. It’s the ultimate expression of personal visualization.

I like to see this development as a form of empowerment, if we get it right (if not: see the works of Keiichi Matsuda or various dystopian SciFi).

Similarly, AR vis can show data that’s relevant to you but maybe not to everybody.

You might be wandering along the aisles of your supermarket, being bombarded by messages of abundance. Package design in recent years hasn’t necessarily developed towards making nutritional information easier accessible. I value the hours I spend staring at small print labels, results of months of heated discussions between industry and administration.

While there are databases for nutritional information freely available, typing product names into a search box, selecting the right one, checking the info and repeating it for every sparkly box on the shelf sounds … unappealing.

What if the machine could help you with that, take your own preferences when it comes to nutrition (and dietary restrictions to boot) and display a simple heat map, pointing you at just the right products? Once this filtering is done you can still go through the final candidates and make your own, much more informed, decision. And everyone gets their own custom heatmap, since there’s more than enough space for that in virtuality.

What all these projects have in common, is that they’re mainly interfaces for data access. The info is usually somewhere out there on the internet but only accessible via text search or obscure API calls. The beauty of AR with its camera-based object recognition is that the whole world becomes the interface to this data: just look at something to receive more information.

Arguably, the same way we did information access since the dawn of time.

3D visualizations

You heard me right, I’m also crossing that line here: I think that 3D visualizations could potentially be much more useful and even become mainstream with Augmented Reality.

3D visualizations are shunned in the datavis community, fueled by scores of bad 3D Excel charts and blatant marketing deception. But if you look at the perceptual science behind them, they might actually not be that terrible (see Robert Kosara’s great discussion on his blog). Sure, they suffer from occlusion and perspective distortions, but maybe the additional spatial dimension might make up for that. Especially when we combine them with the most important feature of AR: situatedness in the real world.

Imagine having your awful awful 3D bar chart situated on your coffee table, right in front of you. There’s occlusion (front bars occlude back bars) and distortion (back bars look relative smaller).

But in AR, the main problem of 3D vis - the virtuality of it all - is less pronounced. The abstract bars become parts of your environment — it’s clear that the bars in the back look smaller than they actually are (compare them to the physical book next to them). Similarly, occlusion is easily solved by moving around the table, just as you would in the real world. Additional stereo cues (hard to show in a 2D photo) make the virtual bars seem more real than they are.

This follows the ideas of Embodied Cognition, a theory that postulates that our cognition is much more closely coupled to our bodily existence than might be apparent. With AR, you still have your own body available to you to explore data as part of your environment. This is in contrast to Virtual Reality, where you’re completely isolated from both your environment and your body — which can be highly disorienting.

Sporadic research here and there points at AR 3D vis being a promising direction (Ware and Mitchell created a highly efficient 3D node chart in 2005, and there’s a workshop series on Immersive Analytics), but I figure that the proliferation of AR toolkits will lead to a lot more results in the near future.

And yes, there will be the trend of cramming the first AR apps to the brim with flashy 3D stuff. Just like the first new 3D movies insisted on always throwing virtual objects at the audience. But that will subside eventually and make 3D a viable approach in the data visualizer’s toolkit.

Floating screens everywhere

Finally, AR solves the Mobile Vis dilemma of never enough space by simply letting you create as many arbitrarily-sized and -shaped displays as you need. Free-floating around you, still 2D, but as big as you need them.

Since full AR devices have their physical screens right on your nose, they can create an infinite number of arbitrarily-sized virtual screens, situated in your environment. Head tracking lets you switch between them just as you would with physical screens: simply by turning your head.

These virtual screens are interesting when it comes to resolution — since they’re just simulations their resolution can be as finely grained as you need them to be. The physical resolution of the AR headset always stays the same, but when there’s only part of a virtual screen visible (since you’re standing close to one), the full physical resolution is mapped to this part of the virtual screen. This works well, since our eyes are not capable of seeing everything infinitely sharp. We usually move around our environment, coming closer for in-depth inspection. Same with virtual screens — while they might be relatively low-res from afar, you can move as close as you like to see infinitely fine details.

There is some ideas in this direction — Isenberg et al. describe “Hybrid-Image Visualizations” that show different types of visualizations at different viewing distances. Similarly, the static Fat Fonts encode multiple layers of values through symbols and brightness, showing different aspects for different viewing distances. And, of course, highly detailed paper-based visualizations inherently allow accessing the data at overview- or detail levels.

These virtual 2D screens are also a great transitional technology, until all legacy applications have been mapped to an AR-context (if that ever happens). You can think of them as virtual monitors, same as your regular physical ones, just instantaneous, free and 100% eco-friendly.

And another advantage, especially when it comes to visualizing sensitive data — be it personal or business — in a public setting: they’re completely invisible to everyone else. The problem of “shoulder surfing” does not exist in AR, so no one can see what’s on your virtual screen, even though it might fill your whole field-of-view¹.

Shut up and take my money

So, how much longer until we can get to work on AR visualizations?

The release of tools like ARKit will make powerful AR apps much more common in the next years. Another nice side-effect for Apple is that once display technology becomes mature (and especially thin) enough that they can release their own set of AR glasses (which their patents point at), they already have an App Store full of AR-compatible apps.

Microsoft’s HoloLens is already available, not very portable and dorky, but almost magical when you put it on. Google has Daydream and Tango, which might make a powerful combination for Augmented Reality. We’ll see which of the tech giants will be the first to bring AR to the mainstream.

But in any case: smartphones are a transitional technology — we’re actually working towards a world without screens, where everything is a screen. And in this world, all the screen space you could ever want for your data visualizations is instantly available.

(This is part 1. Read part 2 on creating silent augmented reality here.)

If you like this article, please ❤/👏 or share it! For more like this, follow me on Medium or follow me on Twitter for general datavis ranting.

Notes

¹ Admittedly, shoulder surfing might be possible for someone standing so awkwardly behind you that they can read what’s on the inside of your high-resolution glasses. All the spoils would be deserved.

About me

I’m Dr. Dominikus Baur, an award-winning datavis designer and developer. You can find the projects I’m most proud of and more on my website: https://do.minik.us.

You have a fascinating project to work on? You want to turn these ideas into reality? Let me know!

Can Augmented Reality solve Mobile Visualization? was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

A few days ago, Apple launched their first TV show “Planet of the Apps” to great fanfare. It’s more or less a spin-off of “Shark Tank” with teams of developers pitching their app ideas to an all-star jury with such tech icons as Jessica Alba and Gwyneth Paltrow. The winners receive funding from a VC firm, enabling them to live their dream of developing software.

While few people might have been aware of the show’s existence, this weekend’s Twitter outrage over a (by now pulled) ad for it might have changed that:

Jason Fried on Twitter

Pathetic... even Apple is promoting workaholism now. Check out this ad for their Planet Of The Apps show.

Reactions, at least in my Twitter feed, consistently fell on the spectrum between outrage and disgust.

Similarly, freelance marketplace Fiverr had some fun with ads on the New York Subway a few months back:

it's B! Cavello ✊️ on Twitter

The "gig economy" is literally killing us. Most depressing ad of the day goes to: @fiverr 🙃

Fiverr’s business model is like Uber for designers — instead of charging adequate and life-sustaining prices for your work, you give it away for literally a fiver. Subsequently, “doers” are preferred.

In addition to the predictable Twitter outrage+disgust package, the ad also spawned an in-official follow-up:

So, why would companies pay for ads like that? Is it just about the there’s-no-such-thing-as-bad-publicity? Are they purposefully triggering Twitter for free exposure?

It could also be that we are just not the target audience.

A staple of the internet of the naughts were 419 or Nigerian scam mails. Millions of super-rich uncles, presidents and princes had to perish in order to leave behind Quadrillions of Dollars for hopeful recipients. When reading one of these emails (they still trickle in from time to time) one might be wondering who would fall for such an obvious scam. And that is exactly the point.

A study by Cormac Herley from Microsoft (and popularized by Levitt and Dubner in Freakonomics) discusses the basic economic problem that scammers are facing: while it’s cheap to send out millions of spam mails, you only have very limited resources for interacting with (and actually scamming) your victims. Every interaction via email or phone costs time, so by adding typos and coming up with ridiculous stories you automatically vet people who would never fall for your scam in the first place. Only the most gullible (and greedy) remain and you can throw your emails at them, hoping for a big payout.

Herley explains that this seems like the most economically sound strategy to avoid “false positives” (that is, people who answer a scam email but where all manpower to convince them to hand over their bank account will be wasted).

While Apple and Fiverr’s business models are obviously different from that of Nigerian Scammers, their ad strategies for the above examples might work along similar lines.

Just like scammers, Planet of the Apps and Fiverr want to keep their rate of false positives as low as possible. They want to make sure that everyone they interact with and spend time and money on will perform as well as possible for their purpose.

App developers who in the middle of the show’s season decide that these weird little humans from their emails might be worth spending time with are to be shunned, just like designers who at some point realize that sleeping at least once in three days might be preferable to doing another website for far less than minimum wage.

By putting out ads like the ones above you make sure that such people won’t even think about applying for your service. People whose initial reaction is furiously hitting the Quote Tweet button for snark while mumbling something about work-life-balance are not the targets. Not only do you avoid sorting them out later (at much higher cost), but they might even produce free, outrage-driven publicity in the process (this article not excluded).

This becomes even more effective by picking extreme, over-the-top slogans: just like Nigerian Scammers wouldn’t tease you with $250 but always promise at least a Googol of cash, these ads make sure that their targets are still excited about the offerings despite the most awful downsides:

It’s not that you might spend less time with friends as an app developer — you’re no longer able to take care of your children, the people that are probably the most reliant on you.

It’s also not that you leave the office a little later from time to time as a designer — you completely forgo the needs of your body, burning out both mentally and physically.

And if these prospects don’t thrill you— congratulations, you just saved Apple and Fiverr a false positive.

If you like this article, please ❤ or share it! For more like this, follow me on Medium or follow me on Twitter for general ranting.

Let’s take a simple example to make my point, one “static” visualization that is almost cliché: Napoleon’s march by Minard.

This graphic has been thrown around so many times since Tufte brought it up in his landmark book that you’d be hard-pressed to find a visualization course that doesn’t mention it.

Let’s look at it — once more:

If you’re like me and you spent far too much time on this stuff, you’ll probably just brush over it. It triggers the “Napoleons march by Minard” association in your head and your good ol’ lazy brain thinks: ok, been there, done that.

Try to read it, though (do it for me — I’ll wait here in my parentheses).

If you actually read Minard’s visualization, you take in one concept after the other. You might start with the headline and give up after too much French. Then you take on the main, brownish stream. Glance at the depressing black backwards stream. Then brush over some numbers and maybe the scale at the bottom.

Having understood what’s there and how it’s organized, you actually start to take in the data. You read from left to right, because that’s how you learned to do it, absorb Polish city names, look at the smaller and smaller army arrow and the weird turns they make. Once at Moscow, you take the trip back, follow the black area-turning-line. After that, you realize why Minard put a clever temperature scale at the bottom, providing another helpful explanation to make sense the disaster.

While reading the visualization the data unfolds.

And this is no accident, this is just the only way we can ever make sense of a visualization.

If we tried to visualize your reading the visualization (so meta) it might look like this:

If you read a visualization it’s no longer static. It can’t be. Just like words, reading always happens in time.

We can’t have it any other way. We as humans are temporal beings, one moment after the other until the last.

While going through time, our mind changes, our ideas and concepts of the world. We learn new things constantly, if we want it or not.

Think about every visualization you’ve ever seen (I won’t wait this time). You might have the habit of splitting them into “static” and “interactive” (the ones with buttons), because that’s how you learned to do it. We people are just amazingly fond of drawing borders.

But in this case, it does not make any sense.

Here’s a shitty infographic:

So little data! So big bubbles! So little interaction!

But interaction is all there is. Think through how you would read it. Again, starting with the headline, looking at visually prominent parts, reading the big big numbers, understanding the concept and finally absorbing the data.

No matter what you’re taught — even the thinnest infographic is inherently interactive. It can’t be any other way.

Reading any visualization is a two-step process:

1. Understanding the encoding
2. Reading the data

The first part, grasping the encoding, the visual mapping or whatever you want to call it, teaches you the meaning of all those circles, rectangles and lines. You find the connection between some topic (soldiers of the French army) and geometric shapes (width of arrows).

And usually, there’s more than one type of data to absorb. In Minard’s case, we also see the temperature, we see city names and even dates (usually, if a graphic only contains a single type of data we tend to sort it into the shitty-infographic-category — just too little data to bother).

If we have something like visualization literacy, we also come pre-equipped with a set of charts we readily recognize and understand. Making sense of something like a map or a timeline seems obvious to us, but it wasn’t always. Every type of chart has been invented at some point and we just learned it. Now, visualization designers can build on that knowledge, making this first step far easier, and come up with ever-cleverer combinations of charts.

In Minard’s example, we have a shrinking area moving along a map and a timeline at the bottom. In the shitty infographic, we have circle area/angles.

You could call this the “onboarding” part. Teaching people what they have in front of them and how to use it. If it’s done well, this step is painless.

If it’s done poorly, people will get lost in geometry soup.

Once we understand what the shapes mean we can start reading the data. And while the first part — if well-done — is relatively straightforward, this second part of actually understanding what’s happening is everything but.

Depending on your preferences, your knowledge of the data and subject matter, you can start reading wherever you like. Clearly structured visualizations like Minard’s give you a good idea here — the visual shapes point you towards the right way (sorry). Visual hierarchy establishes a reading hierarchy.

But other examples aren’t as obvious:

Especially in scatterplots, every entry point makes sense. Start with the overall blob of points in the middle – the “normal” – or look at the outliers. Follow one axis or the other. Search for specific points of interest to
you.

Once a visualization no longer tells a linear story, it’s choose-your-own-adventure. Any story you can find is fine — it’s your very own.

Except for the main two-step-process of understanding and reading, every other order is made by none other than the reader.

Here’s a great example by artist David Hockney (via Nicholas Felton’s excellent Photoviz and Frank Chimero’s ‘The Web’s Grain’ essay):

Hockney encodes the temporal aspect of the Scrabble game through spatial geometry. Looking at the piece, readers decode the game’s time by looking at this part, then that, then the other one. Playing it back in their own
heads, using their own eyes.

Arguably a visualization, The Scrabble Game again follows the two-step process, even without any explicit instructions.
We see familiar shapes of faces and a scrabble board, notice the inconsistencies and understand that time is encoded as overlapping photos. We then start reading it, jumping from picture to picture, taking in the data and forming the story.

Only a visualization no one ever sees can be static.

Visualization designers sometimes forget that visualization is a collaboration between them and the readers. And a visualization without an audience is a useless artifact. Making the encoding unnecessarily impenetrable or unlearnable or picking extra-shallow data just to create something aesthetic.

Someone invested time and energy into building the thing just to lock it up in a cupboard.

Even our “interactive” visualizations are nothing but static visualizations in time. If well-done, the encoding is consistent and it doesn’t matter if you filter for this-or-that first. It also doesn’t matter if you highlight this item or the other one. Nothing jumps around the screen and the encoding doesn’t morph halfway into something else.

You can restrict a reader’s journey through the data by pressing the “interactive” parts into a video, forcing them to sit through 2.5 minutes of you exploring the data for them, but again: all this happens in time, it has to, and even in a given frame, every one looks at something else (see the famous Selective Attention experiments in psychology).

And this is the same for any medium. Books and other textual mediums give us a relatively clear structure, which we still constantly ignore by going back and forth. While websites exist as infinite rectangles, people scroll up and down, focus on this and that. What we think of as a static website is really an endless series of short glimpses, bits of text and image parts, strained through our mind’s sieve.

Since visualizations are nothing but a practical form of pressing data through our eyeballs, we can’t escape their restrictions. Small eye movements (saccades) happen every 20 ms at rapid speeds. Every time we take in a small part of the world. Our brain manages to present us the illusion of a stable environment instead of a crazy roller coaster of dashing shapes.

One level up, the same happens when reading. We take in information, we understand and then ponder it. Our minds trick us into believing we had absorbed a whole text, while in actuality we’ve only kept the main points — everything in bold and whatever seemed fishy to us — and that we finished it and can take it off our bucket list.

Reading a text or visualization is exactly the same. Which is why visualizations can never be static, no matter what your textbook says.

We temporal beings are trapped in moments. We gather new information constantly, learn new things, but have them interact with what we already know and our more immediate memories. This beautiful dance of thoughts can only work in time.

And just as you can’t understand a visualization by looking at it (ooh pretty!), you can’t make sense of them without taking them in in your own time.

If you like this article, please ❤ or share it! For more like this, follow me on Medium or follow me on Twitter for general datavis ranting.

Also, let me know what you think in the comments!

About me

I’m Dr. Dominikus Baur, an award-winning datavis designer and developer. You can find the projects I’m most proud of and more on my website: https://do.minik.us.

You have a fascinating project to work on? You want to turn these ideas into reality? Let me know!

Static visualizations do not exist was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

After writing about the potential death of interactive visualizations, I want to touch on why they’re still absolutely worth it.

We as humans are notoriously bad at forming a balanced and comprehensive picture of anything more complex than our shoe strings. We’re plagued by various cognitive biases that span everything from an extreme interest in things that confirm our existing ideas and beliefs to prefering lots of completely useless data to less, more focused information. One of the most notorious ones is our tendency to overvalue singular examples: anecdotes are our favorite way of making sense of a situation - or more often to shoot down unpleasant truths (I’m sure you also know someone whose grandpa was a heavy smoker and made it to the age of 93).

Fortunately (and despite all these biases), we also invented a method called statistics, which promises to sieve data and only leave pure information behind. After applying various types of statistical analyses (or more newfangled machine learning methods) we usually arrive at a set of numbers that clearly and unambiguously describe our data’s distribution and whether our hypotheses were right or wrong.

Nathan LeClaire on Twitter

NERDS: Statistics is pretty cool guys ok WORLD: whatever NERDS: :( NERDS: it's called Machine Learning now WORLD: OMG MUST HAVE IMMEDIATELY

However, statistics sometimes seems to be too much for our feeble human minds that deep down long for stories, emotions and things they can potentially grab and smear with mud.

So how do we bridge this gap? How can we relate to parts of our data while at the same time being able to see the bigger picture?

The answer is micro-macro.

What I mean by micro-macro is trying to get a better understanding of the world by accessing it on two levels: for one, there’s the micro-level of anecdotes where we get the good feeling of looking at actual, concrete aspects of the world instead of abstract mathematical descriptions. But we combine this with the macro-level to understand how these relatable anecdotes fit into the whole.

This dual approach enables us to estimate if a given example represents normalcy (a stand-in for how things “usually” are) or is an outlier and does not allow conclusions for all cases.

Plus, we’re also avoiding a problem with statistics: by reducing complex data to simple numbers, we’re of course losing information. All outliers and other weird aspects of the data simply get smoothed out until we end up with a clear bell (or other) curve. The smoking grandpa would just never show up in a simple histogram.

Fortunately, there’s data visualization which is the premiere tool for enabling us to browse data sets in a micro-macro way.

micro-macro + datavis = ❤

Take as an example for this thoroughness of data visualization Periscopic’s U.S. Gun Deaths in 2013 visualization:

This disturbing piece shows how many people have been shot in the US in 2013 and the resulting number of years “stolen” (based on US Census data). Every victim becomes one arc in the visualization, with their years alive colored orange and the rest in a somber grey. By showing every person as a visual element, the visualization does not only provide a more striking image of the extent of these tragedies but also avoids reducing these people to aggregated statistical numbers (average age when shot or something). Every haunting story is still in the data and the graphic.

But these anecdotes (micro) come in the context of the visualization (macro): it’s clear that a given arc is not the whole story, that a person might have been exceptionally young or old or the norm (as morbid as that is).

Gun Deaths is also interactive and it’s possible to hover over arcs to get name, age and place of death of the victim. With interactivity it becomes possible to learn about these cases and also the outliers, like the story of 107-year old Monroe Isadore, apparently the oldest victim in the data set. But thanks to the macro-level, nobody would pull him out as a representative example for all gun victims or declare gun wounds a new major health crisis for 100+ year olds.

Data sets consist of data points. At some point, these points have been collected and aggregated. This abstraction step leads to broader but less detailed descriptions like statistics. With datavis, we’re no longer trapped on the macro-level — undoing the abstraction and uncovering the original data points in all their glorious micro-level nature.

When we’re looking at gun deaths in the US, for example, we have victims, their ages and place of death. For a proposed health care reform, we have people with specific needs, financial situations and existing insurances. If our topic is well-being across the world, we’re looking at specific regions with various attributes.

Statistics gives us the overarching patterns of the data in an abstract way. A visualization can show these patterns and more: it can even give us back the original data points.

Of course, there are limits to this approach: browsing through every single data point does not scale and would certainly be more confusing than helpful. But every single aspect, every little story we explore by hovering over a dot makes the data more humane and makes it easier for us to tie it back to our own experience of the world.

The superpower of interactive datavis is this fluent switching between the macro-level of overarching patterns and the micro-level of raw anecdotes.

There are also certain types of charts that lend themselves more to a micro-macro approach. The more statistical types of charts like bar or line have a certain aggregation already built-in and make going back to the original data less obvious. Scatter plots are probably the prime example for a micro-macro compatible chart type, with their data-points-become-circles approach.

Enriching anecdotes

The micro-macro idea holds an additional promise: transgressing the restrictions of the visualization itself and providing an even richer picture.

When creating a data visualization, certain aspects of the data just inevitably disappear: some parts might not correspond to the patterns the designer wanted to shed a light on (don’t forget that even datavis designers are authors as Moritz Stefaner argues). Aspects that couldn’t be fit into our meager two visual dimensions were dropped. Or, since data is primarily human-made, the ones that were simply not quantifiable into neat digital numbers were removed.

Nobody can keep designers from making these aspects available through interaction, though!

Pudding’s The Unlikely Odds of Making It Big is one such example. The great piece does away with the “Everybody can make it big!” narrative, by example of New York bands.

When reading about successful bands, it’s all too common to hear about their humble origins and coming to the (utterly wrong) conclusion that playing in grimy bars is always just a phase and super-stardom is just around the corner. This again very effectively demonstrates the dangers of emphasizing the micro without mentioning the macro: by reading scores of articles about successful bands without ever being confronted with the reality of the vast majority of musicians, we’re inevitably biased towards overestimating the chances of success.

Making It Big clearly shows the sad reality of only 21 out of the 7000 bands in the data ever playing at bigger venues.

This new context for the anecdotes is very powerful. But the piece also provides additional context lost in the visualization: the actual music. For all the bands in the data that made it big, there are audio samples available to get an idea of what type of music they’re playing. With music being so central to a band’s success (probably), this is a great example for enriching the anecdotes while keeping the context.

The ultimate step in enriching anecdotes and enabling micro-level inspection is showing the actual raw data.

This is something we did in Selfiecity — a project that looks at selfies as a cultural phenomenon through a data lens. One part of it was the Selfiexploratory playground where you can explore the full data set of (currently) 3840 selfies:

There’s no aggregation or reduction in complexity— except for shrinking the image resolution, the selfies at the bottom of the Selfiexploratory are the same ones we’ve based the overall analysis on. It would be hard to enrich those anecdotes even more.

Providing access to the raw data is much more than a gimmick: as Gregor Aisch argues in his excellent piece on interactive graphics it can create trust through transparency. By making all the data available, the visualization becomes less of a black box and makes fact checking possible.

Since a large part of Selfiecity’s analyses was based on computer vision, we ourselves wanted to be sure that the algorithms that had extracted things like age, gender, pose, facial features and mood from the images had the right idea.

And the Selfiexploratory makes this fact checking a painless process:

Filter for a certain aspect of the data (for example, head tilted to the left) and see if the resulting images make sense (try spotting the feline outlier in the screenshot to the left).

If you spend some time playing with the filters in this way it becomes clear that the computer vision algorithms aren’t perfect — but definitely good enough to reach some overarching conclusions about selfies.

Showing the raw images enables even more. Similar to Pudding’s Making It Big, there’s more to the data than can be expressed in numbers. When you’re thinking about selfies, aspects like head tilt or closed eyes might not be the first associations that pop into your head —a lot of people’s first thought (mine included) is probably duck face.

And despite there not being a reliable duck face algorithm available, having access to the raw data lets you do this analysis yourself on the micro-level: simply by browsing some pages of images with a filter for the usual suspects (young women) you can make up your own mind.

A micro-macro look on the world

When trying to make sense of the world, all of us are overwhelmed by the wealth of data. Taking recourse in fluffy anecdotes heavily distorts our perception, while looking at statistics usually leaves us cold.

Data visualization (as part of data humanism to use giorgia lupi’s great term) can give us the best of both worlds: showing us relatable parts from the data’s micro-level while ensuring we understand how they fit into the overarching macro picture. Going as far as showing the raw data enables fact checking and increases trust.

And by providing aspects that go beyond the limits of computers and visualization we can enrich those anecdotes even further, thus dragging the data down from lofty analytical spheres and making sure it fits in with where it came from: our own world.

If you like this article, please ❤ or share it! For more like this, follow me on Medium or follow me on Twitter for general datavis ranting.

Thanks to Moritz Stefaner for suggesting the term ‘micro-macro’ and both him and Alice Thudt for great feedback to the draft.

For more on this topic read these excellent articles by fantastic people:
Giorgia Lupi: Data Humanism, the Revolution will be Visualized.
Gregor Aisch: In Defense of Interactive Graphics.

About me

I’m Dr. Dominikus Baur, an award-winning datavis designer and developer. You can find the projects I’m most proud of and more on my website: https://do.minik.us.

You have a fascinating project to work on? You want to turn these ideas into reality? Let me know!

(edit: Gregor released a new blog post, clarifying some of the aspects and made some great points on the benefits of interactivity)

Last year I was lucky enough to go to the Information+ conference in Vancouver where Gregor Aisch, who works at the New York Times, gave a talk about the publication’s graphics and their impact. And the scary resumé of the talk was: Barely anyone interacts with the New York Times’ graphics. The New York Times makes arguably some of the best interactives in the field, which made Gregor’s talk even more depressing. His number of only 10–15% of people clicking on buttons — even essential ones — tells you that interactives are a waste of time and money.

Gregor’s editor, Archie Tse, talked about this earlier in the year at the Malofiej conference, and turned this fact into some utterly depressing rules. One of them was, for example, “If you make a tooltip or rollover, assume no one will ever see it.”

85% of page visitors simply ignore them, missing out on information hidden behind interaction. On top of that, interactives are expensive to make — they have to work across devices, using trackpads and fingers. They’re error-prone and can tarnish the publication’s reception within their audience.

So why even bother?

I’ve been working on data visualization for almost ten years now, first as a PhD student in Munich, then as a researcher and now as a freelancer. And if I had to name the one aspect that most fascinates me, it’s their interactivity and the potential therein.

Of course, the power alone to compress complex datasets into approachable and even appealing graphics is fascinating. The craft of shaping data to perfectly fit into the interface of our visual processing systems. The sheer wonder of being able to squeeze thousands of data points into a picture most of us are intuitively able to grasp.

But interaction lets them do even more. If you think about visualizations as a mass medium, something made for huge audiences, interaction turns them into very personal tools.

If you’re doing interaction well, it can turn your visualization from a well-made newspaper that gives you the bullet points into a conversation almost. As if you were having a tête-à-tête with an expert on the data, patient enough to explain you everything.

That’s the ideal, at least.

First of all — what do we mean by interaction?

Basically, interactive infographics describe visualization systems that have ways for the end user to change their attributes. This can be super-simple, such as changing the currently visible part of a map or tapping on a circle to get a detailed description of that data point. Interaction can also be more complex though, like drawing an example of the data you’re looking for and the machine finding it.

So, it’s somewhat of a catch-all term for clicking or tapping somewhere on that digital surface.

Doing a full definition would probably take a lot longer than the length of this talk, so for our purposes, let’s go with this simple one:

Interaction in visualizations changes the lens on the data.

This can mean to filter certain datapoints, select a different area of the data or even changing the type of visualization altogether. The important point is: an interactive visualization is no longer static and doesn’t represent a single view on the data. Interaction enables people to adjust a visualization to their own needs and ask it different questions.

Interaction does not have to happen with clicky-things on websites.

Interaction with an infographic on a high-resolution medium such as paper can be a powerful experience.

Edward Tufte likes to present the above chart in his workshops and let participants really drill into the data. Explore the chart not using your hands, but using your eyes. Focusing on this corner, then that. Building up an image of the data in your head. And even going so far as forming hypotheses in your mind and trying to confirm them by looking at the relevant section of the visualization.

The design studio Accurat also did several brillant infographics for newspaper ‘Corriere della sera’ and their Sunday supplement ‘La Lettura’. Highly detailed, complex graphics, with unusual ways of data encoding and presentation.

Since we now have a rough idea of what interaction can be — can we maybe find the criteria for successful interactive pieces? What are the requirements for those?

While I don’t have an algorithm for that (sorry), maybe we can get some idea. My main impression however is: we data vis people spend too much time thinking about the interactions themselves and less about the audience who is supposed to be using them.

And then, well, they might end up NOT using them.

We might be super excited about some clever interaction trick, but maybe we’ve already lost our audience before they even saw the graphic. So, as always in design, be aware of your assumptions and your personal bias.

Going back to the New York Times, their pieces are made for a very specific situation: dealing with news in this realm of data journalism means that speed is everything. Content has to be produced quickly and timely. It’s extremely hard to do an interactive graphic for something unexpected and some of their pieces have weeks of work put into them.

Speed is also the most important aspect on the other end of the chain: how do you consume your New York Times articles? Are you sitting down for half an hour with your iPad in hand after breakfast, carefully scouring the Times’ website, making notes? Or is it more that someone on Twitter or Facebook shared an article and clicking on that link takes the same amount of time as clicking on the little ‘X’ button in the top-right?

TIME

Being able to really appreciate something like a visualization or even interacting with it needs TIME. Which gives us a clue for one of the requirements for successful interactive infographics: you have to be aware of the audience’s context, if they’re actually in the mood for in-depth data exploration.

Think back to the image of the newspaper versus the data expert. If you’re sending that expert to some random bus stop to talk to people who have about two minutes before their buses leave you can imagine how much they’ll get out of it. And how many datapoints they will miss.

That’s why I really like doing interactive installations.

End of 2014 I was working with Moritz Stefaner, Lev Manovich, Daniel Goddemeyer and a couple other highly talented people on a project called ON BROADWAY, something heavy on the interaction side, with which you could browse along the Broadway in New York and its associated digital data.

Last year, Moritz and I had another installation project for the Ecole Polytechnique Federale de Lausanne, the university in Lausanne, Switzerland. For their new wing they wanted to have a visualization representing the wealth of data that they had collected about their university — aspects of their teaching as well as research capabilities.

The end result, called THE DATA MONOLITH, is a pretty massive thing. It’s over 4 meters tall, contains two touchscreens, one 4K display and a back projection at the very top. The touchscreens work as remote controls for the big screen, with which you can change which part of the data you’re looking at.

The visualization is organized along three perspectives on the data — PEOPLE, TOPICS and IMPACT.

PEOPLE is about students, teachers and researchers in the university. Each person who has anything to do with the university becomes a little bubble and re-organizes themselves based on the current visualization. TOPICS presents the network of researchers and research topics — almost like a neural network that maps out the university’s research interests and their corresponding brain power. Finally, there’s the IMPACT view on the data, that shows the impact that EPFL’s researchers have on the science landscape and their collaborations world-wide.

This massive dataset with its various perspectives would be too hard to boil down to specific messages — interaction is absolutely required.

Fortunately — and that’s the great thing about developing interactive installations — in such contexts people have the time. Most people go to museums or such installations in general to be entertained and learn something. They don’t mind “working” their way through the data with interaction. If presented in the right way, this interaction can even be part of the fun.

GOALS

In addition to time, successful interaction also has to take into account the goals of the viewer. And these can be pretty different.

In the EPFL Monolith case, students and teachers at the university are a big part of the audience. They might have fun trying to find themselves in the PEOPLE visualization. University visitors, though, might be more interested in getting an overall picture of the university climate and personnel. And datavis people might just be interested in breaking stuff ;)

Interaction, if done well, caters to all these groups. They’re able to shape the data based on their own interests. Like being able to ask the data expert specific questions.

Talking about goals, these goals that an audience has can also be of a more practical nature. It doesn’t have to be ‘I want to learn about X’. It can also be ‘I want to do Y, show me how’. I like to call this idea Visualization As Interface. A way to access the data in a more humane way, a middle ground between untraceable black box machine learning answers and spreadsheets.

One project from last year where I went this route together with Daniel Goddemeyer was SUBSPOTTING.

If you’ve ever been to New York and used the subway, you might have noticed that, officially, there’s no cellphone reception on the subway. But if you’re aware of where to look, you can actually find reception on the subway here and there, there are pockets of connectivity, with antennas from aboveground being able to penetrate down to the subway tunnels. It’s just a little hard to tell where.

That’s where Subspotting comes in. For the project, Daniel and I were interested in the how extensive these hidden cellphone networks actually are. So we started looking for the corresponding datasets … but couldn’t find any. And finally decided to collect this data ourselves.

We had a case with four iPhones (for the four carriers) in it as a logging device. So, it was a super guerilla approach as you can see. And of course we needed someone to actually bring that case along the over 1000kms of subway network found in New York. So, Daniel went on his merry way, since I wasn’t in the city… no, actually we hired a task rabbit to do it for us who was pretty excited about getting paid to ride the subway, read a book and press a button at every stop.

The end result was a pretty massive dataset of the cellphone reception for each of the 25 lines. After some data cleaning, we turned to paper as our favorite high-resolution visualization medium and created two posters, one highlighting the geographical aspects, the other the subway lines themselves. They’re great for exploring the data, following the lines and checking out specific carriers.

But of course they’re not much use for actual people riding the subway and looking for a cellphone signal.

For that, we created an additional app, called Subspotting. And here’s where the Visualization As Interface aspect comes in. Users can draw a more concrete benefit from having access to this data, by finding out where on their daily commute they can get cell phone reception and plan accordingly.

The data is organized as a series of cards, showing each subway line as overview or in detail. Cell phone reception strength is encoded as bar charts mapped to each part of the line. Switching lines happens by swiping left or right, moving to different parts of the line works through scrolling and you can filter for different carriers. This way, interaction enables quick access to the complex dataset and visualization makes the data understandable.

Subspotting shows that interaction can be successful if people are actually getting something out of it. When it comes to simply teaching them about a certain dataset and its aspects, the immediate benefit might not be as clear cut. If they want to find out where exactly they will have cellphone access on their morning commute, an interactive visualization is probably the best way to give them quick access.

In which other ways than respecting their TIME and their GOALS can we make sure that people will actually be interacting with our visualization pieces.

Again: be aware of your assumptions. Especially us datavis nerds often have the dangerous assumption that everyone is as crazy about datavis as we are. I mean, just think back to my grandiose introduction of visualization in the beginning here.

But, of course, that’s wrong. And that’s healthy! It’s a good thing not everyone is fighting about pie charts on Twitter!

So, given that most people aren’t that interested in visualization (let’s be honest here), we have to find a way to make them CARE about it. Because if they don’t care, they won’t look at it, let alone interact with it.

And even before they start caring about it, you want to make sure that you’re not closing the door right in their face. Don’t throw bar charts at them until they close that browser tab.

ONBOARDING

One aspect of that is the new fangled term “user onboarding” (how to get people on board). Basically, what happens in the first moments after a person opens your app or website or even newspaper page. This can shape the rest of people’s experience with your infographic. And of course also if they’re quickly frustrated and just close it and go do something else.

Onboarding is something that even happens in print. If we go back to Accurat’s La Lettura visualizations, each of them has a short introductory paragraph on the top left (where you would naturally start reading with a Western background) and then a section titled ‘How to read it?’. That gives you a clear idea of how to work with the graphic before diving into the details.

Other great examples for tutorials and guided exploration of visualizations are Nicky Case’s projects — if you haven’t played with ‘Parable of the polygons’, for example, you should definitely check it out. A clean text explanation of what this project is about plus interspersed interactive elements for exploration.

…. and those are basically all great examples for user onboarding in visualization that I’ve found despite my super-thorough research.

But seriously, it’s something that I and the visualization community really could become better at.

Good onboarding is the data expert giving a short introduction to the data and what they know about it, instead of just looking at you… blankly … staring into your soul…

But beyond simply teaching people how to read and understand your visualization, respecting their time and interests can go even further.

CARE

You want to make sure that they CARE about your visualization. Now, how to do that?

If we’re really _really_ honest about ourselves, what’s the one thing that’s endlessly fascinating to us and we could talk about forever?

Right. Ourselves.

So one simple trick to draw people into a visualization is by appealing to this inherent narcissism. Quickly answer the question ‘Why should I care. What is it to me? Why would I give any expletive about the situation in so-and-so?’

If you provide them with an answer right away, telling them what this dataset could mean to them, they might actually listen.

This is a pattern that you often see in visualizations:

In this piece by the BBC about humanity crossing the number of 7 billion people (those were the days), they ask you a few questions to arrive at your very unique and personal place in the world.

Here’s another one by the New York Times, that shows you how much less your house is worth because it’s built on Main Street instead of Ocean Boulevard. Again, you can enter your own street name and explore how that changes things.

But instead of directly asking people you can also be more subtle about it. An easy way to get access to this type of contextual information is through the various sensors hidden in our smartphones and laptops. You could almost call that ‘passive interaction’.

Moritz Stefaner and I did a project for the OECD in 2014 called “Regional Well-Being”. The OECD is spending a lot of time capturing factors of well-being in their member countries and with this project, they decided to dive from a national to a regional level. So it was no longer about the quality of life in Germany, but in Bavaria versus Berlin and so on.

This of course also made the dataset much more complicated — while their Better Life Index contains 11 dimensions for 35 countries, the Regional Well-Being data contains 11 dimensions for 395 regions! Since this can make the data pretty overwhelming at first, Moritz and I decided to start with something that our audience could relate to — the quality of life in their own region.

So, when you open www.oecdregionalwellbeing.org your browser asks you to give them your current location (browsers can do that, and if you’re not comfortable with it you can also select it from a list). And the visualization then starts at this location, so you can look at how life is around you. From there as a starting point, you can branch out in your exploration — either looking at spatially close regions or regions that are similar to your own.

Or even go somewhere completely different. Starting in your own region lures you into the visualization and gives you a reason to actually care about it.

Ok, after these ideas when it comes to interaction, going back to our original question:

Are interactive infographics dead?

My admittedly super-lame answer is: it depends (as always).

It very much depends on your audience and their context. Maybe the heyday of interaction actually is over this period of total excitement about all the things that have suddenly become possible. The field of data visualization is maturing and that also always means cutting away the wild growth that has sprung from all the original possibilities.

Think about your audience’s TIME — do they have 30 seconds or 15 minutes?

Also think about their GOALS — what can they get out of your visualization? Is it maybe more than just fun factoids? Can they use the visualization as an interface helping them with something specific?

Finally, think about what they CARE about — guide them into the visualization, make your visualization about themselves and show them why the data is relevant to them. And if that’s not possible, maybe intricate interaction won’t be required anyway…

Once you’re done thinking about it, add your interaction or leave it. Just doing it for interactivity’s sake doesn’t help anyone. Don’t force it on everybody just because you can.

If you like this article, please ❤/👏 or share it! For more like this, follow me on Medium or follow me on Twitter for general datavis ranting.

About me

I’m Dr. Dominikus Baur, an award-winning datavis designer and developer. You can find the projects I’m most proud of and more on my website: https://do.minik.us.

You have a fascinating project to work on? You want to turn these ideas into reality? Let me know!

The death of interactive infographics? was originally published in Startup Grind on Medium, where people are continuing the conversation by highlighting and responding to this story.

Even this headline.

Recently, I’ve stumbled upon this interesting article about evolution or atheism or something (I don’t quite remember what it was), but I do remember that it tugged at my heartstrings in just the right way. The first few sentences just brought the point beautifully across, a point which my mind - thoroughly stressed-out by the neverending roar of social media updates - just gladly accepted. Yes, these few sentences rang true to my predispositions and were so palatable as to provide me with some relief from the constant onslaught of rings and popups and reminders and all the other noise and let me focus - a rare experience these days.

Which is why I instantly decided to share it.

Once that thought had gotten hold of my mind I couldn’t really focus anymore on the article itself. Yes, it went on like that, something about dinosaurs and the milky way (the author obviously knew his awesome-o-pedia), but my brain was already somewhere else. On the long end with hundreds of likes and shares and friendly comments touting me as a sharer of wisdom, an amazing human being, somebody whose existence is acknowledged. So once I hit a really nice quote at the beginning of the third paragraph or so I just copypasted that to Facebook, put the url below it, and added some deep comment. The first sentence of the third paragraph - that is as far as I got in an article I found so enticing that I decided to share it with all these friends, co-workers and conference folk I’d barely talked to.

And this is symptomatic for all of the internet: everything on the internet is twice as long as it has to be. When was the last time you actually finished reading something on the web? Thoroughly follow an author’s argument without skipping to the conclusion paragraph? How long is your Instapaper reading queue?

Even the web’s most peculiar result, Twitter, with its laughable 140-character posts, follows this rule: while the first half of a tweet contains actual information, the second is invariably cluttered with snark, emoticons, links or that bane of the internet - hashtags. My rule of thumb is: in 90% of all cases you’re fine with just reading the first half of everything you find on the internet - even tweets.

Because, seriously, would any author ever be so cruel as to start their post with some well-meaning, insightful observation before diving into utterly destroying that argument in the latter half of their article? Thus fooling all people who had just (seemingly) read enough to get the gist of it? And share it to all their Facebook friends?

Or just repeat the same sentence over and over again ad nauseam? Repeat the same sentence over and over again ad nauseam? Repeat the same sentence over and over again ad nauseam? While interspersing some new words to make it look different? But still repeat the same sentence over and over again ad nauseam? Repeat the same sentence over and over again ad nauseam? Repeat the same sentence over and over again ad nauseam?

That could never happen.

Everything on the internet is twice as long as it has to be was originally published in I. M. H. O. on Medium, where people are continuing the conversation by highlighting and responding to this story.