Posts from a previous life

Published in

Design, Strategy, Data & People

75 min readDec 28, 2019

Six months ago I left Qlik. I’d spent almost 8.5 years with them and they’d taken me from London to Malmö, on to Copenhagen and then to San Francisco — it was a great journey and I met a lot of brilliant people and had a lot of fun, but it was time for a change.

Whilst with Qlik I wrote a bunch of blog posts for them loosely related to Data and Business Intelligence — mostly from my designer perspective. I recently found out that they are still up on the Qlik blog even though I thought the policy was to wipe past employee’s posts. So I decided I capture them here (just in case).

Between January 2015 and September 2017 I wrote 28 short posts for them (500 word target was the rule — but I’m a little wordy so rarely made it). Some of those posts were derived from longer pieces already posted here on Medium.

Here is a single super post containing all 28 posts. However, to start with here’s the list (linked to the original posts on Qlik):

The posts:

Touching is Trusting — Jan 19, 2015

Humans use their hands to qualify a thing, to gauge its authenticity, it’s our most trusted tool for measurement.

As humans our sense of touch plays a huge part in gauging quality, authenticity and ultimately enabling us to place trust in a thing. Whether we trust what we are looking at, using or being told is core to all exchanges and it comes from the sum of the experience.

However, for me there’s something intriguing around the influence of touch on trust. Does touching something help you understand it more even if that thing is virtual or abstract, such as a chart?

It seems counterintuitive — all you can physically feel is the glass. However humans have an amazing capacity to ‘virtualize’ their senses. I’ve often observed how people engage with touch-based UI. How they reach out to explore and are far more willing to try things out and experiment, to be curious. The way the UI reacts and the feedback is essential to the experience but it’s that natural act of reaching out and touching that is key. This experience can have a very real impact on the how people perceive and understand the activities that are taking place. I’ve heard how when a brokerage service shifted its customer-facing application for choosing a mortgage from laptop to iPad they saw a significant improvement for deals closed in-store, on the day. Prior to this, the company had a laptop-based tool that the broker would sit down with in front of a prospective customer and show them the various options and data on how different loans would work. When they replaced the laptop the results were astounding; they had a huge lift (30%) in the amount of deals closed in the store, on the day. I think much of this is to do with the shift in interpersonal dynamics that the iPad creates. How it changes from when an operator behind a machine reveals the results, to a person sitting with you helping you understand what you are doing. But here the big shift seems to be the fact that the customer is doing the exploring, the analysis, the ‘poking around’. They are simply more confident in what they are seeing because they are in control. There’s no ‘savvy operator’, no man behind the curtain weaving their magic and dazzling them with his skills, it’s in their hands.

We trust our hands. Being able to hold and touch helps us judge a thing; after all our hands have been our metric of choice since before we abstracted it away to rulers and scales. They are our most known and trusted measure.

The deep trust we have placed in our hands translates to touch-based UI. Touching a thing allows us to qualify the content in a very natural way, one that taps into deeply human evaluative processes. We weigh it, turn it, flex it, stroke it, tap it and feel for notches and extremities that could be changed or modify its form. The more a piece of screen-based content, such as a chart, can support these forms of examination the easier and more natural it is to work with. This leads to a better understanding of it and more confidence in our assessment of what it’s telling us.

Touching is key to trusting.

Responsiveness Is An Ethos — Feb 16, 2015

Responsive design is seen as a method or technique for reflowing content so that it fits the screen. But there’s more to it than that.

Ever since we “went digital” designers and developers have been chasing pixel perfect replication. It did not matter whether it suited that screen or the person viewing it, it had to be consistent. In fact, it had to be identical. It was a fool’s errand.

The variants of the technology were always too high, from screen size and screen resolution, gamma & color profile, down to OS & browser version. Things never displayed exactly the same. On top of this you have the person viewing it. Do they have vision impairment? Do they use assistive software? Or do they simply have a slightly different cultural understanding of what you mean when you use a particular color or phrase?

This attempt at a perfect reproduction is the result of concentrating on the artifact instead of the people who will be using or experiencing it. And being artifact-centric doesn’t help enough people. To improve this, the people who will utilize the artifact need to be placed at the center of the decisions on how the thing is made and distributed. After all, the artifact isn’t the point: it’s what it triggers and the actions taken that really matter.

This realization is in many ways behind Mobile First and Responsive Web Design thinking. Mostly, responsive design is seen as a method or technique for reflowing content so that it fits the screen. But there’s more to it than that. The real leap forward is not in the technique but in breaking away from the perfectly reproduced artifact and concentrating on maximizing the usefulness. Think about the value it provides for the person, whenever and wherever they interact with it.

My favorite example of this in Qlik Sense is the responsive charts. We don’t ask the person building the chart to make multiple versions to display at different screen sizes and resolutions. You create it once, keeping the focus on the content and usefulness rather than the artifact and distribution. Qlik Sense adapts the content and renders the chart in the best way to suit the available space, retaining the chart’s meaning and integrity. If it’s a small space we show hints and a suggestion of the shape, reducing the noise and erroneous parts so that the information sent is as strong as possible. When viewing these small charts it’s all about information foraging and spotting that indicator that says “look here, this is interesting”. Give the chart more “space” and more is revealed. The same key content is there but you see progressively more detail, enriching the chart with more information and tools for taking your inquiry further. This helps the chart be meaningful across a wider range of contexts and uses, without the creator needing to think about them all up front. It enables content created in Qlik Sense to move seamlessly through an organization and reach the people at the edges not just the ones whose desktop system is the same as the content creator’s.

It’s ultimately about getting the most value from a thing by realizing its maximum utility. Delivering usefulness and value to the most people whenever and wherever they are. And it’s all thanks to a shift away from thinking about the artifact to instead concentrating on the people using it.

Hook, Snack and Banquet — Mar 16, 2015

Messages and ideas that travel lightly are the most pervasive and often the most persuasive.

Ideas, innovations and even insights from data need help getting a foothold in any organization. But often they are trapped in the unwieldy mechanics of enterprise systems and their artifacts.

Restricted by both policy and inflexible design, their flow and adoption slows to a halt.

Billions of dollars have been spent on workplace systems specifically engineered to help people work together. From ultra-complex megaliths to newer, leaner services that focus on solving discreet workplace problems. Whatever their scope, the most successful and well used systems tend to be built on behaviors and activities, rather than on documents, files and archives.

By focusing on activities rather than artifacts, you get a shift in the way people engage with the system. It comes alive and becomes an ecosystem rather than a document graveyard. A place where the exchanges and interactions between people hold all the value. Encouraging this requires the removal of barriers in the creation, sharing and consumption of content. The most effective of these show very high levels of creativity, broad networks of users and simple methods for reuse and feedback. Think of Twitter — it’s not the artifact, a fixed formatted tweet, but rather the spread of it and the engagement with it that is valuable. This is also how we think about BI and data discovery in Qlik Sense. The focus is on the activity, the question asking and analysis, and the sharing of those discoveries rather than reports and document archive. The aim is to engender an environment of exchange, enabling people communicate persuasive ideas based on data, in forms that are easy accessed and reused. This is important because ultimately we use BI to persuade people to do something, to get them to act in ways that can improve our businesses.

For this to flourish we need to make the paths to action as friction free as possible. Often the things we use to discover great insights are complex data models and sophisticated applications and visualizations. But the way we package those up for others must not be overwhelming. In fact, it can’t be or it will fail. That’s why in many other systems you see poorly simplified views of dashboards for specific people, or even everything ripped out and recreated in PowerPoint. Unfortunately, creating that custom fix introduces new barriers when it comes to the flow of engagement.

To do this well, I’m going to posit a three step approach.

The first step is the initial hook, followed by the simple snack. But if it engages us, to be truly persuasive we need the depth, the banquet. Enabling a friction free flow from hook (the fact or insight), to snack (the visualization or narrative), and on to the rich banquet (the analysis and questioning) is key to persuading people to act. As I’ve mentioned before, if you bring people into the analysis activity, the results and insights are far more persuasive as there is greater trust. It takes a system that supports different types of users and the myriad of contexts and devices they use to enable this. That’s when behavior based systems are naturally at their most collaborative.

What may start with an idea and a link to a couple of simple charts can quickly enable a team to coalesce around that idea. By enabling them to work together seamlessly, in ways that suit each member’s capabilities and preferred device, we get engagement. As each member explores the idea, they add depth and sophistication through their own analysis practices, creating value through their input and collaboration. It creates an environment where good ideas grow up and innovative thinking can flourish.

Progressive Disclosure And The Art Of Having Just Enough — May 8, 2015

It’s keenly important to design for many users rather than simply the novices and experts.

Here’s a dilemma faced by technologists all the time as described by Nielsen Norman Group Principal Jakob Nielsen in a 2006 blog post:

Users want power, features, and enough options to handle all of their special needs. (Everybody is a special case somehow…)
Users want simplicity; they don’t have time to learn a profusion of features in enough depth to select the few that are optimal for their needs.

To resolve this dilemma interaction designers use progressive disclosure.

Progressive disclosure is a very powerful and effective technique that addresses this duality. When done well it helps technologies be more useful for a wider range of people. It’s a great way to shift the experience from people encountering a series of plateaus to a gradual continuum. That’s important as plateaus can restrict a user’s growth with a technology and create a novice vs. expert divide. Plateaus often appear in user interfaces in the form of ‘wizards’. These step-by-step, simplified choices (based on a richer underlying feature set) are great initially as they support the novice.

But unfortunately they often imprison them too. When the user needs to deviate or do even a little more than what’s prescribed, they are required to work in the space intended for the expert. That can feel like stepping into the cockpit of a 747: an environment that relies on years of training to be able to successfully understand the myriad of switches and dials. It’s too much and it becomes cognitive overload for all but the most well-practiced. That expert knows where everything they need is: because they’ve practiced and put the hours of work in to learn it.

The thing is, investing all those hours simply isn’t practical for most of us. It’s fine training experts, but if we want wider adoption and broader use, interfaces need to work for a lot more than just the chosen few. That’s where techniques like progressive disclosure can really help. They create continuums that support the learning and uncovering of the depth and breath of features in a way that suits the individual using the interface. It happens on their terms and can allow for varying degrees of knowledge with out the need to leap from novice to expert in a single bound.

Here’s a few examples of how we use it in Qlik Sense:

Displaying Visualizations — The responsive thinking in our charts uses progressive disclosure to help balance the amount of information shown. For instance, when the chart is small and part of a dashboard, the shape of the data is stressed and the supporting details removed. When expanded and given more space, we disclose the additional information and offer more features to help you work with the item that is now the focus of your attention.

Creating Charts — Here we use it to help people build a chart, by giving feedback and showing the pieces falling into place as the user adds or changes elements. It’s instant reward and feedback. It helps the user decide when they’ve done enough, without being overwhelmed by all the features and possibilities.

Adjusting Properties — This really is at the heart of the continuum idea. I can create a chart with drag and drop and it all works, thanks to automatically set smart properties. If I want I can then adjust a property and choose a different approach. If I’m more advanced I can adjust the nuance of that approach to perfectly match my vision.

Those are just three examples of how we use progressive disclosure in product design. The ultimate goal is to always have just enough choices, paths and tools to hand at any given time, whether you are novice, expert or anywhere in between. Creating an experience that enables you to effortlessly get into a state of flow; and as if by magic, the perfect item always pops up. You don’t fumble around for what’s next and conversely if you’ve done enough you don’t have to do any more.

Our Design Philosophy At Qlik — Jun 8,2015

Why is conceptual thinking such an integral part of product design?

I’m a great believer in the importance of conceptual thinking when creating new products or designing new systems. By that I mean defining ideas and notions that can help frame the why, what and how of bringing a product to life.

They focus the vision, giving innovation a rudder to steer by. Sometimes these concepts are created upfront and laid out in detailed strategy models, in other instances they start loose and coalesce as the project unfolds. I’ve always found the latter works best, as it’s only by working something through that you really understand it and its implications.

Over the course of researching and designing Qlik Sense, four concepts quickly established themselves. These evolved as we learned more about how people using our software behaved and how our vision aligned with their world and needs. Together they act as a philosophy to guide us and to ensure that the features and ideas we come up with fit together into a cohesive experience. One that supports our overall intention of increasing the access to and usefulness of data analytics.

Here they are:

Keep it natural — The best tools fit us, extending and enhancing our skills in ways we understand instinctively.
Natural covers many aspects from usability thinking, such as keeping things simple and removing cognitive overload. It’s people-centric and is what enables products to be intuitive so that you know immediately how to work with them.
Encourage flow — When we are focused and in a ‘flow state’: we are at our most effective and creative.
Flow in a product can be many things. It can be the movement and navigation around the product, the steps taken to fulfill a task or the stream of interactions with a person. However, the ‘flow state’ is a psychological phenomenon that is personal and unique to an individual. In this state, actions become second nature and we are filled with confidence, we are ‘cooking with gas’.
Embrace dialogue — Debate and discussion brings us together and adds energy to our actions, nuancing our understanding.
Dialogue is the exchange, the feedback and the conversation. It is essential to understanding and to collaboration. It’s more than the conversation between individuals or a person and a system, it’s also the conversation in a group. It’s a mix of telling stories, active debate and most importantly listening. This mix is key to our success, as together we can understand an idea, spot a threat or opportunity and even share an experience.
Consider context — The time, the place, the situation and the participants change how we interact and experience things.
Every experience we have is shaped by the moment in which we have it. That moment is influenced by environmental factors, the sum of my experience to date, what my expectations are and whom I’m with. Understanding the where, when, why and how of an experience is essential, as it helps us empathize with people and enables us to design for their world.

Each of these concepts help us to ask questions and interrogate our thinking. They enable us to query our ideas from different perspectives, be it the usefulness of a capability, the impact of a requirement or the potential of a prototype.

I’ll be going into more detail on each of these over my next few posts. Stay tuned!

Keep It Natural: Design Philosophy Concept #1 — Jun 9, 2015

The innateness of natural skills and behaviors makes them immensely powerful as we can wield them without significant effort.

The best tools fit us, extending and enhancing our skills in ways we understand instinctively. We say things are ‘a natural fit’ when they work perfectly together.

When we talk about ‘natural’ we aren’t talking flora and fauna, it’s about intuition and making use of the skills we inherently have, rather than demanding we acquire new and esoteric ones. Many of these natural skills and behaviors are not front of mind, they happen on a semi-conscious level. Their very innateness makes them immensely powerful as we can wield them without significant effort.

Part of our mission at Qlik is to enable everyone to make better decisions through the use of data and analysis. To do this it’s essential to put people and how they behave at the heart of the tools we design. By keeping things natural we can design to be more inclusive; making our software more accessible to a broader range of people, not just the experts.

In the past we’ve spoken about ‘Natural Analytics’. This is one of the ways the natural concept has influenced our thinking. By identifying and working with our innate analytical behaviors such as association, comparison, anticipation and narrative, we can enable people to easily explore complex data. This is one of the ways we practice people-centric design. Rather than relying on a prerequisite knowledge of the ‘dark arts’, we ask “what do people already know how to do?”

This natural thinking has long been at the heart of people-centric design. A great example of this is ’affordance’. It’s a phrase coined by Donald Norman (Nielsen Norman Group) and it means that a person has an innate understanding of how use a thing by the way the thing manifests itself. It’s the same way we naturally recognize a pattern, we naturally recognize that a chair is to be sat on and a lever is to be pulled. This understanding is a blend of both learned behaviors and intuition. It is reinforced in the positive that this is indeed what it is for and how it works. Using affordances based on expected behaviors increases the intuitiveness of an interface, it simply makes it feel more natural.

Gaining an understanding of what’s natural is about observing people. Watching how they reach out and their first touches to understand their emotions and reactions. By observing people using our software early and often we can ensure this natural ease of use is deeply rooted in the tools we create. If it feels right, then the path to sophisticated and habitual use is much smoother.

By keeping it natural and rethinking the steps needed to work with data we aim to smooth the learning curve for analytics. Making it a little more human, a little less daunting and a lot more accessible.

It’s a journey that builds on the confidence and natural understanding of those first touches. It’s what leads to the ‘flow state’, which I’ll talk about in the next post.

Encouraging Flow: Design Philosophy Concept #2 — Jun 10 , 2015

Encouraging flow as part of your design philosophy helps people be amazing!

When we are focused and in a ‘flow state’ we are at our most effective and creative. It’s that sensation when time and the rest of the world drops away.

A feeling of being absolutely focused and working on something with a near uncanny ability; when everything is seamlessly falling into place. Flow, as a psychological state, was originally proposed by Mihaly Csikszentmihalyi back in the 1970’s but references go back much further. It’s often associated with great artists, musicians and scientists. They were capable of losing themselves for days while gripped by the creative fever. But thanks to Csikszentmihalyi’s work, we know it’s not some exclusive religious experience, it’s a state we can all experience. It’s also something we can design to encourage and promote, as when in this state we can accomplish amazing things.

Getting into the flow requires a range of things to happen.

The activity you are engaged with must be challenging and require skills to achieve the goals you set. The key here is the interplay between the challenge and your skills. If it’s too easy, flow doesn’t happen as it becomes boring or menial. You tend to get distracted. If it’s too hard, you struggle and the need to get help interrupts the activity which creates no flow. Designing for that sweet spot, the point where skills and challenge are closely aligned is very tricky, as it’s ever-changing and varies from person to person. By designing experiences as continuums rather than plateaus we can smooth the steps in the required skills. For example, replacing the jarring experience of being asked to leap from novice to expert, so prevalent with UI ‘wizards’. A continuum enables individuals to develop their skills as they choose to take on new challenges. A progressive disclosure UI approach greatly enhances this. It reveals the depth and sophistication of the platform as the individual decides to take on the next challenge, to do more, to work at a deeper level.

Distraction and interruption are major barriers to achieving a state of flow. Being able to focus on the task at hand is key. By designing calm interfaces that hint and give information scent rather than flashing and colorful ‘suggestions’ we help reduce distraction. It’s the difference between knowing a chair is to be sat on (by recognizing its form) versus a big red sign above it saying ‘sit on this’. The sign interrupts and distracts: it’s an advert and it gets in the way of the goal.

Another key element to flow is feedback. When we speak about flow we often cite artists and craftsmen as examples of people in a state of flow. Here it’s their relationship with the tools and the medium that is key. It’s the feedback they get from the activity that hones their skills and draws them deeper into the state. The key with feedback is to be subtle enough not to distract yet nuanced enough to inform and guide the activity. When we design interfaces we work hard to create that same feedback between the user and the UI by continuously observing people using the software and shaping that exchange.

Encouraging flow helps people be amazing, and its core to our design thinking at Qlik. But there’s more to flow than just the individual’s experience. Csikszentmihalyi suggests it can also be experienced in a group. It’s possible for a collaborative experience of flow. Interestingly, it has been suggested that charts and data visualizations are incredibly useful tools in achieving this. They help engage and focus the group more efficiently. I’ll talk more about this collaborative experience in my next post ‘Embracing Dialogue’.

Embracing Dialogue: Design Philosophy Concept #3 — Jun 11, 2015

Harnessing the exchange, the feedback, and the conversation.

Dialogue is the exchange, the feedback, and the conversation. It’s the debate and the discussion that brings us together and adds energy to our actions, and nuance to our understanding.

We love to think of the lone genius, the auteur with singular vision and full control. But it simply isn’t true. Even those who appear that way are rarely working in a vacuum. They inevitably have their collaborators, muses and mentors. The debates, discussions and arguments they have with these people are key to achieving great work. They ensure quality, focusing and strengthening the idea. They provide the rigor. I recently watched the great designer Paula Scher presenting the visual identity she designed for The New School. She is the designer but she’s not alone, it’s the interplay between herself, her team, the agency (Pentagram) and the client(s) that ensured the delivery of a great project. As a member of the client team put it, “one of the greatest tools is dynamic discourse… a really robust dialogue that pushed… the design to be a great as it can be”.

By incorporating dialogue into our systems, we help support the feedback loops that happen when people debate and discuss things. This helps people iterate, challenge and defend ideas. It helps them build persuasive fact based propositions. It’s an essential part of the decision making process.

Dialogue can take many forms, from human conversation to human-to-machine interactions, and even machine-to-machine exchanges. What enables the success of these exchanges is shared understanding. It helps when we all know what we are talking about, who’s talking, and when to open our mouths. This matters just as much in a group as it does between two people, and the simpler the common language is, the more accessible and open the dialogue. To support dialogue we use the following three cornerstones:

Artifacts — These anchor a conversation and give a group a common reference point to base the discussion on. They help us understand what we are talking about and can be easily referenced and pointed to. Whether that’s a chart, a tool or a number, it’s essential that we all can see it and if necessary challenge it.

Trust — It’s important to understand who you are talking to and the authenticity of the thing being discussed. Feeling secure that the conversation is private or contained to a ‘safe’ group can be essential for healthy debate. Equally, when things are opened up to a wider group, the group should be able to trust the source of the information.

Timing — Knowing when to shut up, when it’s time to engage and when it’s time to interrupt is the art of good conversation. Getting the feedback right is essential to ensuring a conversation flows, and that all parties can remain orientated in the discussion.

Those three are valid for all dialogue, be that human-to-human, the UI or the APIs. Through them, and the debate they support, we can strengthen our understanding of the data and insights we share. Helping us to step beyond blind faith to active engagement framed by the moment we are in. And as you’ve probably guessed, the next piece will be about that moment and the importance of always ‘considering context’.

Consider Context: Design Philosophy #4 — Jun 12, 2015

By always considering context we can better understand the where, when, why and how of an experience.

Every experience we have is shaped by the moment in which we have it. That moment is influenced by environmental factors, the sum of our experiences to date, what our expectations and intentions are, and the people we are with.

The influence of environmental factors on experience is easy to understand. It’s why there’s currently a shift away from designing solely for mobile to designing for context. This shift recognizes that it’s not simply about the technology or the device. As the technology becomes more invisible and ubiquitous, it’s our understanding of the situation that becomes essential. This requires a step beyond simply saying that ‘they are smartphone users,’ to modeling and observing real scenarios full of personal and social nuance. These scenarios surface how the environment, not just the device, colors the experience and shows where history and intent can subtly steer how a person feels in that moment. Understanding this takes deep empathy and paying attention to the people you interact with. Which, after all, is the essence of good user research.

The when and the where

The when of an interaction is incredibly telling. Times of day, work cycles, as well as frequency and depth of interaction, all impact the experience. What’s more, they vary from person to person. Great experiences feel like they are crafted especially for you. They fit the moment. When we combine this with other environmental factors, such as location or velocity, we can craft the experience even further. This creates new opportunities for delivering the right information, in the best format, at the optimum moment. By thinking about the when and where of an experience we can design flexible and responsive systems that fit with diverse needs and interaction rhythms.

History and intent

Each time we engage with a thing, we bring to it the sum of our experiences. There’s an old UX adage, “users use best what they use most.”* It’s why we talk about muscle memory, design patterns and learnt behaviors, because learning new ways to interact can be hard; our existing habits can block us. However, we can quickly adopt things that ‘feel’ natural.’ Intuitive design is the result of leveraging common understanding and the natural behaviors that fit the activity at hand. This combination of history and intent frames the when and where of the activity, giving us the context.

By always considering context we can better understand the where, when, why and how of an experience, helping us to empathize with people, and ensuring we design for their world.

So that’s it, a brief overview of the thinking that underpins the things we make. Not a set of rules, or design guideline but four simple concepts:

We use these to frame the problems, opportunities and solutions we discover throughout the design process. In return they help us create software that works with people rather than against them.For an introduction to all four concepts take a look at “Our Design Philosophy at Qlik”

*this is a UX twist on the now famous quote from Zuzana Licko: “readers read best what they read most” (Here’s a great article looking at the intricacies of how we read — http://alistapart.com/article/how-we-read)

My Data, Your Data, Our Data — Jul 6, 2015

We recently attended the Quantified Self Conference in San Francisco and that got us thinking about data ownership.

A couple of weeks ago I visited the 2015 Quantified Self (QS) Conference in San Francisco. It was a great event, packed with hundreds of talks by interesting people sharing how they are using data to better understand and change the way they live.

Although we don’t often talk about it, here at Qlik we have a lot of quantified selfers. For many of us, the Qlik Analytics Platform is our go-to technology when it comes to analyzing all our data, whether that’s for work or play. From tracking wine or beer appreciation to fitness and managing the data from multiple accounts and devices. Lots of Qlikkies have created apps and dashboards to be better at the things that matter to them.

Quantified self activities are broad and very diverse. It ranges from tracking steps and sleep quality through to looking at your microbiome and blood markers, or even analyzing the emissions of electromagnetic frequencies in your apartment. Much of it is focused on personal health and self improvement, and it’s the blend of devices, sensors and data that makes it a true instance of the Internet of Things. The range of skills and expertise needed to understand the data varies immensely and community sharing makes it possible. It opens up new and exciting ways to analyze your life. Some people are building sensors and devices, others hacking their data together, while more and more people are creating services to help everyone get involved.

At the QS Conference I heard stories and saw examples of how people were tackling many of the same problems that Qlik Sense helps us solve. Almost everyone there had realized that a single data set wasn’t enough. It wasn’t until they started combining different data sources that they started to get real insights — human problems are rarely one dimensional. Finding ways to combine and explore data from differing sources is key. Of course this is what the Qlik Associative Indexing Engine (QIX) does best. That’s why I can have an app that takes my fitness data and smashes it together with social data, and I’ll be adding weather to that soon via Qlik DataMarket. It’s also why the Apple health data categories are so broad (even if not quite broad enough just yet). Being able to see and explore the whole story of you in your data is essential to understanding and improving your life.

One of the most contentious and interesting areas around in the QS community is in regards to ‘my data’. For many, this data is very personal; it’s very revealing and sensitive. The idea that your data is positioning you as an outlier or abnormal to the population is also a hot topic. Some people are choosing to do QS in ways that aren’t part of larger services and communities. This could be seen as the ‘selfish’ side of QS, but the sharing ethos is so strong in QS that it’s more about being open to diversity in the data and rejecting someone else’s idea of the modeled norm. The community knows that sharing data and practices is how other people really can find their own insights, and so collaboration is key. These activities enable people deal with real issues in incredible ways, whether it’s a small group pooling their data on an understudied illness or the mass services with large-scale data sets that may help the greater good. Services like uBiome and 23andme are building up broad data sets that they are making available for scientific studies. Something that the Apple Health is also designed to support.

The quantified self is not selfish, it’s citizen science done in a way that engages and matters to the people taking part. The phrase “Quantified Self” was coined by Kevin Kelly and Gary Wolf, and is already 8 years old. The QS movement now has an international conference and a growing global community of enthusiasts, startups, products and services. Many people may not consider what they’re doing as part of the QS trend, but today millions of us are tracking our steps and activities. As this develops more and more are seeing self knowledge through data as the way they will interface with health care professionals (and insurers) in the future.

Exciting and revealing times lay ahead. So grab your data and see what the story of you is!

External Resources:

If you want your iOS Apple health data — here’s an app from the folks at QS labs which will pull it out to a CSV file for you: http://quantifiedself.com/access-app/app
For more on the Quantified Self check out their main website here

Quantified Self Qlikkies Examples:

Fellow Qlik blogger Michael Anthony creates an annual QS report available for viewing here on his website
For a plug-and-play RunKeeper example see this downloadable asset from Qlik Show blogger Patrick Tehubijuluw
You can also read more about how Northeast Solution Architect Manager Jason Yeung was able to better track his fitness using Qlik Sense in a blog post here on Qlik Community.

Space To Think — Aug 7, 2015

Creativity and collaboration is easily sparked when given more room to ideate.

Today we are beginning to hear a lot about glances, glimpses and micromoments. These are the very varied and short interactions that technology like smartwatches bring us. They are intimate, small scale, up close and personal, and all about the surfacing of small pieces of information.

At the other end of the scale is being immersed in your information. Where it’s about being part of that space and exploring it. Where you are working at a human, physical scale, rather than viewing the digital world through a keyhole. These immersive digital environments are created through multiple or massive displays and Virtual Reality. They are the sort of thing you’ve probably experienced via the teleconference room, or the Network Operations Center or maybe even your organization’s “insight room”. These places enable us to inhabit them, walk around, touch, point and explore.

When we think of analysis we often focus on the analyst. The single person locked into their PC, crunching away at the data in isolation. In a way the traditional PC or laptop forces that. It traps us in the two foot zone of you and your screen. But when we do other forms of thinking we tend to want to and are even encouraged to break free. We create ‘war rooms’, we use whiteboards, perhaps we even go for a walk. IN fact, a recent study from Stanford found that walking improves creativity and suggests that “a person’s creative output increased by an average of 60 percent when walking”.

When we want to think creatively we need to break out of our confinement. We need room to think. The ‘endless canvas’ has long been a popular metaphor in software, a space where you can lay up an ‘infinite’ number of items that can be viewed and perused. It’s an idea that reflects how we like to solve problems. We take the information we have and lay it out so that we can see it. It makes the most of our amazing visual analysis skills. This helps us to draw connections and tease out meaning. It’s our preferred behavior. So much so that wall space is often the most sought after resource in creative workplaces. In addition, it’s also how we work most effectively with others. We need spaces that we can gather in and be ‘present’ in. Where we are immersed together, telling stories, debating and pointing to artifacts. It’s why the workshop, brainstorm and conversation are so much more productive than the emailed report.

Working at different scales and on a variety screens influences how we experience data and information. The way we behave also changes as it reflects what we are comfortable doing in that space, in that context. This doesn’t change the analytic facts, but can change how we communicate them, how willing we are to explore or question them, or even how much trust we place in them.

So if you’re lucky enough to have an insight room at your organization or space with several large screens, take Qlik Sense for a spin in it. Thanks to the responsive design and the ability to use multiple browser tabs, it’s great for working collaboratively at scale, both up close and at a distance.

Let me know what you experience. Did it effect the way you worked, the way stories were told and debated, or even how the decisions that were made and recalled?

People Are Srmat: Data Literacy and Broad Audiences — Sep 29, 2015

Data literacy shouldn’t be seen as a barrier but as an opportunity.

I love this. It’s been floating around the web since 2003 and even though it’s a hoax (there is no Cmabrgde Uinervtisy research) it still manages to demonstrate just how advanced our literacy skills are.

When reading we not only process the letters but the word shapes, sentence structures and also apply our understanding of the context set by what we have previously read. It’s about legibility, readability and cognition.

This is the same for data and data visualization, although data literacy is more than simply being able to ‘read’ the data. To be truly literate with data we need “the ability to read, work with, analyze and argue with data” as written by Rahul Bhargava (MIT) and Catherine D’Ignazio (Emerson).

But let’s start with reading and seeing the data.

One of my favorite design adages is “Readers read best what they read most”. This makes total sense when applied to data and visualizations. Elsewhere on this blog, Patrik Lundblad takes us through best practices for the visual encoding of data. This looks at how we visually perceive values, and it’s a great way to tackle the legibility and readability of data. In addition to that, our familiarity with a thing has a strong influence of our ability to understand and make use of it. Often until we ‘have knowledge’ of a thing, we tend to misread. That’s why the safe choice for many audiences is the bar chart. However bar charts are sometimes seen as pedestrian and limiting to those who are a little more literate. So how can we support this diverse group, this range of literacy? Do we always have to bow to the lowest common denominator?

Well, yes and no.

Being able to read the visualization is the first step. However people learn and can improve their literacy very quickly if we help them. That doesn’t mean force-feeding them more complicated and advanced charts. As with many things, people will abandon the difficult and unfamiliar if the change is too great. It’s essential to help people grow at their own pace, to challenge and teach in understandable ways. If a more advanced chart is an extension of a known concept or is delivered in a context that people are familiar with, it is often quickly learned and understood. I spotted a great example of this during the BBC’s 2015 UK General Election coverage.

This traditional style map comes from the BBC’s online coverage (the live broadcast show used a hexagonal binned map). For years the various news agencies have presented the election results as a map of the UK using a standard geographic projection.

Each party is represented by a color and each constituency (seat) is shown using its geographic boundaries. With a general election, the color of the map plays a strong part in communicating the results. By using geographic boundaries some seats appear very small and difficult to see. This can make it a little tricky to visually assess what proportion of seats each party has won.

For the 2015 Election many news agencies (in this case The Telegraph) did something a little different. Instead of showing the precise geographic boundaries they switched to hexagonal binned maps, with each seat represented by an equal-sized hexagon.

This is a great approach as the UK’s constituencies are based on ‘equal-ish’ sized populations, thus rendering a better representation of the overall result. Of course this does mean that the UK looks a little odd, and Scotland (as usual) gets the raw end of the visual deal.

This is not a ‘basic’ data visualization yet it was used for a mass audience, with very diverse levels of data literacy. The important thing here is that it was meaningful, fit the context, and extended a concept that was already well-known.

It’s not about adopting the ‘lowest common denominator’; it’s about understanding the usefulness for a broad range of people. By carefully selecting and framing the data visualization we can introduce people to more sophisticated forms. As familiarity grows, these new forms enhance data literacy. So when considering these forms, think about how to introduce them. Ask yourself how can I frame this so that someone who has never seen this approach before will be able to read it without needing complex explanations.

The Second Age Of Information — Oct 26, 2015

Information may well be power, but the real value comes from our ability to extract meaning.

It seems that our reaction to the age of information is changing from ‘information is power’ to ‘OMG there’s so much information out there’. With this comes the realization that the primary value is not the data per se, it’s the literacy to read it and ability to work with it and skills to elicit meaning from it.

Information is not simply the data, just as meaning is not simply the information. It’s the connections, paths, clues and insights you can tease from it that become the new intellectual currency.

No surprise then that the data scientist is the hot job of the moment — the skill and knowledge to analyze and make use of data is far more valuable today than the knowledge of how to store data and manage it. But it goes beyond that data scientist role.

The first Age of Information was one of asymmetric information; a period when a select few held the information and many were kept in the dark, without access or means to use it. The data activists, the citizen data scientists, the quantified selfers are changing this, and in turn this is opening up even wider use. Today we are in a transition period where individuals have access to more and more information, but we are also generating more and more data than ever before. This is why data literacy is so important, it’s key in enabling people to understand, work with and transform data into meaningful and actionable information.

The way to shift the needle away from asymmetric information is for people to actively engage with data. It’s already happening in many ways. The first evidence came from access to information and the discussion of it — sharing what’s known and personal experiences. You see this in the marketplace where buyer and seller now have roughly the same level of information when it comes to product purchases. No longer do I have to ‘trust’ the seller, I can check if it is the best price, if it is reliable, if the features are as good as they say.

Another great example of this change is in our relationship to our general or primary healthcare practitioner. For many of us, when we visit our doctors, we come armed with information and knowledge. We research and focus on the specific things relevant to us and create hypotheses before we visit — to better or worse ends. This sometimes means that the knowledge about this specific situation often lays with us in the discussion rather than the practitioner. The ‘General’ Practitioner may or may not know specifically about X or Y condition. We may have also brought with us health data, such as heart rate or blood pressure measurements or even our microbiome. And of course a host of ideas about what that data is telling us. For many of us the doctor is no longer the oracle and sage of the past, no longer the guardian to a world of information that we are not privy to. Today, much of that information is also available to us. The trick is, knowing how to find it, read it, understanding it and apply it to our individual situation.

This also varies hugely from area to area, doctor to doctor. If the General Practitioners’ catchment is affluent and educated the chances are they will also be well informed. If not, then access to the information may not be available or the level of data literacy required may not be there. Asymmetry in information hasn’t disappeared just yet. But we can improve that, by improving data literacy. If we can build people’s skills in reading, working with, analyzing and arguing with data, we can embrace a second age of information that’s far more “evenly distributed”. One where we make better decisions by not only understanding the information we have but also being able to debate and interrogate the stories we are told.

The Dark Arts of Persuasion — Dec 7, 2015

Anytime we set out to persuade an audience there’s always a lot more going on than simply relaying the facts.

In one way or another when we set out to persuade we set out to sell. Whether it’s an idea, a fact or a falsehood, the goal is for the other party to buy it. One of the most powerful tools we use for persuasion is storytelling. And when it comes to relaying insights through data narratives it’s a must-have skill.

The key to data storytelling is in leveraging the hot engaging power of the narrative to support the cool, neutral facts in the data. The narrative helps us frame them, position their impact and stress the importance we have assigned to them. But more importantly the narrative enables us to string the facts together in a way that creates a coherent and persuasive argument. The thing is, as with most narratives, that argument belongs to the author and is often full of misdirection and dark arts. Whether it’s by deliberately leaving things out to help ‘clarify the message’, or making ‘best guesses’ when we don’t have the supporting data, or skipping over some data, or even simply changing the order of things to add drama and a more powerful finish. Our stories are crafted for an audience and make full use of that relationship. To tell a data story is like telling any story: it’s about setting up the positions of author and reader, performer and audience, or even preacher and congregation.

The more skilled we become at telling stories with data the more sophisticated our toolkit. We know what it’s like to be the consumer, the person on the other side of the story. They want to be engaged and have adopted the position of audience: “here we are now, entertain us”. In doing so they are open to all the tricks of the persuader, all those hidden needs and buttons that advertising and PR has been pushing for years. And this is where it can get even murkier. In our efforts to persuade, to engage, to communicate that oh so important idea we have, sometimes we concentrate more on the persuasion than on the reason we are there in the first place. After all, these situations in the business world are usually a team or group striving towards a common goal. It’s not some random megalomaniac attempting to persuade us that unless we believe that “variance in spend on X has a significant influence on long-term profitability” we will be doomed to be alone in the world.

The more we try and persuade through the dark arts the more likely we are to ultimately run into rejection and mistrust. The key is to not be stuck in the author or narrator mode, but instead open up the discussion. Embrace exploration of the idea together as a group, the story should be little more than a placeholder for debate. A starting point to explore alternative hypotheses, to check facts and even to change our own minds. Persuasion must not be brute force, it has to be a collaborative experience where ideas, proposals and actions arrive together. As Blaise Pascal pointed out almost 400 years ago:

“People are generally better persuaded by the reasons which they have themselves discovered than by those which have come into the mind of others.”

Thanks to Maria Popova’s brilliant Brain Pickings for the Pascal primer.

Here Be Dragons — Jan 18, 2016

What else is your map telling you?

When it comes to maps things aren’t always what they seem.

Whether we are using them to tell us about geography, location, place, process or for way-finding, maps are incredibly useful and sophisticated devices. But all maps require that you to know how to read them, and this ‘literacy’ can be surprisingly diverse.

Today we live in a connected world with the ubiquity of highly detailed and very granular online maps such as those from Google or Apple. However, not so long ago most of our maps were paper; you purchased your detailed maps in very small parts or you viewed larger amounts of the terrain but with reduced detail. The use of many maps and combinations of maps was the only way to understand the territory. Of course if you lived in Britain 50 years ago you could “ask a policeman”. Police walked the streets in the small area they patrolled and had a detailed mental map of that place. They knew the people, the businesses and activities. They also had a narrative for the place, a psychology of the space, a psychogeography created by the history, events, crimes, threats and stories of the streets.

A diversity of maps is important and comes from our need for visual, physical, mental and cultural meaning in relation to place. Below is a poster showing a map of London. It’s not a strict cartographic rendering, but rather an illustration of place. It shows the location of London’s design studios by describing them as if you were with a friend having breakfast in a greasy spoon cafe. It’s place-making combined with storytelling and context, and it’s a very human way of helping others to understand a place.

But maps also have agendas, they have power and they can include or exclude readers, peoples or groups. My colleague James Richardson has talked about the Mercator projection before, and how it is a deeply political map that has become our default ‘view’ of the world. Here’s another projection, this one is considered a more accurate view:

It’s the Gall-Peters projection (see this great clip from the West Wing arguing why we should use it over the Mercator projection). Except, it’s also inverted, to remind us that there’s really no “right way up” when viewing from space. In fact, our view of north at the top and south at the bottom is a construct, an ‘enforced order’. It’s how Northern Europeans decided to draw the world and thus it’s how we read it (even NASA has to rotate images of the globe so that they appear the ‘right way up’ to viewers). But it was not always so, even at the time of Mercator we had ‘upside down’ maps, such as the portolan chart below by Vesconte Maggiolo (1512). It was drawn from the perspective of the explorer rather than the authority of the state.

And more recently we have maps like Buckminster-Fuller’s brilliant Dymaxion map that strives to show the ‘connectedness and oneness of the planet’. As you can see, maps always have agendas.

Maps are created by the people who need them, whether that’s to navigate, define, control or communicate. Often they are drawn not just to help locate spatially but also to reflect many other attributes of a place. And sometimes the ‘exactitude’ of the spatial representation is subordinate to other meaning and messages. Or it simply doesn’t fit the cartographic language we may be accustomed to, like the maps of the indigenous peoples of America, Australia and Polynesia.

When it comes to maps, context is still the key. It shapes them and layers on meaning, sometimes hidden, sometimes overt. And despite Google Maps we still use many other types of maps, such as the tourists maps we pickup on vacation with their exaggerated landmarks, or the directions we give a stranger. Those personal maps reflect ideas about a place we hold in our heads, that when revealed tell more about us than the geography (see below).

The header image is Lewis Carroll’s now infamous ‘ocean-chart’ from The Hunting of the Snark. The map is a featureless, empty space reflecting the ocean and the unknown; a dark place full of dragons and monsters waiting to be explored and brought forth into the light. However, here’s another ‘map of the ocean’ this time from the Marshall Islands. It’s a ‘stick chart’. The shells represent islands, and the sticks represent currents and lines of swell. It’s the unknown and hidden made visible — which is surely the goal of any visualization.

A map is for an audience. When we create maps we must strive to understand who will be reading them. Is this a map for me, for us, or for them? Are they literate to the language of the map? When we use maps in data visualization our approach and the choices we make in the cartography and visualization does not only communicate the quantities, the metrics, but can also confuse and express a bias we may not be aware of.

Always ask yourself: “Will my audience know how to read this map?” and “What else is this map communicating?”

####

Image Credits:

Graphic Thought Facility: http://www.graphicthoughtfacility.com/british-council-work-from-london-poster/

By Strebe (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

http://www.bibliotecapleyades.net/mapas_antiguos/reanaissance/316.1.htm

Eric Gaba — Wikimedia Commons user: Sting

John Fulford, The Walk to South School, from You Are Hear: Personal Geographies and Other Maps of the Imagination by Katherine Harmon

Flickr: jmcd303

My Signal, Your Noise — Feb 19, 2016

What happens when we throw our data into the mix?

Personal data is exactly that, personal. When we view it we are looking at an individual, a person, someone who is wrapped in circumstance and context, it’s not just a simple data point.

Analyzing survey data based on demographic segments is one thing, but analyzing the incredibly granular data that comes from self-tracking and sensors is quite another.

Take the recent discovery that one Fitbit owner made. They spotted a sudden change in the data coming from their wife’s device, thinking it was a sensor error they asked the community about it.

“My wife’s Fitbit is showing her heartbeat being consistently high over the last few days. 2 days ago, a somewhat normal day, she logged 10 hours in the fat burning zone, which I would think to be impossible based on her activity level. Also her calories burned do seem accurate. I would imagine if she was in the fat burning zone she would burn a ton of calories, so it’s not lining up.”

Another user asked in response to the data “could she be pregnant?” and it turns out she was. This was very, very early in the pregnancy, completely out of sync with the useful healthcare rhythms and cycles.

https://www.instagram.com/p/BBnNheUh-g9/?utm_source=ig_embed

Here is the pic of the screen that started it all! Mom and baby are doing ok. Spoke to the doctor yesterday.

A photo posted by David & Ivonne (@babyfitbit) on Feb 10, 2016 at 8:36am PST

For now I’ll put aside the data privacy concerns and algorithmic opportunities this raises and instead concentrate on the shift this brings in understanding ‘my data’, in relation to the group’s data.

This was an amazing ‘signal’ in the data, but deeply individual and specific to the person. It displayed a recognized indicator for pregnancy, but one that is also an indicator for many other things. If the data had been part of a larger aggregated group it could have been seen as noise (especially if the additional pregnancy insight was unknown). If the data had been anonymous and maybe even had gender identifiers removed, then what? Would similar occurrences have skewed the group?

So how does my data fit with your data, and should it?

For years, IT departments resisted people using their own devices on corporate networks — but not anymore. Many of us now work in organizations that encourage us to “bring your own device”. The same shift is happening with data. We are beginning to hear more and more about “bring your own data”. In the same way as familiarity and expertise with a device improves productivity so it can with data and analytics. Of course the trick is balancing what must be governed and ‘official’ with what can be added to and augmented. Context, granularity and focus are key to knowing what the signal may mean at the level you are viewing the data at. For many of us navigating the depth and breadth of the data available will be the core skill for identifying those important signals.

This is the dizzying frontier the healthcare industry is facing, working from the extremely close up and granular view of the individual to the pulled back view of an entire populous. As electronic health records (EHR) become more sophisticated and connected, the more the view of the individual changes. The cadence changes and the EHR shifts from a historical document of record to a near real-time pulse. It becomes a useful diagnostics tool that we can augment with personal data supplied by the sensors an individual wears. That’s a lot of data, that’s a lot of messy, complex and sometimes unreliable data. But that is a rich and fertile ground for finding new signals and insights about groups and individuals. These layers; from official to informal, from individual to group, from clean to dirty, from exact to fuzzy, from past to present form the data-scape we are beginning to inhabit. Choosing the right frame for the data we have access to will be the key to drawing meaning from it.

The Art of Big Bang Data — Mar 15, 2016

Recalling the exhibits shown at Somerset House in London.

Last week Donald Farmer, James Richardson and I met with David Reed from DataIQ for a chat and wander around the Big Bang Data exhibition currently showing at Somerset House in London.

The exhibition draws from a diverse group of artists and designers, some who are data practitioners and others who are dabbling in data as a medium or as a means to support a bigger idea. It was a great to see some of the data literati alongside historical influencers and pure artistic endeavors. This created a great setting for inspiration and discussion. After perusing the history of data containers (punch cards, through to floppy disks and on to DNA) the John Snow map caused a little chuckle, as the description trotted out the myth of how it was used. This got us talking about narratives and how even a great visualization may be missing key data and often can’t avoid suggesting an idea or agenda. This was beautifully illustrated by the Charles Joseph Minard map of Napoleon’s Russian campaign. Although an instrumental map in the development of data visualization, it tells you nothing about why or how so many of the French troops died before reaching Moscow, but it does falsely imply the winter had something to do with it (but that’s for another post).

The pieces and topics at Big Bang Data range far and wide, but one of the intriguing things we found was how the screen-based (at least the small screen) projects were much less engaging than the tangible ones. The posters, prints and artifacts were a lot more compelling. Getting a chance to see the Nightingale Rose (the first polar area diagram) and Stefanie Posavec and Giorgia Lupi’s wonderful experiments in personal data vernacular with the Dear Data project was a real treat. There were a few large screen installations, but unfortunately none seemed to be interactive. They focused more on data as a medium for visual expression than on subject or meaning. It wasn’t that the screen-based projects were less interesting, it was the context in which there were presented. The Selfiecity project by Lev Manovich and Moritz Stefaner is a fascinating project, but when reduced to a small monitor in a large space, access proved difficult when part of a larger group in an exhibition setting. The exhibition context meant that all screen-based interactive projects needed large scale touch screens to get me engaged. Let me play and more importantly let me play with others. The video-based storytelling worked well as to sit and watch a screen is an established activity in a gallery space. This passive consumption of a story or data narrative via a visualization plays particular well to infographics like those of David McCandless, where his agenda or idea is core and there’s no opportunity to challenge it.

Seeing data being used in these diverse and inspiring ways makes me all the more convinced that we must give people the opportunity and means to question and explore the data we serve up to them. Presenting data as unquestionable artifacts set in stone (or paper, or PDF, or PowerPoint) that are intended to prove beyond a doubt what you are saying is correct relies on acceptance and trust from those consuming it. But I’m not a fan of blind obedience, as it’s open to bias and deceit. If as Rahul Bhargava and Catherine D’Ignazio suggest “data literacy includes the ability to read, work with, analyze and argue with data”, then we have to do more to help others engage in the debate. That means opening our closed arguments and enabling exploration and the means to question what we say the data means.

Big Bang Data was well worth the visit as underlying many of the pieces was a passion for the democratizing of data. I’d just liked to have seen a little more of that for the audience.

Images: James Bridle “Fraunhofer Lines 002” (2015), Morag Myerscough “Big Bang Data”, David McCandless “Debtris” (2010).

Designing for Data Literacy — Apr 18, 2016

Access to data is one thing, being able to successfully work with it is quite another.

A couple of weeks back I had the pleasure of talking to Catherine D’Ignazio about her work with Rahul Bhargava on data literacy. What makes their work so important is how it re-frames working with data as a cultural need and not merely a technical, mathematical or scientific activity.

And they are not alone in this. Back in January 2015, President Barack Obama said:

“… we want every American ultimately to be able to securely access and analyze their own health data, so that they can make the best decisions for themselves and for their families.”

This will take a lot more than simply making data available. It will require a massive and widespread improvement in data literacy. One that necessitates a cultural shift that makes working with data a commonplace activity. Businesses are already acutely aware of these challenges. Often the data literacy levels within an organization are very diverse. But as we know, working with data and using it for decision making is an essential skill for all parts of the business.

Here’s a definition of data literacy:

“the ability to read, work with, analyze and argue with data”.

This is from the paper Designing Tools and Activities for Data Literacy Learners by Catherine D’Ignazio and Rahul Bhargava. Their approach is to look at it not as the ability to use a specific tool, but to promote ways that encourage learning and improve data literacy. They suggest four design principles for supporting learners when building data tools and assets, they are:

Focused — drives to do one thing well

Guided — introduced with strong activities to get the learner started

Inviting — introduced in a way that is inviting to the learner

Expandable — appropriate to the learner’s ability, but also offers paths to deeper learning

If we think about this from a BI perspective it’s plain that one report, one chart or one fixed dashboard for everyone who may need the information simply doesn’t cut it. Often it may require a variety of entry points to the data, ones that support the range in the skills of those who are using them. The trick is to ensure that everyone can read the data clearly, understanding any immediate actions, able to make judgments based on what’s shown and understand what messages and stories are underlying the data representations.

The ultimate goal with data literacy is to support critical thinking about any data or presentation a person receives. This is shifting the individual from consumer of statements to questioner, debater and explorer. For me, this means thinking about data assets in terms of services rather than documents. Using them must be a progressive experience. They may start with simple, basic charts and defined processes but lead through example and engagement to open data discovery and analysis. Here statements are questioned, facts verified, and new hypotheses brought forth.

When designing for data literacy, think about the principles that D’Ignazio and Bhargava put forward. Set a context and focused use for the assets you create. Don’t build sprawling, complex apps that switch states or work with multiple concepts across unrelated problems. Ensure that any steps needed in the analysis process are clearly signposted. Be wary of forcing people through a flow. Where possible allow for them to find their own path, even if that means bypassing your intentions. Keep the content meaningful and make sure that it’s inviting and not overwhelming for initial use. Don’t simply dumb it down, find ways to reveal the important methods and ideas in the analysis as well as surfacing what’s happening. Consider using approaches like natural language explanations of chart data to communicate information about a chart alongside it, thus helping learners spot the sort of information they should be looking for.

Most of all remember to leave things open. Find ways to encourage people to explore and question the data further. Give them a means to create fresh interpretations and help them voice their findings. Because ‘arguing with data’ is the key to better decision making. To be truly data literate we need to be able to use it both in support of our own ideas and to reveal the flaws in them. That’s how we get to better decisions, made by more people, more of the time.

For further reading on data literacy, here are a few links:

Designing Tools and Activities for Data Literacy Learners — Rahul Bhargava & Catherine D’Ignazio

http://www.kanarinka.com/ — Catherine D’Ignazio

What Would Feminist Data Visualization Look Like — Catherine D’Ignazio

Talking Visualization Literacy at RDFviz — Rahul Bhargava

Pick a Card, any Card: Nudging your BI strategy — May 16, 2016

A fun tool to help stimulate creative approaches to BI.

Let’s face it even if you’re using the best technology on the market when it comes to BIand analytics it’s easy to fall into the “that’s how we’ve always done it” frame of mind. It’s the reason so many organizations still produce the same KPI reports for years, or get stuck with partial adoption of BI and a persistent Excel culture.

It’s why you see managers prioritizing gut feel over data when making decisions, or have some teams that simply get data and others that simply refuse to. Some even fail to recognize that their staff’s data literacy needs work to get the most value from their information assets.

So how do you shake things up?

Where can you find the stimuli when you need it most?

A couple of months back James Richardson and I started looking at how we could distill some of our ideas for challenging traditional BI thinking into a useful and fun resource. The result is a deck of strategy cards that we previewed at Qonnections — if you asked a question after our presentations you might have been lucky enough to be given a deck.

The deck; Data, Decisions and Design — 48 nudges to push BI strategies further consists of a set of 48 cards split into 4 suits. The individual cards deal with a discreet idea designed to help you challenge outmoded approaches, received wisdom and entrenched BI beliefs. Our intent is whether you’re planning a fresh analytics project or reviewing your strategic use of data: reach for the deck and see how you could approach things a little differently. For instance, when considering an analytics challenge like:

How can we move beyond reporting to being more analytic in our use of BI?

Simply draw a couple of cards from the deck to kickstart your thinking…

These aren’t a formal set of methods or techniques but rather they offer hints on how to push the analytics culture of your organization a little further. This way, rather than trying to force yet another new ‘process’ or set of methodologies, you can take the ideas and work with them in a way that suits your situation and pace.

The Power of Lists: A Listicle — Jun 15, 2016

In or out, first or last, top or bottom: inclusion, rank and order are just part of the power of the almighty list.

People love lists. They help us to order the world, prioritize what matters most and remember what needs doing. But lists have many interesting properties and implications, especially when it comes to data and decision making.

One of my favorite lists:

Jorge Luis Borges’ list of animals from the fictitious book Celestial Emporium of Benevolent Knowledge. The list challenges us to try and fathom its taxonomy and reminds us that all classification carries with it a point of view.

a) those that belong to the Emperor,
b) embalmed ones,
c) those that are trained,
d) suckling pigs,
e) mermaids,
f) fabulous ones,
g) stray dogs,
h) those included in the present classification,
i) those that tremble as if they were mad,
j) innumerable ones,
k) those drawn with a very fine camelhair brush,
l) others,
m) those that have just broken a flower vase,
n) those that from a long way off look like flies.

Why we love lists
When it comes to processing information, knowing what you’re working with makes all the difference. The list format is a model we understand and know immediately how to read. We can scan it quickly, and that’s attractive as it feels like an efficient way to take in information. Lists also have a built-in imperative that suggests importance and a definitive set of information, not to be missed. They appeal to our curiosity and we simply have to know who’s top, bottom, first or last. We also can’t help our own assumptions: guessing who or what made it onto the list, and if we’re right that makes us smart, right?
Top # — where to draw the line
Is that a “Top 5”, “Top 10”, “Top 20” or “Top 100” list? Where we decide to cut the available data matters. As soon as we reduce and exclude items we reduce available information. The “Top 10” list is the king of “Top” lists. A quick googling for it shows that its popularity is way ahead of the rest, on page hits alone.

It’s been suggested that this is closely tied to our cognitive preference for round numbers. Truncating results is useful as it supplies a manageable set of information. But just missing the Top 10 means that perceptually an item falls into the next tier, thus is more easily disregarded, even if it is still a valid candidate. One study found that the largest perceived vs. actual differences in value was between items ranked 10 and 11.

Be careful where you draw the line, if you want to use a “Top” set, then where possible cut the set based on how significant the information and difference is for each item. And always state what the decision is based on. After all a “Top 13” list, may not be popular but it could well be more meaningful.

3. Whose ranking is it anyway?
Being first, numero uno or at least in the top 3, makes all the difference, whether that’s in the Olympics or Google’s search results. But those positions are dependent on how we choose to sort the set. A list of names naturally gets ordered A-Z but if another value is used to rank them (say a score or location or other associated value) the position of the name will change, layering on the meaning associated with the other value. But that meaning is only relevant in that context. It’s important to be transparent about what is determining the sort order, don’t leave it unstated. Better still, give people the capability to apply their own criteria to sort the list and help them see the data in ways that may matter to them.

And we can’t forget how incredible being able to sort and resort a list is. Imagine hand-sorting 1000s of items! The mechanics of programmatic sorting are fascinating, here’s a geeky example showing how quickly different sort approaches perform: http://www.sorting-algorithms.com/.

Most of the data we work with is held as collections of lists in the form of tables that contain rows (list items for the table) and columns (a list of associated attributes or characteristics for an item). When we present or work with data the way we present those lists also carries meaning. It frames how that information is processed. So next time you create a Top 5 for that dashboard, take a few extra moments to think about whether you might be implying a significance that isn’t really there.

Lost in Translation — Jul 18, 2016

Your dashboard design needs translating just as much as the labels.

The way something looks has a profound effect on how we understand and process the information it holds. Advertising, packaging and information design has been exploiting this for decades.

No matter how much we information designers strive for neutrality, we can’t escape giving information a particular voice when we design it. Neville Brody may have been being ironic when he said: “If we were honest we’d only ever need one typeface.” But that realization that every typeface and every design decision carries with it a motive and ultimately an implication on how a piece of information is understood is more relevant today than ever.

The ‘design voice’ of a piece of information varies depending on the stylistic whims of the times and the viewer’s level of visual literacy. Both of these are framed by current cultural trends and the viewer’s historic cultural understanding. The understanding I personally have of aesthetics is framed by my western upbringing. The modernism, white space and ‘clean lines’ I love and think of as a clear authoritative design voice is a western cultural construct. When it comes to internationalization and localization, just porting a design from one culture to another is unlikely to be effective. Often it will under-perform and in some cases even miscommunicate.

The thing is, it’s not just the big things like how in Eastern societies people are more likely to rely on holistic reasoning than in the West, where analytic reasoning is move prevalent. As studies have shown, this can effect not only the mental models we have to help us organize and categorize the world, but even the way we perceive it. Or how about the fact that even our spatial understanding of time is not globally shared. In English, time flows from left to right — thanks primarily to our written word. In some other societies time flows from top to bottom (again referencing their written word) and in others it’s more closely linked to the passage of the sun or the cardinal directions.

It’s also the little things that catch us. Even the basic building blocks of a designer’s toolkit can be complex little beasts just waiting to turn and bite you. Whether that’s the availability of a typeface, and as Errol Morris in the New York Times online showed us, that can have a huge impact on a reader’s likeliness to believe or trust a piece of information. Right on down to the colors we naturally reach for when we want to express loss or growth or love or festivity. There is rarely a one choice that fits for everyone. The most common example of color tripping up information designers is in finance. For western stock exchanges and money markets the index grids and price tables often use red for negative, meaning loss, and green for positive, meaning growth. However in China it’s the opposite. Red for growth and green for loss. Red in Chinese culture signifies good fortune and financial prosperity.

As with all design and communication, in the end it’s all about people; knowing them, what motivates them, what frames their activities and what they understand.

There’s an old marketing anecdote about the Coca-Cola executive that tries to communicate the benefits of their drink in Saudi Arabia. He lacks Arabic language skills so he decides to convey the benefits as a pictorial story laid out through three posters, like so:

But no one told him that Arabic is read right to left.

So when porting that dashboard to the next market, take the time to test it with the locals — as the things we find second nature are always the ones to catch us.

The (un)Natural Language of Analytics — Aug 15, 2016

For self-service BI to work it needs to be accessible to the many, not just the few.

Mathematics and statistics are (thankfully) devoid of many of the idiosyncrasies of human language. As you’ll often hear, there’s a purity in the numbers. But numeracy and data literacy vary widely, both in society and within our organizations.

For many of us, the spoken and written word is still how we best understand information. It’s the descriptive power that turns cold, hard data facts into meaningful messages that people can communicate and act upon. And that takes skill. You need to be able to ‘read’ the data, (or visualizations) and then articulate what they are ‘telling’ you. That’s what makes a great analyst — not just discovering the insight but also communicating the abstract in an understandable way.

As we know, data visualization is a powerful medium for both revealing and communicating data facts. Unfortunately it relies heavily on the viewers ability to read, process and understand what they are looking at. Much of the problem with the lagging adoption of analytic practices is that the tools and data artifacts are rarely designed to help people in that endeavor. At best it’s designed for the trained analyst, which is great, if you’re a trained analyst. But data-driven decisions need to be made by a wide variety of people with varying skills. For ‘self-service’ BI to work it needs to be accessible to the many, not just the few. That’s where artificial intelligence and natural language generation (NLG) can help out. NLG in this scenario, is the ability to automatically communicate analytical observations in clear, understandable language.

The guidance that NLG brings to a data visualizations, is in the form of a descriptive layer and can be extremely useful for building data literacy. If we consider D’Ignazio and Bhargava’s 4 principles of designing for data literacy; it can supply focus for the reader by highlighting the data facts of importance. It can guide the reader by surfacing the sort of insights that can be drawn from the data visualization. It’s inviting because it uses terms and phrases the reader understands and it’s expandable because it can help introduce advanced and very sophisticated visualizations in a way that helps people to learn how to read them. Of course the impact and usefulness of NLG goes far beyond improving data literacy. It can support a wide range of ideas for creating people-centric data experiences.

All this fits nicely into how we approach designing for analytics at Qlik. The Innovation and Design group has been busy for a while now exploring the benefits of NLG and creating people-centric ways to analyze and work with data. Qlik was the first BI company to debut NLG as part of an integrated and interactive dashboard, through our partnership with Narrative Science.

Our partnerships with Narrative Science and YSEOP Savvy over the past year have led to intuitive and powerful natural language extensions that can be used to add narratives to data visualizations in Qlik Sense apps. Best of all they are tied to Qlik’s associative model so that they rewrite their narratives instantly the moment you make a selection, asking a new question.

Our journey with natural language started with the early work on the search capabilities in Qlik Sense. The latest release of Qlik Sense demonstrates how we are moving ever closer to naturally interacting with our data, creating a genuine dialogue between us and the data. And that immediacy and fluidity makes all the difference when it comes to getting the best insights for decision making. After all, a conversation beats a monologue every time.

Volume, Variety, Velocity — Nov 14, 2016

What do the Three V’s of data mean 15 years after their introduction by Doug Laney?

When Gartner’s Doug Laney introduced the 3 V’s of data (volume, variety and velocity) back in 2001, he framed the need for machines to step in and help us understand our data.

As any or all the V’s increase, data rapidly starts to escape human abilities to process, work with and react to. We need intelligent machines to help lift the cognitive burden of working with these massive datascapes, to help us with the high speed decision making that is now part of everyday business practices.

Andrew Ng pointed out in a recent Harvard Business Review article how artificial intelligence (AI) or machine learning can assist businesses today:

“If a typical person can do a mental task with less than one second of thought, we can probably automate it using AI either now or in the near future.”

We can have machines step in and take over these simple cognitive tasks in the same way we created mechanisms to take on physical tasks. But as the task chains get more complex, decision dependencies creep in. Handing over decision control to algorithms accelerates decision making, but with that speed and scale the impact of errors and biases can be magnified. We may have handed over the control of the decision, but rarely the responsibility for the action or effect.

At the end of the day it’s people who feel the impact of errors and it’s a person who is held accountable. Madeleine Clare Elish calls these situations ‘moral crumple zones’. It’s the point where we have handed control over to the machine and it fails, leaving the human to absorb the impact of that failure. These can be small or large, from the human customer service representative that has to ‘soak up’ the emotions of a customer where an automated decision has gone against them — “computer says no”. To the mass misidentification and misclassification of individuals, revealing underlying bias in the model. Or even the non-driving, ‘driver’ held responsible for an accident caused by a ‘self-driving’ car. If we continue to use intelligent machines as high velocity, automated decision systems then we must build in oversight and the ability for humans to act and intervene if we are to avoid being caught in the moral crumple zones of the models.

But what if we didn’t think of AI as the unquestionable voice of logic or a way to relieve us of decision making. What if we instead saw it as a creative tool? Recently there has been many reports of machines creating art. But in almost all these cases it’s really that the machine has applied a process or technique to create a set of artifacts that reflect an artistic style. Even with the most elegant and sophisticated of these examples the machine does not drive the work. The machine did not ‘decide’ to initiate it. The human has the creative intent, sets the project in place, trains the machine. Art and creativity are ultimately about intention. That’s why even when we want to hand over our decisions to machines, the impact of the decisions must be our responsibility as the intent was ours in the first place.

So how about focusing on that intent? After all that’s why new, interesting, challenging, innovative things happen. Let’s consider leveraging the machine for its ability to explore and iterate around a problem, not simply supply a binary answer. There’s been some amazing work in this area, often called generative design. This is where a discreet problem space is explored algorithmically at the hands of the designer. A great example of this is Autodesk and Bionic studio’s redesign on the Airbus 320 partition. The design team used algorithms based on natural processes to generate and iteratively adapt to improve on the partition structure; creating a wall that’s 45% lighter than the original and still achieves the same load bearing strength. It’s that active engagement with the machine as a tool rather than a black box decision switch which makes it interesting.

Finding ways in which we can use intelligent machines, not to admonish us from decisions, but instead help us to rapidly explore new ways of working and fresh ideas is a far more interesting future, than simply being left accountable for the mess.

Data, Time and Trust — Jan 16, 2017

What does it take to build a digital archive?

I’m a graphic design geek and recently I’ve been spending time with the Letterform Archive in San Francisco.

The archive consists of over 30,000 items spanning 2,000 years of lettering, typography, calligraphy and design. It’s an incredible resource and the people behind it are currently embarking on the digitization and cataloging of these files. Although we mostly think of Qlik Sense as a data visualization tool, I love making use of its amazing associative engine to search, filter and explore qualitative information. I’ve used it to look at the 1 million images the British Library, The Tate’s online collection as well as the Internet Archive’s 65,000 books with over 24 million pages (although I haven’t quite finished reading them just yet).

For me, the Letterform Archive is as much a fascinating data problem as it is as an opportunity to get my hands on some amazing design artifacts. However, as I said, they are at the beginning of the archival task, and it’s pretty daunting. It takes deep domain knowledge and library science skills to set out the metadata structure. In addition, it also requires many other experts to add the really exciting and supplementary data for an item, such as who designed it or what fonts are used. Of course, what supplementary data is needed or useful may not be obvious immediately or even known right now. The standards will need to evolve.

As this is a visual archive, the digital images are incredibly important. A single ‘raw’ image data file at the maximum available resolution weighs in at around a whopping 500MB and there can be hundreds of pages in an individual file. The drive space and backup issues alone are staggering, and on top of that there is the time taken to carefully photograph each item. Furthermore, what about the descriptive data: the information about each item, its ID, title, creator, format etc.? The standard for most libraries and archives is the MARC record, which has been around since the Library of Congress started using computers in the 1960s. This method is effective, but not easy to work with from a data exploration standpoint. High quality cataloging and data collection is slow and time consuming.

Libraries and archives have to play the long game. The key to the usefulness of an archive is the longevity, stability and authority of the information systems it supplies as well as the items it houses. But ultimately to continue existing it needs to be used, it needs to be accessed and supply value. This is the same for all data.

So how do you get on with the long term task of building the authoritative archive and in the meantime deliver interesting data and value?

First up you need to be willing to break up and diversify the data. We can think of the base archival system as our governed ‘one version of the truth’. It’s stable, slow moving, carefully maintained, and governed by standards and policies. The trick is rather than attempting to load that system with every other piece of knowledge, we simply layer it on incrementally as and when it’s created. This opens up the possibility of having various degrees of credibility and authority in the data, which is fine as long as that is explained and the core data is kept safe. Of course this requires that each item is uniquely identified and that the key is used across all the data collections. But once in place, this system opens up some great possibilities, like utilizing the domain expertise of specialist groups or simply a ‘many eyes’ approach, such as how the New York Public Library is using website visitors to help improve the data around their NYC historical maps.

Building a visual archive is a long way from most BI and analytics projects but many of the problems faced pertain to any data project. When you start your next data initiative ask yourself:

How can we deliver value before we are ‘finished’ collecting and cleaning all the data?
How can we add new layers of information to the core data?
How can we balance the governance, manageability and accuracy of different data sources with the information needs of the users?

To finish with, here’s something from the Letterform Archive: it’s the artwork for a hand crafted chart by William Addison Dwiggins, who is often credited with coining the term ‘graphic designer’.

Dashboards, Models and Mud — Mar 14, 2017

When it comes to getting the most from your data, sometimes you have to be willing to get a little dirty.

I recently discovered Mission Control: A History of the Urban Dashboard, by Shannon Mattern. It’s a fascinating article on the use and limitations of data and dashboards in the context of urban planning and management.

Now before I hear you shout “sacrilege!”, let’s take a look at what’s behind the idea of the dashboard.

The term “dashboard” originates from the name given to a protective barrier at the front of a horse-drawn carriage, a board which protects the rider from getting covered in mud and filth (and any of the horse’s biological outputs). When this was adapted for the automobile, it protected the driver from the engine (and the output from the mechanics). The modern data dashboard protects the viewer from the mess of the raw data. It’s a way to handle the problem of signal and noise and information overload, thus helping people focus on the information that matters most. Now as Mattern points out, many complex systems such as cities simply can’t be boiled down to a few pertinent KPIs, no matter how much we’d like that to be the case. The dashboard ultimately restricts our view, giving us blinkers (to continue the horse and cart metaphor). This can be useful, but it can also lead to a kind of tunnel vision. As James Richardson loves to point out, KPI can very easily come to stand for “Killing Personal Initiative.”

Lately, I’ve been thinking a lot about that idea, especially in relation to automation and computer models. I’ve been experimenting with a few computer vision APIs and their use for seeing and describing the content of images. It’s incredible what they can achieve, even the publicly available ones are pretty impressive. It is especially impressive compared to four or five years ago when we first talked about natural analytics and the advantages of using our human skills for processing visual information.

But most of these models are still blinkered, in the same way that the viewer of the dashboard can be. The ones that work best are either extremely broad (high level, generalized categories as above — ‘a bicycle parked on field’, rather than ‘a blue Swedish Pilen bicycle, with black mudguards, standing on a daisy strewn grass lawn, near a marina in front of a cement works) or super narrow (focused on a very specific attribute, such as license plate reading). When the subject of the image is beyond their frame of reference they often struggle. At best, those blinkers can result in some humorous mistakes when things that aren’t part of the model’s realm of understanding come into view.

At worse it results in false positives which can lead to bias, prejudice and discrimination, and when that’s reflected back from the model at scale, it can be very ugly, as Google found out a few years back.

The thing is, we often need a lot more than a simple score, closed set, or single perspective when trying to understand the information at hand. We need a variety of views or layers, each enabling us to frame the previous, to give it context and shed light on any misreading or misunderstanding. Mattern gives the example of Patrick Geddes’ The Outlook Tower in Edinburgh from the early 1900s. The tower enabled people to understand the place (Edinburgh), by examining the ‘live’ view, both in macro and micro detail (through a camera obscura). Then it enabled them to position that information in an ever larger context (city, country, language, continent and world) by walking down floor by floor through the tower’s exhibition. This idea of framing and then reframing the observed or reported data is a fundamental part of research and analysis. It’s why for many years people have created Qlik apps using the idea of a stepped analysis, from dashboard (indication), to detail (analysis and association), to report (sharing insight and evidence). That ‘detail’ step is essential as it adds context to the data, but it’s not always enough. Sometimes you need to get down in the mud and the filth to truly understand what your data is and isn’t saying, or what you can and can’t see. Not only that, but you may need to bring new perspectives to the mix, so that you can reframe and re-contextualize what it’s telling you, in the same way that Geddes reframed the data from the view of the city using country or language. Those additional perspectives may even come in the form of qualitative data, and all its personal, physical, and human messiness.

Dashboards and models help us process and synthesize the overwhelming mass of data we have at our fingertips. But remember, you are always getting a reduced view of the world and sometimes you’ll need to jump down into the mud and get dirty, as that’s where the real story is often hiding.

For those who like an even longer read, here’s a link to Shannon Mattern’s Mission Control: A History of the Urban Dashboard

What You Hear Isn’t Always What You Need — May 22, 2017

Just because you trust those dulcet tones it doesn’t mean you shouldn’t question them.

How a piece of information, an insight, or fact is represented plays a big part in how it’s received and evaluated. The graphic designer has long been at the center of how messages are given meaning and power beyond the words and ideas they feature.

Whether it’s the color, layout or choice of typeface (see Errol Morris), the design of an item often plays with the credibility and urgency of the content. Mostly this is about manipulating our preconceived notions of what we ‘feel’ whether it’s official, trustworthy, serious, dumb, frivolously or down right suspicious. Of course it’s not all one way, those preconceptions vary massively from individual to individual and are often framed by culturally accepted ideas of authority. My favorite example of this is Michael Beirutdiscussing the perceived trustworthiness of the font Baskerville: “In my mind, Baskerville speaks with a calm, confidence-inspiring English accent, sort of like Colin Firth. No wonder it’s so trustworthy.” I’m not going to get into the anglo-centric nature of that today — I’ll save that for another post.

So what about the impact that actual tone of voice and accent has concerning how we receive and evaluate information?

We are seeing the swift adoption of voice-first systems, a way to get straight to the fact, the insight, without all that ‘pesky’ analysis. Simply ask the question and get the answer; it sounds great. But as we pair back the information and deliver just the most actionable and immediate pieces, we rely more and more on the credibility of the system delivering it. When that happens in an audio-only context, most of us are not equipped to critically engage with the information we are receiving. It’s often all too easy (for some of us) to trust Colin Firth’s voice, but perhaps not so much if it’s Joe Pesci’s or Pee Wee Herman’s that we’re hearing. Perhaps you have already placed absolute trust in Susan Bennett — the original voice of Apple’s Siri, or maybe feel more convinced by the construct that is Amazon’s Alexa. These are both carefully crafted with dialect and persona to appeal to their audiences, much more than mere word to speech.

It has to be more than trust.

All information we receive is colored by the medium it comes through. Even the most credible system will present you with information that doesn’t match the question or context you hold in your mind (even though it may seem to). It’s not that it’s wrong or illicit, it’s just a misunderstanding. The key is enabling people to continue the dialogue and qualify what is returned, removing the reliance on blind faith. When dealing with audio-only, there are very few cues to help people do that.

For analytics over audio or voice-first systems, it’s best to use the voice for information requests and then supply a visual means for the reply. Speech is very effective for requests and orders, but visual information is better for cognition and understanding, in part because it supplies all those extra visual cues that we are very skilled at spotting, especially when that’s in the form of data visualisation. By all means supplement it with an audible statement or ‘key insight,’ but always back it up with the visual evidence. It’s that which enables people to qualify whether the system understood them, spot other interesting data points, and even build data literacy. Also, it’s a lot quicker to visually scan a list of possibilities than sit through someone reading them out. Yes, the intelligence behind the curtain could just give you what it thinks is the most useful, but then you’ll never know what you’re missing.

In the age of alt-facts and fake news, it’s more important than ever help people check and qualify the information they receive.

So remember: be prepared for doubt, be open to questions and imagine your users are channeling the Lou Reed song Last Great American Whale: “Don’t believe half of what you see, and none of what you hear.”

What if the noise is really the signal? — Jul 28, 2017

There’s more to the story behind your data than rows in spreadsheets and charts.

When we discuss working with data we often talk about teasing out the ‘signal’ from the ‘noise’. Essentially filtering out the distractions, enabling us to work with only the most valuable or useful data.

We do it in lots of ways and for lots of different reasons. Sometimes we are trying to clean out errors that have occurred during data collection, or stripping out outliers and anomalies, sometimes it’s simply the transformation of data so that it’s easier to compute or work with, or for us to read. At other times we might be deliberately excluding data to help support our hypothesis or agenda (however I realize no one reading this would ever dream of doing such a thing). We often talk about these activities as an essential part of getting the value from a data set. But what if all that ‘noise’ held hidden riches?

What started me thinking was one particular article about digital vs analogue music recording. In it, the author was discussing how the intensity of the background tape hiss of a recording shed some light onto the track layering and over-dubbing required. This in turn went some way to revealing the techniques and processes that created the track. Another example was how on some tracks the studio machinery such as a background air-conditioner can be heard, or when listened to closely enough there are snippets of conversation. Studios have been fighting to remove this noise from their recordings for many years, but for some people it’s become a rich vein of insight. The ‘signal’ may be the track laid down, but the ‘noise’ expands the scene, adding context and depth beyond the idealized rendition.

This idea that there’s more to data than the rows you are seeing in the spreadsheet or data points on a chart, is essential to keeping the potential of your data alive. Jer Thorp calls this the “data system” and suggests that we always think of data in terms of it being a system and not simply an artefact. That we always view it as a series of processes and activities around; collection, computation and representation. Each of these in turn filter the ‘noise’ and magnify specific ‘signals’. When we engage with data we have to be conscious of the effects of those processes; what policies were applied during collection, what was missed, what was removed, what unknown errors lurk in the precision and ‘truth’ of those numbers? When presented with a formatted chart or cleaned data set, in can be very hard to find ways of answering those questions.

The answers lay in lineage and history. It’s well known that overtime we humans have a tendency to change our stories, to fit the ‘facts’ to our preferred narratives. Journalists and researchers know that contemporary secondhand records can be more reliable than firsthand recollections (you can thank Ebbinghaus’ “forgetting curve” for that). Data has a little of that too, each transformation, cleaning, interpretation moves it further along a preferred narrative. When we keep the rawness, noise and mess of that first point of collection we can retain more of that history. As we find new ways of seeing the noise, through machine learning or new techniques, we may well start finding fresh insights. It could be that we will be able to infer the policies influencing the collection or undercover hidden signals that reframe the entire data set, stretching its use beyond anything thought of when first collected. The more we maintain of the contemporary records the more possibilities and potential are stored in that data system. The more we record of the influences, approaches and policies that drove the collection of that data alongside it, the more we will be able to understand what else it might be telling us. We could even start to hear beyond the hiss to reveal the participants and processes that formed it.

Data as an Ecosystem — Sep 14, 2017

The shift from users and tools to analytic systems.

In my last post I mentioned Jer Thorp’s idea about treating data as a system. This is the idea that we look beyond the artifact (data-set, visualization, dashboard) and include the context it was created in, the decisions that formed or transformed it, and the techniques and medium used to express it.

This reminded me of a conversation I had with Donald Farmer a few months back about analytical systems. Framing BI and analytics as a system is very useful, as it helps us understand and learn more about the usefulness of an artifact and more importantly it opens up the interactions beyond a user and a tool or a reader and a report. For me, BI and analytics are the most effective for organizations when viewed as a system, and not simply as a series of artifacts and tools.

An important difference is that individuals use tools but people participate in systems, and they aren’t the only participants. Modern information systems contain a whole host of characters (people with differing roles, skills or intentions), digital services, bots and intelligent agents, and automated activities. The diversity and sophistication of these non-human participants is set to grow astronomically over the coming years. And it’s the exchanges and learning happening between all these participants that increases the value of the system, augmenting both the human and machine intelligence with in it.

When we talk about BI and analytics we often talk about its adoption in an organization as a way to access its value and impact. We talk about how many people are engaged with their data and actively using it for decision making. This is usually measured based on their access and use of analytic tools and artifacts like dashboards and reports. Traditional BI tools have a very simple system: a small group of data experts use tools to create reports for a slightly larger group of people to consume. It’s a one-way exchange that means if new ‘questions’ need to be asked then a new run through of the process is required and a new artifact created. That’s the technological/business process view of the system. In reality the system is more complex with far more exchanges between the individuals, more discovery and feedback, usually mediated by the data experts and BI architects. Also, there’s far more going on outside of that artifact creation process. Unfortunately in many of the analytic tools people are given to use these exchanges are rarely considered and the system gets reduced to this linear flow. This is usually because the focus is on end points and artifacts, neglecting the exchanges between people and the tacit knowledge they hold. In part this is why we often hear about how only 25% of an organization is actively engaged with analytics. Giving people access and more user friendly or even more sophisticated analytical tools has helped but it doesn’t seem to have helped shift that number very much. A self-service analytics approach based on “if you build it they will come” simply isn’t enough if it’s still a tool and artifact model. It needs to be a BI system that services the diversity of the skills and intentions of the people in the organization.

As the law of requisite variety (according to Dan Lockton) suggests; in order to deal properly with the diversity of problems the world throws at you, you need to have a repertoire of responses which is (at least) as nuanced as the problems you face.

One tool, one artifact type, one size fits all thinking simply can’t reach all the people in an organization. The key to engaging more people with their data and helping them grow both their confidence with analytics and their data literacy is in servicing them in ways that fit their behaviors and needs. For some it will be a static report, for others free form data exploration or drag and drop dashboarding, and for others perhaps it’s a BI bot or intelligent virtual assistant. But all this (and more) should form part of a rich analytic system that has learning (for machines and people) and governance at its core — creating an informed, learning and data literate decision making ecosystem.

And that’s it, 18,868 words — doubt if anyone will make it this far but I wanted to capture those few years of ideas and topics in a single view.

Now that’s purged 2020 is wide open for a new set of ideas.