SqlDBM
Published in

SqlDBM

Supercharge Your Thinking Through BI Tools, Really!

Enhancing human cognition through data visualization

A spark ignites the flame Photo by Elia Mazzaro on Unsplash

Rumor has it, Elon Musk is developing Neuralink — an implantable brain-machine interface to augment human cognitive abilities. What if it was possible to augment your cognitive abilities through a brain-machine interface right now, at no extra cost, and with absolutely zero holes drilled in your cranium?

Think, Rotate, Drop?

Let me start by making two predictions:

  • You’re familiar with Tetris, having played it countless times.
  • You’ve never stopped to consider that there are two reasons to rotate a shape: to fit a selected slot, obviously, but also to help you think about which slot should be selected.

David Kirsh and Paul Maglio discovered this phenomenon in a study¹ at the University of California when they analyzed performance data from human Tetris players and found many “superfluous” actions. Only later did they realize that the actions weren't superfluous at all. They were evidence of an important human adaptation: the ability to use our environment to enhance our cognitive processes.

We have found that some of the translations and rotations made by players of this video game are best understood as actions that use the world to improve cognition. These actions are not used to implement a plan, or to implement a reaction; they are used to change the world in order to simplify the problem-solving task.

— David Kirsh, Paul Maglio

That’s a long-winded way of saying that playing Tetris isn’t always “think, rotate, drop.” Rather, we, as humans, can extend our cognitive abilities through visual aids, such as on-screen shape rotation. Of course, you are free to challenge this assertion. Commit to rotating a piece only once you’ve decided where it should go. Enjoy Tetris hell.

EMTs Are Here to Help

The behavior I’ve just described in Tetris is an example of Extended Mind Thesis (EMT). It was first described in a paper² published by Andy Clark and David Chalmers in 1998. In this paper, Clark and Chalmers argue that human cognitive processes are not restricted to the brain or the body but rather, extend into our environment through tools such as diaries, calculators, and smartphones.

Long term memory can be off-loaded onto written records. Working memory can be transcribed and uploaded later for long-term recall (i.e., note-taking). Entire armies can be commanded with nothing more than a stick and some sand. Humans have been externalizing their thinking for so long that it almost feels innate.

Moreover, it may be that the biological brain has in fact evolved and matured in ways which [sic] factor in the reliable presence of a manipulable external environment. It certainly seems that evolution has favored on-board capacities which [sic] are especially geared to parasitizing the local environment so as to reduce memory load, and even to transform the nature of the computational problems themselves.

— Andy Clark, David Chalmers

The claim of EMT goes beyond the mere use of tools to enhance cognition. EMT suggests that the tools actually become part of our cognitive system.

If you’re still not convinced, there's a very simple experiment you can perform. Just ask the person next to you if they know the time. Observe as they answer “yes,” without a hint of irony, before reaching for their phone.

Business Intelligence, Literally

So, what do you do?

As a BI architect, the best answer I’ve come up with is: I turn billions of rows of data into diagrams that people can easily interpret. Of course, behind the scenes, I am analyzing source tables, transforming, modeling, testing, and various other -ings that BI involves.

But take testing, for instance. It certainly feels like work, but is it also a form of thinking? Extended cognition through BI tools?

The approach for testing a movie library for duplicate titles depends entirely on whether this library is real or virtual. If it were a physical DVD collection, I would think about sorting it first, then comparing adjacent titles — pattern matching. If it were stored in a database, I would use a HAVINGfunction, and think about the best way to group the rows — aggregation.

Humans are excellent pattern matchers — from finding Jesus in our food to recognizing a tune no matter how distorted or remixed. In the 1950s, Alan Turing suggested that the definitive test for distinguishing a human from an AI should be a conversation through a closed-door. He was wrong. Today, the definitive test of human intelligence is a blurry image (Captcha.)

I am not a robot. Therefore, despite my spectacular skills at identifying crosswalks, I also have a major cognitive limitation: my working memory. Unlike a computer, which can store near-limitless amounts of data, humans can keep track of about seven items (± 2) at a time (Miller’s Law.) Personally, I’m not discouraged by this in the least. I have learned to embrace my strengths and couple them with technology — to extend my condition.

When I sit down at a database terminal, I use my pattern recognition strengths and coupling them with massive computational capacity and parallel processing. Identifying the top five revenue-producing clients from a transactional system of billions of records. Finding the ten most common search terms from millions of searches. A database can zip through volumes such as these in seconds, and my working memory is fully capable of contextualizing the result. And I haven't even left my black-screen, text-based terminal.

Entering the Third Dimension

What if, instead of the ten most common search terms, I wanted to see twenty. Would I be as quick to detect common themes among them? To intuit the relative popularity of term number fifteen from that of twelve? Miller’s law suggests that even with the original ten terms, I was pushing my cognitive luck.

We have already identified working memory as one limiting factor to human cognition. Now let’s consider another.

Sales by region, searches by popularity — these are two-dimensional analyses. Even if I were to consider sales by region by store, I am still just looking at one KPI across one Dimension, albeit a slightly more granular one (region/store). Now let’s add a third dimension: time.

Sales by region by month, searches by popularity over time. Working in a purely tabular format such as SQL or a spreadsheet, analyzing the same top 5 or frequent 10 entries over time suddenly becomes as tedious as sorting DVDs by hand.

Let’s extend our analytical cognition using our visual circuit. If I am trying to understand the sales of each region by month, the stacked bar graph below can help me do so in a matter of seconds.

Monthly Sales by Region — visual

At a glance, I know that sales spiked in July, peaked in August, and have been declining since. East grew the fastest, but West has been the biggest contributor overall.

Without visual processing, we’d be looking at a table of 8 months by 3 regions (24 records assuming maximum aggregation), with thousands in sales to put into context. Wait, which month had the sales peak?

Monthly Sales by Region — tabular

For ease of understanding, I certainly prefer the chart. Yet, for most of my BI career, I rarely made the jump from SQL terminal to a reporting tool for ad-hoc analysis. Given that both the query and the chart took roughly one minute to create, my attachment to SQL was entirely out of habit and failure to recognize the utility of extended cognition.

What led me to change my ways?

A fairly easy bit of analysis that couldn’t be done without a visual cognitive enhancement.

Visualizing Time, a Case Study

The Requirement

Suppose you are in the regional retail business, and your company depends on securing product contracts with local suppliers. Your sourcing agents are critical to your company’s success because they negotiate product contracts with the suppliers. For your business to grow, you have to ensure that key popular products are always contracted at competitive prices and that your sourcing agents are adding, and not losing, new products from each supplier.

Traditional performance tracking (e.g., sales, clicks, race performance) is a point in time metric. What makes contract analysis tricky is that you have to consider spans of time.

Most contracts are valid for about a year. However, some seasonal deals last only a few months. Maybe you got a great price for that period, but you remain uncompetitive the rest of the year. Maybe there are gaps in your contract coverage. Maybe you got great margins on various miscellaneous products but lost the key popular item. Maybe the agent didn’t renegotiate anything at all, extended all the existing contracts, and went on holiday.

You can easily track the visits in your SRM system, but how do you judge their outcome?

The Data

A contract has a very simple structure: it’s valid for a span of time, for one product, from one supplier. However, any contract attribute can be changed if agreed with a supplier — meaning, the valid dates can move. The key product flag, the price, and the contract validity span are the important factors.

The SRM events track the dates on which a visit to a certain supplier took place.

Table structure
0. Raw data

The Approach

If SRM events are the trigger for this analysis, then a before and after comparison seems like the way to go. Since we can’t know if the visit was in reference to any particular contract or if it resulted in a renewal or an extension, it makes sense to perform the analysis at the supplier level and look at a whole year of available products.

Using the VALID_FROM/TO dates, we could write a query that would capture contracts valid within a year, before and after the visited date.

Aggregating (AVG) price over a year is simple. Product and Key Product Flag can be aggregated as KPIs by converting them into COUNTs.

That should give us two simple and easily comparable tables:

1. Creating a before and after snapshot from the visited date

Which can be aggregated and pivoted into a single table for comparison:

2. Aggregating and pivoting the before and after KPIs

Which can be then be totaled into a score by comparison dimension:

3. Comparing aggregated KPIs

This gives us a simple table with a qualitative summary of each supplier visit. In just one row, we can tell that our product mix and price remain steady, and our agent scored a 13% reduction in misc. product prices.

The Problem

Comparing conditions across spans of time with aggregated yes/no/gt/lt/eq indicators is too simplistic. The resulting comparison hardly addresses any of our original concerns. Are the contracts being renewed or extended? Do we have gaps in our coverage?

The raw data in image 0 contains all the details necessary to address these doubts, but my brain can’t contextualize it. The table in image 3 is intuitive but lacks nuance. It’s not a data problem, and it’s not a logic problem. It’s a problem of cognitive capacity.

The Solution

If we engage our visual circuitry and extend our cognition beyond logical operations, we can begin to detect data patterns hiding in plain sight.

After visiting the supplier, we really do have the same product mix and a much better misc. product cost. The KPIs don’t lie — but they don’t tell the whole truth either.

4. Combining tabular and visual data

Plotting the FROM/TO dates visually, we instantly see that the contract renovation isn’t as stellar as the KPIs would have us believe.

The after period suffers from significant coverage gaps in both key and misc. categories, and we lose product K2 after just one month, as it hasn’t really been renewed.

The devil, as they say, is in the details. However, understanding those details doesn’t have to be a devilish task.

See your Data, See your Database

So far, we have been looking at examples where visual tools can help us think about and contextualize data. What about the very tables where this data is stored?

This has been the driving vision behind SqlDBM — an online, collaborative data modeling and diagramming tool. Thousands of companies worldwide are already using SqlDBM to provide them with clarity and efficiency in visualizing and developing their data models.

In nearly every database client, one of the first things we see is a list of tables and views. However, the true understanding of a database landscape doesn’t come in the form of a list. It comes as a visual tapestry of relationships and classifications.

You don’t have to choose between a list and a diagram to understand your database.

Client table list vs. SqlDBM diagram

WithSqlDBM, you can have both!

Both! Tables listed and visualized in SqlDBM

The GPU in our Brains

Extended cognition may be a fairly new concept in philosophy and theory of mind, but visual pattern recognition has long been understood in anthropology as a key advantage in human survival. Our ancestors — those without exceptional visual cognition — those who failed to identify the sabertooth tiger hiding behind a bush — well, those are not our ancestors. How’s that for a Captcha!

Anthropology aside, enhancing cognition through visualization has been the cornerstone on which the entire BI industry is based. However, BI modelers, data scientists, and even data analysts frequently neglect to take advantage of them.

To bridge this gap, database providers, such as Snowflake, are starting to incorporate visual elements directly into their UI, blurring the line between themselves and dedicated visualization tools like Tableau and Power BI.

As we’ve seen in the examples from this article, life presents us with plenty of challenges that can easily overwhelm our cognitive capacities. However, it also offers us plenty of tools to natively augment their threshold. Whether you’re trying to detect a simple trend or go full-on-visual-story-telling-Hans-Rosling-spectacular, don’t forget that the tools for supercharging your cognition are already at your fingertips.

Hans Rosling takes us on a visual, data-driven, story-telling odyssey through time.
  1. Kirsh D., Maglio P. (1994). On distinguishing epistemic from pragmatic action. Cogn. Sci. 18 513–549. 10.1207/s15516709cog1804_1 [CrossRef] [Google Scholar]
  2. Clark A., Chalmers D. (1998). The extended mind. Analysis 58 7–19. 10.1093/analys/58.1.7 [CrossRef] [Google Scholar]

--

--

--

All about SqlDBM — Cloud based Data Modeling Tool for Snowflake ❄️, AWS Redshift, MS SQL Server, PostGreSQL & MySQL

Recommended from Medium

How much AI do Smart Assistants need?

5 Step Guide to Successfully Implementing Robotic Process Automation

As AI implementation continues to widen — The Department of Defense is developing AI ethics…

How To Start YouTube Channel Using AI Tools

Episode II on AI in Automotive: Operationalizing ML in Autonomous Vehicles

The Future of Artificial Intelligence!

Interesting AI/ML Articles On Medium This Week (Dec 5)

How would you describe your dream home?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Serge Gershkovich

Serge Gershkovich

Food for thought, meals essential. Shrine your mind, build your temple

More from Medium

Data governance for Snowflake: What you need to know

Crafting your enterprise data strategy

The Value of Embedded Analytics

Curation in Data Discovery