Effective Visualization of Multi-Dimensional Data — A Hands-on Approach

Strategies for Effective Data Visualization

Image for post
Image for post

Introduction

However, dealing with multi-dimensional datasets with typically more than two attributes start causing problems, since our medium of data analysis and communication is typically restricted to two dimensions. I have written a couple of popular articles around effective data visualization in the past and have also spoken in conferences about the same. This article will be a compilation of my experience so far in dealing with structured and unstructured data! In this article, we will be exploring the following aspects:

  • The Grammar of Graphics
  • Effective strategies of visualizing multi-dimensional structured data (ranging from 1-D up to 6-D)
  • A brief into visualizing unstructured data including text, images and audio

Examples will be shown in Python, however, if you are interested you can replicate the same in R or any other platform of your choice.

Motivation

Can you guess what is common among these very different looking sets of data?

Image for post
Image for post
The Datasaurus Dozen — What is common among these diverse datasets?

Answer: Summary statistics for all the datasets are exactly the same!

Image for post
Image for post

This is a fun variant of the well known Anscombe’s quartet, which many of you might be very familiar with, as depicted in the following figure.

Image for post
Image for post
Anscombe’s quartet — Different datasets with same summary statistics

The key takeaway from these demonstrations would be, “Do not trust your data blindly, and start modeling on your data”. Summary statistics can always be deceptive. Always visualize and understand your data attributes before moving on to feature engineering and building statistical, machine learning and deep learning models.

Another very important source of motivation, particularly for effective data visualization, can be derived from some excellent case-studies dating several centuries back when we didn’t even have computers let alone Python or R! The first one is John Snow’s famous visualization depicting the Broad Street Cholera Outbreak in London, England in 1854!

Image for post
Image for post
Visualizing the Broad Street Cholera outbreak which helped find the root cause of the disease outbreak! (Source: https://github.com/dipanjanS/art_of_data_visualization)

You can see how a simple hand-drawn visualization helped find the root cause of the cholera outbreak in Broad Street way back in the 1850s. Another interesting visualization was built by Florence Nightingale, the mother of modern nursing practice, who had a deep-seated interest in nursing and statistics.

Image for post
Image for post
Causes of Mortality in the Army of the East — Florence Nightingale (Source: https://github.com/dipanjanS/art_of_data_visualization)

The above figure depicts a polar area diagram depicting causes of mortality (death) in the army in the 1850s. We can see the visualization is definitely not simplistic, yet it conveys the right insights — clearly showing the proportion of soldiers who died due to diseases which were preventable, based on wounds or other causes. This should serve as enough motivation for effective data visualization!

A Quick Refresher on Visualization

We will be using the Python machine learning eco-system here and we recommend you to check out frameworks for data analysis and visualization including pandas, matplotlib, seaborn, plotly and bokeh. Besides this, knowing about D3.js is also a must if you are interested in crafting beautiful and meaningful visualizations with data. Interested readers are recommended to read ‘ The Visual Display of Quantitative Information’ by Edward Tufte.

Understanding the Grammar of Graphics

Image for post
Image for post
Understanding Grammar of Graphics (Source: https://github.com/dipanjanS/art_of_data_visualization)

Basically, a grammar of graphics is a framework which follows a layered approach to describe and construct visualizations or graphics in a structured manner. A visualization involving multi-dimensional data often has multiple components or aspects, and leveraging this layered grammar of graphics helps us describe and understand each component involved in visualization — in terms of data, aesthetics, scale, objects and so on.

The original grammar of graphics framework was proposed by Leland Wilkinson, which covers all major aspects pertaining to effective data visualization in detail. I would definitely recommend interested readers to check out the book on it, whenever they get a chance!

We will, however, be using a variant of this — known as the layered grammar of graphics framework, which was proposed by Hadley Wickham, reputed Data Scientist and the creator of the famous R visualization package ggplot2. Readers should check out his paper titled, ‘A layered grammar of graphics’ which covers his proposed layered grammar of graphics in detail and also talks about his open-source implementation framework ggplot2 which was built for the R programming language

I have identified seven major components which usually help me build effective visualizations on multi-dimensional data. The following figure illustrates it with some details about each specific component in the grammar.

Image for post
Image for post
Major components of the Grammar of Graphics ( Source: https://github.com/dipanjanS/art_of_data_visualization)

We illustrate the same using a pyramid architecture to show an inherent layered hierarchy of components. Typically, to build or describe any visualization with one or more dimensions, we can use the components as follows.

  1. Data: Always start with the data, identify the dimensions you want to visualize.
  2. Aesthetics: Confirm the axes based on the data dimensions, positions of various data points in the plot. Also check if any form of encoding is needed including size, shape, color and so on which are useful for plotting multiple data dimensions.
  3. Scale: Do we need to scale the potential values, use a specific scale to represent multiple values or a range?
  4. Geometric objects: These are popularly known as ‘geoms’. This would cover the way we would depict the data points on the visualization. Should it be points, bars, lines and so on?
  5. Statistics: Do we need to show some statistical measures in the visualization like measures of central tendency, spread, confidence intervals?
  6. Facets: Do we need to create subplots based on specific data dimensions?
  7. Coordinate system: What kind of a coordinate system should the visualization be based on — should it be cartesian or polar?

We will now be looking at how to leverage this layered framework to build effective data visualizations for multi-dimensional data with some hands-on examples.

Visualizing Structured Multi-Dimensional Data

We’ll start by loading up the following necessary dependencies for our analyses.

Image for post
Image for post

We will mainly be using matplotlib and seaborn as our visualization frameworks here but you are free to check out and try the same with any other framework of your choice. Let’s take a look at the data after some basic data pre-processing steps.

We create a single data frame wines by merging both the datasets pertaining to red and white wine samples. We also create a new categorical variable quality_label based on the quality attribute of wine samples. Let’s take a peek at the data now.

Image for post
Image for post
The wine quality dataset

It is quite evident that we have several numeric and categorical attributes for wine samples. Each observation belongs to a red or white wine sample and the attributes are specific attributes or properties measured and obtained from physicochemical tests. You can check out the Jupyter notebook if you want to understand the detailed explanation of each attribute but the names are pretty self-explanatory. Let’s do a quick basic descriptive summary statistics on some of these attributes of interest.

Image for post
Image for post
Basic descriptive statistics by wine type

It’s quite easy to contrast and compare these statistical measures for the different types of wine samples. Notice the stark difference in some of the attributes. We will emphasize those in some of our visualizations later on.

Univariate Analysis

Visualizing data in One Dimension (1-D)

Image for post
Image for post
Visualizing attributes as one-dimensional data

The plots above give a good idea about the basic data distribution of any of the attributes.

Let’s drill down to visualizing one of the continuous, numeric attributes. Essentially a histogram or a density plot works quite well in understanding how the data is distributed for that attribute.

Image for post
Image for post
Visualizing one-dimensional continuous, numeric data

It is quite evident from the above plot that there is a definite right skew in the distribution for wine sulphates .

Visualizing a discrete, categorical data attribute is slightly different and bar plots are one of the most effective ways to do the same. You can use pie-charts also but in general try avoiding them altogether, especially when the number of distinct categories is more than three.

Image for post
Image for post
Visualizing one-dimensional discrete, categorical data

Let’s move on to looking at higher dimensional data now.

Multivariate Analysis

Visualizing data in Two Dimensions (2-D)

Image for post
Image for post
Visualizing two-dimensional data with a correlation heatmap

The gradients in the heatmap vary based on the strength of the correlation and you can clearly see it is very easy to spot potential attributes having strong correlations amongst themselves. Another way to visualize the same is to use pair-wise scatter plots amongst attributes of interest.

Image for post
Image for post
Visualizing two-dimensional data with pair-wise scatter plots

Based on the above plot, you can see that scatter plots are also a decent way of observing potential relationships or patterns in two-dimensions for data attributes.

An important point to note about pairwise scatter plots is that the plots are actually symmetric. The scatterplot for any pair of attributes (X, Y) looks different from the same attributes in (Y, X) only because the vertical and horizontal scales are different. It does not contain any new information.

Another way of visualizing multivariate data for multiple attributes together is to use parallel coordinates.

Image for post
Image for post
Image for post
Image for post
Parallel coordinates to visualize multi-dimensional data

Basically, in this visualization as depicted above, points are represented as connected line segments. Each vertical line represents one data attribute. One complete set of connected line segments across all the attributes represents one data point. Hence points that tend to cluster will appear closer together. Just by looking at it, we can clearly see that density is slightly more for red wines as compared to white wines. Also residual sugar and total sulfur dioxide is higher for white wines as compared to red and fixed acidity is higher for red wines as compared to white wines. Check out the statistics from the statistic table we derived earlier to validate this assumption!

Let’s look at some ways in which we can visualize two continuous, numeric attributes. Scatter plots and joint plots in particular are good ways to not only check for patterns, relationships but also see the individual distributions for the attributes.

Image for post
Image for post
Visualizing two-dimensional continuous, numeric data using scatter plots and joint plots

The scatter plot is depicted on the left side and the joint plot on the right in the above figure. Like we mentioned, you can check out correlations, relationships as well as individual distributions in the joint plot.

How about visualizing two discrete, categorical attributes? One way is to leverage separate plots (subplots) or facets for one of the categorical dimensions.

Image for post
Image for post
Visualizing two-dimensional discrete, categorical data using bar plots and subplots (facets)

While this is a good way to visualize categorical data, as you can see, leveraging matplotlib has resulted in writing a lot of code. Another good way is to use stacked bars or multiple bars for the different attributes in a single plot. We can leverage seaborn for the same easily.

Image for post
Image for post
Visualizing two-dimensional discrete, categorical data in a single bar chart

This definitely looks cleaner and you can also effectively compare the different categories easily from this single plot.

Let’s look at visualizing mixed attributes in two-dimensions (essentially numeric and categorical together). One way is to use faceting\subplots along with generic histograms or density plots.

Image for post
Image for post
Visualizing mixed attributes in two-dimensions leveraging facets and histograms\density plots

While this is good, once again we have a lot of boilerplate code which we can avoid by leveraging seaborn and even depict the plots in one single chart.

Image for post
Image for post
Leveraging multiple histograms for mixed attributes in two-dimensions

You can see the plot generated above is clear and concise and we can easily compare across the distributions easily. Besides this, box plots are another way of effectively depicting groups of numeric data based on the different values in the categorical attribute. Box plots are a good way to know the quartile values in the data and also potential outliers.

Image for post
Image for post
Box Plots as an effective representation of two-dimensional mixed attributes

Another similar visualization is violin plots, which are another effective way to visualize grouped numeric data using kernel density plots (depicts probability density of the data at different values).

Image for post
Image for post
Violin Plots as an effective representation of two-dimensional mixed attributes

You can clearly see the density plots above for the different wine quality categories for wine sulphate.

Visualizing data till two-dimensions is pretty straightforward but starts becoming complex as the number of dimensions (attributes) start increasing. The reason is because we are bound by the two-dimensions of our display mediums and our environment.

For three-dimensional data, we can introduce a fake notion of depth by taking a z-axis in our chart or leveraging subplots and facets.

However for data higher than three-dimensions, it becomes even more difficult to visualize the same. The best way to go higher than three dimensions is to use plot facets, color, shapes, sizes, depth and so on.

Visualizing data in Three Dimensions (3-D)

Image for post
Image for post
Visualizing three-dimensional data with scatter plots and hue (color)

The above plot enables you to check out correlations and patterns and also compare around wine groups. Like we can clearly see total sulfur dioxide and residual sugar is higher for white wine as compared to red.

Let’s look at strategies for visualizing three continuous, numeric attributes. One way would be to have two dimensions represented as the regular length (x-axis)and breadth (y-axis) and also take the notion of depth (z-axis) for the third dimension.

Image for post
Image for post
Visualizing three-dimensional numeric data by introducing the notion of depth

But is this effective? Not really! We can however leverage the regular 2-D axes for representing two continuous variables (scatter plot) and introduce the third continuous variable as a categorical variable by binning its values in fixed width bins — popularly these can be quantiles. Based on these quantiles (or bins) we can use size or even hue to represent the third variable here making it 3-D.

Image for post
Image for post
Using the notion of size or hue for representing continuous data in 3-D

A better option would be to use the notion of faceting as the third dimension (essentially subplots) where each subplot indicates a specific bin from our third variable (dimension). Do remember you need to create your bins manually if you are using the scatterplot functionality from matplotlib as opposed to seaborn (depicted in the following example).

Image for post
Image for post
Using the notion of facets for representing continous data in 3-D

The above plot clearly tells us that higher the residual_sugar levels and the alcohol content, lower is the fixed_acidity in the wine samples.

Image for post
Image for post
Visualizing three-dimensional categorical data by introducing the notion of hue and facets

The chart above clearly shows the frequency pertaining to each of the dimensions and you can see how easy and effective this can be in understanding relevant insights.

Considering visualization for three mixed attributes, we can use the notion of hue for separating our groups in one of the categorical attributes while using conventional visualizations like scatter plots for visualizing two dimensions for numeric attributes.

Image for post
Image for post
Visualizing mixed attributes in three-dimensions leveraging scatter plots and the concept of hue

Thus hue acts as a good separator for the categories or groups and while there is no or very weak correlation as observed above, we can still understand from these plots that sulphates are slightly higher for red wines as compared to white. Instead of a scatter plot, you can also use a kernel density plot to understand the data in three dimensions.

Image for post
Image for post
Visualizing mixed attributes in three-dimensions leveraging kernel density plots and the concept of hue

It is quite evident and expected that red wine samples have higher sulphate levels as compared to white wines. You can also see the density concentrations based on the hue intensity.

In case we are dealing with more than one categorical attribute in the three dimensions, we can use hue and one of the regular axes for visualizing data and use visualizations like box plots or violin plots to visualize the different groups of data.

Image for post
Image for post
Visualizing mixed attributes in three-dimensions leveraging split violin plots and the concept of hue

In the figure above, we can see that in the 3-D visualization on the right hand plot, we have represented wine quality on the x-axis and wine_type as the hue. We can clearly see some interesting insights like volatile acidity is higher for red wines as compared to white wines.

You can also consider using box plots for representing mixed attributes with more than one categorical variable in a similar way.

Image for post
Image for post
Visualizing mixed attributes in three-dimensions leveraging box plots and the concept of hue

We can see that both for quality and quality_label attributes, the wine alcohol content increases with better quality. Also red wines tend to have a sightly higher median alcohol content as compared to white wines based on the quality class. However if we check the quality ratings, we can see that for lower rated wines (3 & 4), the white wine median alcohol content is greater than red wine samples. Otherwise red wines seem to have a slightly higher median alcohol content in general as compared to white wines.

Visualizing data in Four Dimensions (4-D)

Image for post
Image for post
Visualizing data in four-dimensions leveraging scatter plots and the concept of hue and depth

The wine_type attribute is denoted by the hue which is quite evident from the above plot. Also, while interpreting these visualizations start getting difficult due to the complex nature of the plots, you can still gather insights like fixed acidity is higher for red wines and residual sugar is higher for white wines. Of course if there were some association between alcohol and fixed acidity we might have seen a gradually increasing or decreasing plane of data points showing some trend.

Is this effective? Again, not really! One strategy to make this better, is to keep a 2-D plot, but use hue and data point size as data dimensions. Typically this would be a bubble chart similar to what we visualized earlier.

Image for post
Image for post
Visualizing data in four-dimensions leveraging bubble charts and the concept of hue and size

We use hue to represent wine_type and the data point size to represent residual sugar. We do see similar patterns from what we observed in the previous chart and bubble sizes are larger for white wine in general indicate residual sugar values are higher for white wine as compared to red.

Now this might be better than the previous 4-D plot but honestly its just alright in my opinion. Yes, the hue helps us with seeing which wines have higher or lower fixed acidity but I don’t quite like the notion of size since it is often hard to interpret. Can we do better? Yes we can! Let’s use facets instead as depicted in the following plot.

Image for post
Image for post
Visualizing data in four-dimensions leveraging scatter plots and the concept of hue and facets

Look at that! Clear and concise visuals telling us fixed acidity is lower for white wine as compared to red wine and also residual sugar being much higher for white wine as compared to red wine samples. Also higher the alcohol level, lower the fixed acidity.

If we have more that two categorical attributes to represent, we can reuse our concept of leveraging hue and facets to depict these attributes and regular plots like scatter plots to represent the numeric attributes. Let’s look at a couple of examples.

Image for post
Image for post
Visualizing data in four-dimensions leveraging scatter plots and the concept of hue and facets

The effectiveness of this visualization is verified by the fact we can easily spot multiple patterns. The volatile acidity levels for white wines are lower and also high quality wines have lower acidity levels. Also based on white wine samples, high quality wines have higher levels of alcohol and low quality wines have the lowest levels of alcohol!

Let’s take up a similar example with some other attributes and build a visualization in four dimensions.

Image for post
Image for post
Visualizing data in four-dimensions leveraging scatter plots and the concept of hue and facets

We clearly see that high quality wines have lower content of total sulfur dioxide which is quite relevant if you also have the necessary domain knowledge about wine composition. We also see thattotal sulfur dioxide levels for red wine are lower than white wine. The volatile acidity levels are however higher for red wines in several data points.

Visualizing data in Five Dimensions (5-D)

Image for post
Image for post
Visualizing data in five-dimensions leveraging bubble charts and the concept of hue, depth and size

This chart depicts the same patterns and insights that we talked about in the previous section. However, we can also see that based on the point sizes which are represented by total sulfur dioxide, white wines have higher total sulfur dioxide levels as compared to red wines.

Instead of depth, we can also use facets along with hue to represent more than one categorical attribute in these five data dimensions. One of the attributes representing size can be numerical (continuous) or even categorical (but we might need to represent it with numbers for data point sizes). While we don’t depict that here due to the lack of categorical attributes, feel free to try it out on your own datasets.

Image for post
Image for post
Visualizing data in five-dimensions leveraging bubble charts and the concept of hue, facets and size

This is basically an alternative approach to visualizing the same plot which we plotted previously for five dimensions. However, considering the difficulty in interpreting size which we observed previously, you can convert one of the variables, if continuous, to discrete categorical using binning and then use that as an additional faceting parameter as depicted below!

Image for post
Image for post
Visualizing data in five-dimensions leveraging scatter plots and the concept of hue and facets

While the additional dimension of depth or size might confuse many when looking at the plot we plotted previously, this plot due to the advantage of facets, still remains effectively on the 2-D plane and hence is often more effective and easy to interpret.

Visualizing data in Six Dimensions (6-D)

Image for post
Image for post
Visualizing data in six-dimensions leveraging scatter charts and the concept of hue, depth, shape and size

Wow that is six dimensions in one plot! We have wine quality_label depicted by shapes, high (the squared pixel), medium (the X marks) and low (the circles) quality wines. The wine_type is represented by hue, fixed acidity by the depth and data point size represents total sulfur dioxide content.

Interpreting this might seem a bit taxing but consider a couple of components at a time when trying to understand what’s going on.

  1. Considering shape & y-axis, we have high and medium quality wines having higher alcohol levels as compared to low quality wines.
  2. Considering hue and size, we have higher content of total sulfur dioxide for white wines as compared to red wines.
  3. Considering depth and hue, we have white wines having lower fixed acidity levels as compared to red wines.
  4. Considering hue and x-axis, we have red wines having lower levels of residual sugar as compared to white wines.
  5. Considering hue and shape, white wines seem to have more high quality wines as compared to red wines (possibly due to larger sample size of white wines).

We can also build a 6-D visualization by removing the depth component and use facets instead for a categorical attribute.

Image for post
Image for post
Visualizing data in six-dimensions leveraging scatter charts and the concept of hue, facets and size

Thus in this scenario, we leverage facets and hue to represent three categorical attributes and the two regular axes and size to represent three numerical attributes for our 6-D data visualization.

Can we go higher?

Image for post

One method is to use more facets and subplots. Besides this, you can also use the notion of time if your dataset has a temporal aspect as depicted in the following example.

Image for post
Image for post
Hans Rosling’s famous visualization of global population, health and economic indicators

This depicts Hans Rosling’s famous visualization of depicting global population, health and various economic indicators across all countries. This was also presented in an official TED conference which I would recommend everyone to check out if they haven’t done it already!

This should give you a good perspective on how to leverage the layered grammar of graphics to visualize multi-dimensional data.

Visualizing Unstructured Data — Text, Image, Audio

Visualizing Text Data

Image for post
Image for post
Image for post
Image for post
Basic EDA on Text Data

Besides this, we can also focus on understanding and visualizing language structure by leveraging techniques like shallow parsing, dependency parsing and constituency parsing. Detailed code examples are present in one of my articles focusing on Natural Language Processing in case you are interested in further details.

The following figure depicts a sample example from my article demonstrating constituency and dependency parsing.

Image for post
Image for post
Constituency & Dependency Parsing — Understanding Text Structure

We can also look at embeddings based on recent deep learning models like Word2Vec, GloVe and FastText to understand text semantics besides structure. I have also covered this is extensive detail in my article on Feature Engineering Methods for Text Data which you can reference if you are interested in the gory detail!

Typically visualizing word embeddings can help you understand the context and semantics between words in your corpus and even leverage these features to build machine learning or deep learning models!

Image for post
An example of visualizing word2vec word embeddings on The Bible

Visualizing Image Data

Image for post
Image for post

One of the best ways to get started is to load up any specific images of interest and look at their channel contributions based on their pixel values as follows.

Image for post
Image for post
Visualizing image channels

Another interesting aspect to look at would be to observe image intensity distribution by checking out image pixel values and plotting a distribution as depicted in the following figure.

Image for post
Image for post
Visualizing image intensity distributions

Considering visualizations which might be useful for downstream machine learning or deep learning, we can look at edge detection which can help identify image edges and even use them as potential features!

Image for post
Image for post
Visualizing image edges

We can also visualize features obtained from HOG, which stands for Histogram of Oriented Gradients. In simple words, this helps in counting occurrences of gradient orientation in localized portions. A simple example is depicted below.

Image for post
Image for post
Visualizing HOG features

You can even use the feature descriptors obtained from the previous technique for building image classifiers, object detection and so on.

CNNs changed the world

Image for post
Image for post
CNNs revolutionized image representation and feature engineering

Visualizing intermediate layers of CNNs always help to understand which parts of the images are being extracted as features and help activate the hidden units in those layers!

Visualizing Audio Data

We leveraged the librosa framework for feature extraction and it also provides excellent visualization APIs. One of the easiest ways to visualize sound is to plot waveform amplitudes as depicted in the following figure.

Image for post
Image for post
Visualizing Audio Waveform Amplitudes

However, more useful visualizations are mel-spectrograms which are literally visual representation of the spectrum of frequencies of our audio signals as they vary with time. The following depiction shows mel-spectrograms of our audio sources.

Image for post
Image for post
Visualizing Audio Mel-spectrograms

These visuals give us an idea of how different audio sources can be and are very useful features which can be used by deep learning models like CNNs.

Conclusion

Image for post
Image for post

I encourage you to leverage these snippets for visualizing your own datasets in the future. Feel free to give me feedback and do share your own strategies of effective data visualization “especially if you can go higher!”

A major portion of these articles was covered in one of my recent conference talks in ODSC, 2018. You can check out the full talk agenda and slides here. The YouTube conference talk is available here.

Have feedback for me? Or interested in working with me on research, data science, artificial intelligence? You can reach out to me on LinkedIn.

Image for post
Image for post

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by +398,714 people.

Subscribe to receive our top stories here.

Image for post
Image for post

The Startup

Medium's largest active publication, followed by +721K people. Follow to join our community.

Dipanjan (DJ) Sarkar

Written by

Data Science Lead @Applied4Tech, @Google Developer Expert — Machine Learning, Author, Consultant, Mentor @Springboard, Connect: http://bit.ly/djs_lin

The Startup

Medium's largest active publication, followed by +721K people. Follow to join our community.

Dipanjan (DJ) Sarkar

Data Science Lead @Applied4Tech, @Google Developer Expert — Machine Learning, Author, Consultant, Mentor @Springboard, Connect: http://bit.ly/djs_lin

The Startup

Medium's largest active publication, followed by +721K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store