Extending D3 with higher-order components

This is the second in a two-part series, the first article looks at how the standard object-oriented approach for designing charts results in complex monolithic APIs. However, the D3 vocabulary of data-join, SVG, rectangles and text elements is quite low-level; for charts we’d like to speak in the vocabulary of legends, series and annotations. In this second post we’ll look at how the d3fc library differs from its peers.

Rather than go into a sales-pitch extolling the virtues of d3fc, I’d like to introduce d3fc through a series of examples, showing how it tackles some of the D3 pain-points and extends it vocabulary to include charting concepts.

First, a quick comment on how d3fc is constructed. With D3 version 4, which was released earlier this year, the functionality of the library was split across multiple smaller modules, each of which can be used separately. If you look at the D3 organisation you’ll see separate projects for selections, shape, transitions, and other core D3 concepts. The D3 website guides users towards a project which simply bundles all these other modules together. In total, D3 is made up of 30 separate projects.

d3fc is constructed in exactly the same way, with the individual modules available via the d3fc organisation. Just like D3v4, you can use these modules separately, or you can use the d3fc bundle which includes the 17 separate modules.

I’m not going to introduce all 17 modules within this article, if you’re interested, each one has a comprehensive set of API documentation. Instead, I want to demonstrate the key concepts of d3fc via a number of charts, each one taken from an existing Mike Bostock D3 example to allow you to easily compare and contrast.

For the first few examples of d3fc, we’ll be taking a look at the chart built as part of the “Let’s make a bar chart” tutorial series.

The finished bar chart as demonstrated by this bl.ock.

d3fc-extent

We’ll start small.

A task that you need to perform whenever you create a chart is the calculation of the scale domain, i.e the range of data in the domain coordinate system. This task often involves finding the maximum and minimum of the data, then perhaps applying some padding, or some other adjustment.

D3 has an array extent function, which sounds broadly suited to this task, but is not in the case of the bar chart example above, because the y axis domain needs to include zero. As a result, the domain is computed from a simple max function:

y.domain([0, d3.max(data, d => d.frequency)]);

The d3fc extent components (linear and date) provide a much more versatile mechanism for computing the domain. Here’s the same example using the extentLinear component:

const yExtent = fc.extentLinear()
  .accessors([d => d.frequency])
  .include([0]);
y.domain(yExtent(data));

I’ll forgive you if you are a little under-whelmed with the above example!

A significant point I want to make here is that d3fc isn’t necessarily about writing less code, although in the examples that follow you will find that for non-trivial cases much less code is actually required. The aim with d3fc is for clarity and, as I have stated previously, to extend the vocabulary of D3. In the above example it is clear that the domain is derived from the `frequency` property and must include the value zero.

As an important aside, the extentLinear component, follows the D3 component pattern. It is constructed by a factory function, with configuration via a fluent interface of setter (and getter) functions. Once the extent component instance has been configured, it is invoked with the chart data.

Here’s a quick example that illustrates a few of the other features of d3fc-extent:

const extent = extentLinear()
  .accessors([d => d.high, d => d.low])
  .pad([0, 0.1]);

The above computes the extent for a multi-value series, e.g. a band series, where the upper value of the domain is padded by 10%. In this case the code is both concise and clear.

I’d probably better show you a more compelling example, before you lose interest …

d3fc-series

The D3 bar chart is built using SVG primitives, with a data-join constructing the required rect elements, setting their size and location accordingly.

The d3fc-series module encapsulates this concept, providing series rendering components for a range of standard chart types.

Here’s how the d3fc bar series component can be configured for this example:

const barSeries = fc.seriesSvgBar()
  .xScale(x)
  .yScale(y)
  .barWidth(fc.seriesFractionalBarWidth(0.9))
  .crossValue(d => d.letter)
  .mainValue(d => d.frequency);

The series component requires the x and y scales (so that it can perform the coordinate system transformation) and a number of value accessors, for example, error bars required both high and low values. Notice that the bar series uses the terminology of ‘cross’ and ‘main’ value, this is because all of the series support horizontal and vertical orientation allowing them to be transposed.

With D3v4 the d3-path and d3-shape modules added support for Canvas, this is also supported by d3fc-series, via d3-shape and d3fc-shape. More on this later …

The bar series is rendered in exactly the same way as the d3 axis, which you are no doubt familiar with, you just ‘call’ the component from a selection or data join:

d3.select(‘#bar-svg’)
  .datum(data)
  .call(svgBar);

This renders the following:

The d3fc-series components also support transitions. Their internal implementations ensure transitions are propagated and logical, i.e. bars transition upwards from their baseline value.

d3fc-chart

One aspect of constructing charts with D3 that I find most frustrating is layout. With HTML you have powerful tools such as CSS / Flexbox at your disposal. However, with SVG, layout is a manual process of positioning, where margins, padding and the relative location of elements must be computed manually. D3 does little to help in this area, beyond providing a simple ‘margin convention’.

The Cartesian chart component, from the d3fc-chart module, takes care of the routine aspects of creating a chart. It handles the creation of the axes, labels and handles margin calculations. Rather than rendering everything to a single Canvas, it uses a combination of div elements, flexbox layout and SVG (or Canvas). This construction technique is easier to style and supports responsive layout (as detailed later).

A chart is constructed using a pair of scales, with configuration once again using the standard D3 fluent API style:

const chart = fc.chartSvgCartesian(
    d3.scalePoint(),
    d3.scaleLinear()
  )
  .xDomain(data.map(d => d.letter))
  .xPadding(0.5)
  .yDomain(yExtent(data))
  .yTicks(10, '%')
  .yOrient('left')
  .plotArea(barSeries);
d3.select('#chart')
  .datum(data)
  .call(chart);

A chart is constructed from a pair of scales, with the scale properties re-exposed via the chart interface. The plotArea property accepts a d3fc-series component, setting its x and y scale accordingly.

Here’s the finished example using d3fc components:

You can find the full sourcecode for the d3fc version of this chart on the bl.ocks website.

If you compare the d3fc and the original D3 versions of this chart, you’ll find that the d3fc is a little more concise, around 30% less code, but more importantly it is (hopefully) much easier to understand.

If you’re a critical reader you’re probably jumping up and down right now and declaring that we’ve done exactly what I’ve accused other charting libraries of doing in the first article of this series, that we’ve hidden the power of D3 within a ‘box’:

That’s a fair accusation! … Fortunately it is not the case ;-)

There are a couple of design decisions that we have taken with d3fc that ensure this is not the case.

The first is that (visual) d3fc components all implement something which we call the Decorate Pattern. It is a simple concept whereby components expose the underlying data-join that is used for their construction. Exposing this provides great flexibility, allowing the user to modify the elements constructed by the component, via the familiar concepts of enter, update and exit.

The second is our use of composition. Each d3fc is deliberately kept small in size, typically less than 100 lines of code, with a relatively restrictive API. If a component doesn’t quite do what you want it to, dig into the source code and make use of the slightly lower-level d3fc and D3 components that it is composed from!

Here’s a simple example to illustrate decoration. Let’s say you wanted to highlight the vowels in the bar chart using a different colour. The following code decorates the bar series, selecting the underlying path elements and styles them accordingly.

const isVowel = (letter) =>
  ‘AEIOU’.indexOf(letter) !== -1;
const barSeries = fc.seriesSvgBar()
  .barWidth(fc.seriesFractionalBarWidth(0.9))
  .crossValue(d => d.letter)
  .mainValue(d => d.frequency)
  .decorate(function(selection) {
    selection.select(‘path’)
      .style(‘fill’,
         (d) => isVowel(d.letter) ? ‘indianred’ : ‘steelblue’);
  });

And here’s the resulting chart:

This chart is actually quite interesting, in that it highlights the fact that vowels are relatively frequent letters (of course axis labels and a chart title would help make this chart a bit clearer!)

What if you wanted to add labels above each bar to indicate their value?

Again, with the decorate pattern this is quite straightforward, just append the text element within the enter selection:

const barSeries = fc.seriesSvgBar()
  .barWidth(fc.seriesFractionalBarWidth(0.9))
  .crossValue(d => d.letter)
  .mainValue(d => d.frequency)
  .decorate(function(selection) {
    // style according to whether this is a vowel
    selection.select(‘path’)
      .style(‘fill’,
         (d) => isVowel(d.letter) ? ‘indianred’ : ‘steelblue’);
    
    // add labels
    selection.enter()
      .append(‘text’)
      .attr(‘class’, ‘bar-label’)
      .attr(‘transform’, ‘translate(0, -5)’)
      .text(d => d3.format(‘.1%’)(d.frequency));
  });

This results in the following:

You can find the full source for the decorated example here.

While most charting libraries have extensive APIs that try to cover all of the conceivable use cases (per-point styling, bar labels, rotated axis labels, …), with d3fc the API is small and concise, with all of these customisations, and many more, made possible via this Decorate Pattern that exposes the underlying data-join.

You don’t even need to know how the series (or other d3fc components), are constructed internally, all you need to do is familiarise yourself with the SVG structure that they create, so that you can write your selectors accordingly.

D3 is no longer trapped within a box!

And the Canvas support I mentioned earlier? Just swap cartesianSvgChart for the Canvas equivalent, cartesianCanvasChart, and do the same for the series. The Decorate Pattern works for Canvas too, where the decorate function is passed the context rather than a selection:

var barSeries = fc.seriesCanvasBar()
  .barWidth(fc.seriesFractionalBarWidth(0.9))
  .crossValue(function(d) { return d.letter; })
  .mainValue(function(d) { return d.frequency; })
  .decorate(function(context, datum) {
    context.textAlign = ‘center’;
    context.fillStyle = ‘#000’;
    context.font = ‘12px Arial’;
    context.fillText(d3.format(‘.1%’)(datum.frequency), 0, -8);
    context.fillStyle =
      isVowel(datum.letter) ? ‘indianred’ : ‘steelblue’;
  });

This renders exactly the same chart, but this time in Canvas. See the complete canvas example on the bl.ocks website.

d3fc-group

Let’s move onto a more complex example, this time using another very popular D3 bl.ock, the grouped bar chart. This chart shows the population age distribution for a few US states:

While the D3 code for the earlier example was relatively easy to follow, using a simple data join. The code for the above chart is not so straightforward, incorporating multiple data-joins, including a nested join!

Let’s see what an equivalent built using a combination of D3 and d3fc might look like.

The first new component to introduce is from the d3fc-group module. In order to render a grouped bar chart, the data needs to be clustered around one of its dimensions. In the above chart it is clustered around state, although it could equally well be clustered around age bands.

The group component does exactly this:

const group = fc.group()
  .key(“State”);
const series = group(data);

This results in a nested array of arrays:

[
  [[“AL”, 310], [“AK”, 52], [“AZ”, 515], …],
  [[“AL”, 552], [“AK”, 85], [“AZ”, 828], …],
  ...
]

This is structurally very similar to the output from the d3.stack component.

d3fc-series has a grouped series type, which acts as an adapter around other series. Here’s how the grouped and bar series types can be used together to render the output from the group component:

const groupedBar = fc.seriesSvgGrouped(fc.seriesSvgBar())
  .crossValue(d => d[0])
  .mainValue(d => d[1])
  .decorate((sel, data, index) => {
    sel.enter()
      .select(‘path’)
      .attr(‘fill’, color(ages[index]));
  });

Here the grouped bar is being decorated, using the index passed to the decorate function, which is the index of each of the series, to set the fill colour.

(Not a d3fc) legend

The final component that is needed to implement this grouped bar chart is a legend. We were going to add one to d3fc, especially as this sounds like the type of component that would be a lot of fun to write!

However, why create another legend when a perfectly good one already exists? Susie Lu has created a fantastic legend component, that is D3v4 compatible:

Incorporating this into a d3fc chart is as simple as decorating the chart component:

const color = d3.scaleOrdinal()
  .range([“#98abc5”, “#8a89a6”, “#7b6888”,
          “#6b486b”, “#a05d56”, “#d0743c”, “#ff8c00”]);
const legend = d3.legendColor()
  .orient(‘vertical’)
  .shapeWidth(20)
  .scale(color);
 
const chart = fc.chartSvgCartesian(
    d3.scalePoint(),
    d3.scaleLinear()
  )
  // other property setters removed for brevity
  .decorate(function(selection) {
    selection.enter()
      .append(‘svg’)
      .attr(‘class’, ‘legend’);
    selection.select(‘.legend’)
      .call(legend);
  });

Here’s the final chart:

Once again, I’d encourage you to compare the sourcecode for the original with the d3fc equivalent.

Note, the legend orientation does differ, with the d3-svg-legend rendering the labels to the right, wheres the original renders them to the left. Why not contribute this feature to d3-svg-legend?

boxplot

Time for one final example, this time a boxplot, courtesy of this bl.ock:

The sourcecode for this example is really quite complex.

The easiest way to tackle this chart is to view it as a combination of a few different series types. The first is the standard box-and-whiskers plot, or boxplot as it is often referred to. The values that fall outside of the high or low values of each box are outliers, and these are rendered as a point series. And finally there are the labels that define the five key components of each box (high, low, upper-quartile, lower-quartile and median). There isn’t a series type for data labels, however, with a bit of creativity these could be rendered as a point series which is decorated to add labels with suitable offsets.

Here are those three series types:

const boxplot = fc.seriesSvgBoxPlot()
  .crossValue(d => d.quarter)
  .medianValue(d => d.data.mid)
  .barWidth(50)
  .upperQuartileValue(d => d.data.upper)
  .lowerQuartileValue(d => d.data.lower)
  .highValue(d => d.data.upperWhisker)
  .lowValue(d => d.data.lowerWhisker);
const point = fc.seriesSvgPoint()
  .crossValue(d => d[0])
  .mainValue(d => d[1]);
const label = fc.seriesSvgPoint()
  .crossValue(d => d[0])
  .mainValue(d => d[1])
  .decorate((selection) => {
    selection.enter()
      .select(‘path’)
      .attr(‘display’, ‘none’)
    selection.enter()
      .append(‘text’)
      .style(‘text-anchor’, ‘middle’)
      .attr(‘transform’, (_, i) =>
        ‘translate(‘ + (i % 2 === 0 ? -50 : 50) + ‘, 5)’)
      .attr(‘stroke’, ‘transparent’)
      .attr(‘fill’, ‘black’)
      .text(d => yFormat(d[1]));
  });

The Cartesian chart has a plotArea that accepts a single series, however this chart is constructed as a composite of three different series. The key to bringing these all together is a multi-series.

Here’s how to merge these three series together (I’ll omit the code that massages the data into the correct form, it’s really not very interesting):

const multi = fc.seriesSvgMulti()
  .series([boxplot, point, label])
  .mapping((data, index, series) => {
    switch(series[index]) {
    case point:
      return flatten(data.map(s =>
        s.data.outliers.map(o => [s.quarter, o])))
    case boxplot:
      return data;
    case label:
      return flatten(data.map(s => [
        [s.quarter, s.data.upperWhisker],
        [s.quarter, s.data.upper],
        [s.quarter, s.data.mid],
        [s.quarter, s.data.lower],
        [s.quarter, s.data.lowerWhisker]
      ]));
     }
  });

Putting it all together with a Cartesian chart component, gives the following:

Once again, the full sourcecode is on the bl.ocks website.

d3fc-element

Another powerful feature of the d3fc Cartesian chart that I haven’t touched on yet is that it automatically handles resizing. Whenever the size of the containing element changes it will be re-rendered (with the render process being suitably throttled). This solves what seems to be quite a common question about how to make D3 charts ‘responsive’.

You can see this ‘responsive’ behaviour in action with the boxplot example.

But what if you want a chart with multiple y-axes, or have some other bespoke requirements? Well, if that’s the case you’re on your own! We want to keep the Cartesian chart simple, following the 80:20 rule.

Actually, that’s not quite true. You’re not exactly on your own. While we want to keep each of our components simple, and focussed, if it cannot be configured to do exactly what you want, take a look at the source-code, copy and paste it into your own component and make your changes there. You’ll often find that there are some lower-level components that are still of use to you.

Taking the example of multiple axes, if you take a look at the Cartesian sourcecode you’ll find that it is constructed using a number of custom elements from the d3fc-element module. These take care of the process of measure and render, giving you the dimensions of your basic building blocks (canvas or SVG) and a way to have your chart automatically re-render.

I’ve re-implemented this dual axis example with d3fc, making use of d3fc-element and also d3fc-data-join.

Summary

I’m really excited about our latest d3fc release, the changes in D3v4 have further underlined our design principles. If you want to give it a go, please let me know how you get on!