A Leanpub Interview With Malcolm Maclean

by Len Epp

published Dec 07, 2015

Recently over email I interviewed Malcom Maclean, one of the most popular authors on Leanpub. His first Leanpub book, D3 Tips and Tricks: Interactive Data Visualization in a Web Browser, has reached over 25,000 readers, and has grown to be over 600 pages in length.

You can follow Malcolm on Twitter @d3noob and read his blog at d3noob.org.

D3 Tips and Tricks

E: What is D3.js and why is it important?

M: The stock answer to that question could be that D3.js is a JavaScript library that allows the creation of web based graphics… and it does. But there are conservatively about a bazillion different ways of doing that, so there’s got to be something else that makes it popular. I thought I knew what the ‘special’ ingredient was for D3 but I’m pretty sure I was only partly right. I thought it was all about the ability for data to be abstracted from the presentation media. After all D3 is named for the phrase ‘Data Driven Documents’. If you provide a linkage between the data and the medium, when you change the data, the medium follows. It doesn’t really matter if it’s a static page that might change every time it gets refreshed or a visualization that has dynamic, interactive content. The data provides the driver for the document. Not the reverse. I have a friend who bemoans the World’s addiction to the A4 page (or perhaps the US Letter page for our American readers) and he’s absolutely right. We have traditionally formatted our documents to suit the notion of the printed page and in so doing we lose some of the ability for the content to drive the message. D3 made me consider that the data was the most important part of presenting information and D3.js is a library that provides enormous flexibility over what can be done with data. Here’s where the first bombshell for this interview gets dropped. It took me about a year and a half after starting to write D3 Tips and Tricks before I realised that I had missed arguably the most important aspect of D3. It’s ability to manipulate the DOM. I can’t really apologise for it since I had no real concept of it when I started writing. It wasn’t until I started to experiment with element, attribute and style control that I twigged that it’s not about graphics, it’s about all the content on the page. I know that will sound lame to those who understand such things, but I wrote D3 Tips and Tricks as a noob without any appreciation of the finer points of web technologies (all right, without any appreciation of some pretty fundamental aspects of web technologies). Perhaps that’s why I think that D3.js is important. Because it is an enabler in a lot of different ways, it has an ability to support a lot of different needs whether you’re a consumer of the end product or a producer.

E: How has the open nature of the D3 framework affected its evolution over time? What’s the community like?

M: In a lot of ways D3 has ‘shown the way’ for a range of different projects which are based on the same principles of representing data. D3 itself has evolved from the ‘Protovis’ project which was led by Mike Bostock and Jeff Heer of the Stanford Visualization Group (with significant help from Vadim Ogievetsky). Where D3 and Protovis have differed is in the type of visualizations they enable and the method of implementing them. While Protovis worked well with static representations, D3 added animation and interaction making complex and dynamic visualizations easier to implement. While the code is open for editing and contribution, there is little doubt that the driving force behind it is Mike Bostock. He has had fantastic support from Jason Davies who is an absolute genius at representing data (with a special penchant for geographical projections). But Mike is the guy. He drives the D3 bus, and like any situation where you have someone who is passionate and gifted at what they do, all of us in the D3 community are really comfortable with the direction he’s taking the project (IMHO). Starting to try any use D3 from scratch is hard. There’s a good reason that there’s so many tutorials and examples for people to follow. It’s not trivial to grok the way it works. At least that’s the way I found it. But at the same time I was really pleased at the time and patience that several of the leading lights in the D3 World had in providing advice. In particular I would single out Christophe Viau for being an all around good guy. Interestingly there is something of a ‘support group’ feel to the way that the community helps each other. That might be because the code can sometimes be ‘tricky’ or it might be because there is a ‘brothers/sisters in arms’ effect where we’re all fighting for a common cause. Certainly I have come across a range of people who were wanting to help with the production of D3 Tips and Tricks. Out of the blue I had contact from Filiep Spyckerelle and Robin Bennett who selflessly donated their time and expertise in proofreading and Jose ‘Tactician Jenro’ who crafted a fantastic cover for the book. I assume that they liked the idea that the book was available for free and they had talents that could help make it better for others.

E: What’s a particularly striking example of useful data visualization using D3? This is my favourite: http://www.brightpointinc.com/interactive/political_influence.

M: That’s a good example that highlights the way that D3 is often used to provide a data visualisation function. The exploration of data and the associated insights that D3 can support are some of the prime drivers for implementing and using the library. However, my favourite examples are mostly the ones that demonstrate something simple but cool. Pierre Guilleminot’s Fourier series visualisation combines a bunch of neat techniques to demonstrate a real world concept. Mike Bostock has a huge range of examples, but the simplicity and elegance of the point-along-path interpolation strikes a chord with me. I’m shamelessly going to include one of my own examples in here, because it was one of those ‘ah-ha’ moments when I learned something new and different about using D3 with HTML inputs and while it’s simple, it’s a good example of the continuous learning of cool stuff that seems to be the defacto standard when using D3. Lastly there’s a neat (recent) example from Elijah Meeks combining two different techniques to show a sankey diagram with particles to illustrate flow.

E: Who is the book meant for?

M: The book is actually aimed at me! I’ve formatted it so that the information is hopefully palatable for others, but it was originally written in a scruffy form so that I could remember the things that I was learning as I was experimenting with D3. Since I started off from scratch there was a lot to learn and so the information naturally steers towards newer users. The tragic thing is that because I don’t have any formal education in using these types of tools, I’m constantly afraid that I will be encouraging poor habits in others, but I figure that there’s enough really talented individuals in the community that can influence people in a positive direction so that any grievous errors I make along the way can be forgiven.

E: Are you still adding new content to the book?

M: In theory no, but in practice yes. That’s one of the neat things about Leanpub. Even though I think that I’ve done enough experimenting for the moment, I will often get comments pointing out places where I’ve messed up or things that can be improved. I regard the book as a living document. Kind of like a Wikipedia page that you can download and take away :-). I don’t have any firm plans to add more content, but if I find something new and exciting that I want to play with (and it looks useful) I’ll add it in.

Raspberry Pi: Measure, Record, Explore

E: For those who don’t know, what is Raspberry Pi and why is it important?

M: To paraphrase raspberrypi.org the Raspberry Pi is a low cost ($20 or $35 USD (there are a couple of versions)), credit-card sized computer that plugs into a computer monitor or TV, and uses a standard keyboard and mouse. In fact two days ago the Raspberry Pi foundation announced a new $5 version called the Pi Zero! It enables people to explore computing and programming and is capable of doing everything you’d expect a desktop computer to do, from browsing the internet and playing high-definition video, to making spreadsheets, word-processing, and playing games. But more interestingly, the Raspberry Pi has the ability to interact with the outside world via an array of different connectors and connection options. This makes the process of building a computer that can be used to ‘step outside itself’ really interesting and this is where things start to get cool. The low cost, open hardware and flexible connection options appears to have been a catalyst for the Raspberry Pi to become the defacto small board computer for educators, hackers (people who make stuff) and people who want to experiment with computers in a ‘low financial risk’ type way. There are a multitude of different types of projects that people have carried out to build stuff and learn about computers (http://www.instructables.com/id/Raspberry-Pi-Projects for a small sample).

E: Who is the book meant for?

M: Again it’s a book that I wrote mainly for myself because I wanted to learn more about the Raspberry Pi. However, because it wasn’t my first book I had a firm intention to try and make it easier for those who might also be interested in the topic. I’m not entirely sure that it’s worked out as well as I had wanted, but like any process, writing has been an exercise in learning more about trying to provide a better product to an audience. The book itself is aimed at those who are starting from scratch and who have ambitions of doing some fairly advanced things. I would hesitate to say that it can achieve that goal, but at the least it provides guidance and a degree of reassurance that even someone who has little to no knowledge (because that’s me) can still do something pretty cool. I could possibly be described as the poster child for perseverance narrowly defeating incompetence. In fairness there’s a bit of cross-over with D3 being used to display data captured by the Pi in the ‘Explore’ part.

E: Can you explain one of the projects that you set up in the book?

M: There are definitely some cool ones (IMHO) however, the one that gave me the most pleasure was definitely the first one that used a DS18B20 temperature probe to measure a single temperature. It wasn’t spectacular or difficult or unique. But it was the first time I had tried to do something like connecting an external sensor to a computer so it was a formative moment that showed me that the process wasn’t rocket science. Better than that, it was achievable to pretty much anyone. The big key for me was that once I had achieved this first goal, I knew that getting more complicated projects was going to be achievable as well. For anyone interested in playing with a Raspberry Pi I cannot stress enough that the key moment will come after that first ‘win’. Once you have passed through that barrier the possibilities are endless. Using a temperature probe like a DS18B20 is pretty simple, but there are others that would fit the bill as well.

E: In your book there’s a cool example of web scraping that involves a couple of Leanpub books. Can you talk a little bit about what web scraping is, and how it can be useful?

M: Yeah, good question. Web scraping is a practice that involves programmatically getting a computer to ‘browse’ to a web page and then to extract some form of data from it. I had never done it before, and I had a strong desire to see how the sales of books on Leanpub compared to each other. Leanpub has a great API for authors to check the data about their own books, but if you want to get a historical trend of sales over time for another author’s books it can’t be done simply. However, because quite a number of authors display the number of sales on the main pages of their books, this represents a piece of data that varies as the book sells more copies. So in this case I have a small script running at a regular interval on a Raspberry Pi that ‘browses’ to the web page, reads the specific piece of the page that has the number of sales and then records the information in a database. The script does this for a few books of note and it does it every hour. Over time this has the effect of building up a good historical record of sales. So here’s the cautionary tale…. Web scraping is a double-edged sword that need to be used carefully. On one hand, used responsibly, it can select data at a reasonable interval without upsetting anyone. On the other hand, if someone scrapes a page constantly, this is a major encumbrance to the web site owner who is having to pay for bandwidth that is created. Additionally the activity may be blocking genuine visitors from getting access. It is also widely accepted that scraping a website to plagiarise a company or person’s data for some evil purpose like monetary gain or theft of intellectual property is a pretty reasonable grounds for being sued, so for those who might be considering doing this, make sure your purposes are for good and not evil!

Working with Leanpub

E: In both your D3.js book and your Raspberry Pi book you write about using a story-like approach to your explanations, rather than a “do this” approach. Can you explain why you made this choice? What kind of feedback around this have you had from readers?

M: I should probably expand on the ‘story-like’ description a bit. Any project is essentially a journey where you have starting position, a series of activities and an end point. In the process of trying to learn about how to do things with D3 or the Raspberry Pi I was constantly looking for examples or information to help move the project forward. In a lot of cases the information I was reading had a fairly high level of assumed knowledge. This meant that where something on the web simply said ‘do this’, I often found myself needing to learn what the hell was going on so that I could move my own journey forward. Since I was learning something, it made sense to write it down and I wanted to do this in a way that felt comfortable. My writing style is to try and make the tone conversational and light hearted. There were many occasions where I was carrying out an activity that I had to describe as ‘magic’ and as a result I have had several instances where readers have contacted me and explained the error of my ways or offered excellent advice on how to understand things using a simpler method. This is really just another way of saying that the feedback that I’ve received from the book has been almost universally positive. I have a bit of a theory about why this is, but sadly it has nothing to do with my outstanding writing ability or technical mastery. I’m pretty sure it’s because I give the books away. Since people can essentially pay what they want for the books (including nothing) I’d like to think that they’re getting value for money :-). And it’s pretty easy to be positive about something that you got for the right price.

E: Your D3.js Tips and Tricks has over 25,000 readers. Could you explain how you managed to reach so many people?

M: It’s pretty crazy really. I’m just some random guy that wrote some information while experimenting with some computer code, so to have that downloaded 25,000 times is a bit surreal. I think that there’s some good reasons why its been popular and there’s some bad reasons as well. One of the good reasons is that when I started writing about D3 there were only a couple of books out on it (Mike Dewar (Getting Started with D3) and Scott Murray (Interactive Data Visualization for the Web)) and while they are excellent books (they REALLY know their stuff), the number of choices were small. The other good reason was the linkage from d3noob.org where I was posting information in a blog while I was writing the book. As you might imagine there was quite a bit of information on the blog and I noted the references to Leanpub on pretty much each post. That definitely helped point people in the right direction and it works out really well for readers since the Leanpub version of the information is always going to be the most up to date. The bad reason why it’s popular is the price. It’s hard to say no to something free! So while it’s impressive to say that 25,000 people have downloaded the book, they may well have done so simply because there was no financial risk in doing it. Whether it was read or even of any use is another matter:-).

E: Is engaging directly with people who have already bought your book important to you, and is there more we could do at Leanpub to help you engage with your readers online?

M: I know that it should be important to me, but like anyone with time pressures, more often than not I find myself with little to no opportunity to be able to respond in a timely fashion to questions. It’s really neat to be able to help someone with a problem, but in a lot of cases it’s a case of the blind leading the blind, so I often advise people to seek assistance on Stack Overflow in the first instance. There are some really sharp folks there willing to help out. I’m not really sure what more Leanpub could do to improve engagement. You already provide an extraordinary service that helps authors reach out in a range of media types and ways. Forums and blogs stand out as the default responses, but you would only be reinventing a wheel that may only complicate the experience. It’s a really good question and I’d like to think that authors and readers alike could chime in here at any time.

E: How did you like Leanpub’s workflow?

M: I think that the secret of success for Leanpub’s workflow is twofold. Firstly there is no single way that an author has to work to create a book. The fact that there are a range of different options, all the way from adding content in a browser to using Dropbox and GitHub means that the barrier for entry is reduced for those who need it and there is greater flexibility and utility for those who want more control. I mainly use Dropbox and find the experience pretty easy. To be perfectly honest, the method just ‘gets out of the way’ and a writer can focus on more important things like adding content. It’s a measure of the ease of use that my main concern is the editor that I use. Markdown totally rocks, and I’m a big fan of MarkdownPad for writing. But it’s only available on Windows. When travelling I use Ubuntu on my laptop and I’ve yet to find a Markdown editor that comes close to MarkdownPad in Linux. I’ve used the ‘in browser’ editor when helping put together the short book ‘A history of Pye Radio Ltd. in Waihi, New Zealand’ and I while I didn’t find the experience as easy, it was awesome to have the flexibility to work from any OS that supported a browser. Ultimately however, the true test of ease of use lies in the fact that Leanpub makes the process of authoring and publishing easy. I am constantly astonished that more people don’t take the opportunity to put information they have squirrelled away into a book. The iterative process that Leanpub allows means that you can afford to get information pushed out quickly and then tune it or get feedback to improve it’s value. For me a big part of this is that Leanpub allows setting a price that starts at $0. When you think about it, that’s a big deal for a company. There’s a lot of support that they have to provide with little to no promise of a return. Part of my reasoning for writing is to try and give something back to the open source communities that have supported the interesting stuff that I enjoy, so to be able to distribute my books for free is just flat out perfect. I’m really stoked to see that the process is paying off for them as well (pun intended). While people have been kind enough to donate enough to buy me the occasional beer to celebrate, it’s a hobby and not going to pay the bills any time soon. However, folks like Roger Peng, Jeff Leek and Brian Caffo have found a niche for distributing books to support their courses which allow their students to get the work for free and in the process, the sheer scale and generosity of the people downloading the books sees them fairly high on the Leanpub earnings pages. That rocks!

E: Leanpub was started by authors who were really opinionated about their writing and publishing tools. How can we improve these, from your perspective?

M: I’m struggling to find a good answer to this. I’m just so grateful that Leanpub runs the service I would be more than happy to write in the dirt with a stick. I know that people will have a preference for doing work in a particular way, but I’m happy to be flexible and the Markdown format provides a pretty vanilla mechanism to allow variation in the type of editor used. If someone’s interested in writing their books in Microsoft Word, that’s fine, but I would be surprised if in 10 years or so we found the same sort of reliance on proprietary document formats that we do today. Life is just so much easier in plain text. Another point to add is the great work that is going on to support Markua. That has the potential to provide a great base for easier distribution and editing of information.

E: Is there anything else about Leanpub that you think we could improve?

M: I’m a Leanpub fanboi, but there are a few of things that spring to mind that I know have been explored before and there are good reasons for and against, but they’re things that would make a difference to me. I’d also like to preface my thoughts with the observation that Leanpub is always super active in dealing with online discussions and being engaged with their author community. Kudos.

Firstly, some sort of ‘hold my hand’ path for production of a hard copy version of a book. I know that Leanpub makes print ready PDF versions, and they acknowledge that the concept of a hard copy service detracts from their core business. I have read recommendations for options to take Leanpub books through different services, but I’d be seriously interested if one partnered with Leanpub to guide the process. It’s really just me being apprehensive about the process. I’ve trusted Leanpub to make my words look pretty in a digital format and they do a great job. I don’t have that same faith for taking the step to print.

Secondly, an option to produce an index for a book. I know the concept isn’t trivial and I’ve read the online discussions and there are good arguments in both directions. But the ability to add an index would be useful IMHO.

Thirdly I’d really like to get access to extra data on other books. I don’t think that there’s really any motivation for Leanpub to do it, but an API that allowed me to view sales numbers of different books, page counts, percentage completion, price, that sort of thing, à la the process that I engineered by scraping would be neat. I’m not going to hold you to it since that would be a bunch of work for you guys with little to no reward :-).

Fourth: (this is starting to add up) I’d like to be able to add or remove books from a collection without having to essentially recreate the collection from scratch. There must be some catch that makes this tricky (I assume that it’s the approvals from different authors with percentages and suchlike).

E: My last question is, are you working on any more books?

M: Well yes. I have a book on Leaflet that is really only about 50% finished, and has been so for quite a while. I’d like to think that I will revisit it when they publish version 1.0 (which should be soon). I’ve also been helping out with a concept of publishing a series of books aimed at providing enough information on a topic to help people get going. I have no idea where it will head, but at this stage the ‘Just Enough’ series includes Just Enough Linux, Just Enough Co-Authoring in Leanpub, Just Enough Raspberry Pi, Just Enough ownCloud on a Raspberry Pi and Just Enough Ghost on a Raspberry Pi. I’d like to add more to this lineup and the idea is to bundle them so that folks can just download the whole kit and caboodle in one hit. At the moment we have them in the ‘Just Enough Collection’ for the staggering price of $0.99.

Parting thoughts

M: Thanks for reaching out to ask if I was interested in participating in the interview. It’s been an opportunity for a bit of reflection that has been very therapeutic. I’ve found that I enjoy the process of writing. Leanpub has provided me with an outlet to do something that I never imagined I would be enthusiastic about and in the process I hope that others have gotten some benefit as well.

– Posted by Len Epp

Originally published at leanpub.com.