A Better Way to Make Books
Every since I started writing books in 2005, I have always felt that something was wrong with the book publishing process. The process is slow. There are lots of handoffs between various specialists. The book files keeps changing formats across various computer programs. All of this lead to lots of back and forth between people, revisions upon revisions and files that sometimes blow up.
I believe every book is a startup — product that needs to be quickly developed and easily changed when new insights and information is found.
In this essay, I am going to talk about the typical publishing process, what we can learn from the world of programming and the push button process I was able to construct with available tools to publishing my latest book.
The Three Kingdoms
A book is created in three different sets of activities
And each of these activities is a very different kind of work.
The process starts with the writing. Authors often have a way they like to write. Call it the idiosyncrasies of creativity — barebones text files, distraction free writing tools that sync with the cloud, or the ubiquity of Word. Anything can work, but it is what works for you.
The editorial process is also particular. Programs like Word can track revisions to the manuscript. Editors can also leave comments. Even with those abilities, heavy edits can be difficult to follow in a computer interface. In 2018, it is not unusual to still get a paper copy of your manuscript from your editor with specialized editorial symbols. The paper copy is reconciled by the author and returned to the publisher for final revisions.
Finally, enter the designers. The main purpose of their work is to layout the manuscript for the page. The text starts to act more like an image as the work is optimized for the reading experience. Changes to the shape of paragraphs and where the text falls on the page further solidify the work for the print page.
The process creates trouble on several fronts. Each contributor works in their own silo. Many handoffs take place between participants. Work often waits in a queue for someone to do the next part. These queues often add weeks or months to a project. If mistakes are found late in the process, they fixed in the designed version of the book. Rarely are those changes integrated back into the manuscript, making it difficulty to use earlier version of the material in other ways.
I think there is a better way.
The World of DevOps
For the last seven years, I have had a front row seat to the the changes taking place in technology function inside large, complex organizations. The organizations are struggling with the need to create and deploy new features at a faster rate than ever. This requires coordinating the activity of programmers writing code, test engineers validating the function of the code, and operations personnel deploying the code on servers for use by end customers.
Does that process sound familiar?
Authors, editors and designers substitute easily with programmers, testers and system administrators.
As customers, we are familiar with software companies who deliver new software once a year or systems we use that go offline for hours or days.
The DevOps movement extolls the idea that companies can deploy changes quickly and maintain the stability of the systems. The name itself was develop to denote the importance of development and op(s)erations working as a single unit. You might be familiar with the earlier Agile movement from the world of technology or the Lean movement that revolutionized manufacturing in the 1980’s. If so, you’ll recognize the concepts and possibly the challenges of implementing them.Going faster requires some fundamental changes to how the work is done.
Here are some of the Devops practices of high performing technology organizations:
- The code has one source — Almost all workflows have multiple people interacting with a single object. In DevOps, there is a single source for where all the code is stored. This means no one is ever working on the wrong version and ALL changes are tracked. Github is a popular version control tool (it also does much more than that).
- Automate everything you can — Technology environments have gotten complicated. Technology groups to deal with tens, hundreds or even thousands of servers. Now imagine needing to make an update to fix a security breach. The effort would be enormous and in a manual process, the risk of making a mistake or missing a server is very real. New tools have been develop in the last several years to automate those server maintenance tasks. Similar strides have been made in testing with programmers using automated test routines and receiving real-time feedback on the quality of their code.
- Fixing errors fast — If you combine a robust process for tracking versions of code with automated procedures to test and deploy code, you get the ability to fix errors quickly. At Google, programmers deploy new code in real-time, the code is automatically tested and notified if the tests were successful. If the automated procedures detect a problem, the programmer who made the change is notified that the error must be address and if it is not address, the system deploys the prior version of validated code.
- Less handoffs — A handoff introduces several undesirable conditions. First is the time lost time between the two people working on the item. The second is the ability for context or knowledge to be lost between the two people working on the item. And if an error is found, work has to be sent back upstream for rework creating wasted time and effort. The goal in DevOps is to create a small batch of work that can be developed and deployed by one person
My goal here was to highlight some of the practices that stakeholders in the book publishing process would recognize. There are many more practices associated with DevOps and still more being developed as the industry learns how to build fast and reliable systems.
Book publishing doesn’t need to be fixed. The work just needs to be organized differently.
My long standing goal has been to create a process that publishes a book with a push of single button. That means creating a workflow that can create a print-ready pdf file, a complete epub file, and complete mobi file from the most current version of the manuscript.
Let me share a workflow comes very close to that goal and adopts many of the ideas from the world of DevOps.
If you want to build the same workflow, you’ll need three ingredients: Google Docs, Markdown and Leanpub.
Google Docs is a widely used tool that writers and editors feel equally comfortable in. The application built in the cloud and creates a excellent record of revisions. The ability to collaborate in real-time creates some interesting possibilities for co-authors and editors. You can write using your computer, your tablet, or even your phone. If you use Docs in Google’s Chrome browser, you can work offline without losing your work. I also like the newer functions like their auto-generated Table of Contents and voice typing, which lets you dictate directly into the document.
To translate the manuscript across the whole workflow, it is important to establish a set of conventions that can be used consistently. For my workflow, I used a version of Markdown, a markup language that was developed by Jon Gruber. The intention of the language was to bring formatting functions to text without having to use longer HTML language. For example, you can italicize text with a single underscore at the start and end of the selected text. The language includes functions for a wide range of situations including headers, hyperlinks and footnotes.
The final ingredient that is important to the process is Leanpub. The company provides a book publishing workflow and an online storefront. The piece we are interested in is the workflow tool. Authors start by setting up a book in their system. There is a flat fee of $100 for each book you set-up in their system. Next, you choose how will you provide the manuscript. You can choose from several options for how you want to write and sync your book: Dropbox, Github account, upload a Word doc or pdf, or you can use their online visual editor. Once you have content uploaded, you start to use the workflow to generate the three main files needed in publishing: a print ready pdf file for printing, a mobi file for use at Amazon and an epub file for most other online digital retailers. What is amazing about the system is that after any change to your manuscript, with a push of a button you can generate new publishing files.
The process started with creating a folder in Google Drive to hold all the manuscript files. I created a separate Google Doc for each chapter. This creates smaller, more manageable units of content to work with and matches how a book is organized in Dropbox for the Leanpub workflow.
The next step was writing and editing. The majority of the work was just putting words on the page. The rest was using Markdown to add headers, format the text, and add coding for images, footnotes and hyperlinks. The folder in Google Drive was shared with my copy editor and proofreader. Editing was straightforward using the track changes, commenting and tracked revisions. Before exporting the files, we resolved all the comments to be sure there wasn’t any unwanted coding in the exported text files.
For my process, I use Dropbox to sync with Leanpub. You can export text files from Google Docs directly. You can also copy and paste from a Google Doc to a text file. Both of those options will give you a chapter of text without any formatting from the Doc. The side benefit is the opportunity to _use_ formatting in your Google Doc without affecting the exported materials. For example, you can use headers in the Google Doc allow the automatic Table of Contents to be generated.
In Dropbox itself, there is a folder for the manuscript and a folder for the previewed files. In the manuscript folder, you place the text files for your chapters and an additional files called book.txt. The book file tells the workflow tool what chapters to use and in what order. Once all of your files are in place, you create the files using the “Create Preview” button. The workflow takes around 60 seconds to complete. The generated files are added to the preview folder in Dropbox.
This overall workflow keeps the creation process in Google Docs and the files can be generated on demand with Leanpub. The most amazing part is how it accounts for writing, editing and design.
There are other workflow tools but I have yet to see one that addresses the problems across all three silos.
- Microsoft Word solves the writing and editing problems. They have added real time collaboration in recent years. The trouble is that Word can’t export ebook files and the exports for pdfs are limited (i.e. 6” x 9” page layout is not a standard option).
- Ulysses on the Mac is a great tool for writing and you can create pdfs using custom CSS files to control the layout. The trouble is how to integrate editing into the process.
- Pressbooks uses Wordpress as the base framework. The interface is easy for writing and generating files can be done with the push of a button. As a hosted solution, it is easy to work on the documents with others, but there isn’t any real time collaboration. There are also no specific editing functions built into the system to track changes or leave comments for others.
- Gitbook is interesting with the strengths of GitHub and the weakness of Github. The platform is largely designed for publishing documentation online and not designed to cover the print book side of the equation.
- Pandocs lets you convert almost any content file into different file type (ex. Word docx -> html, pdf -> epub). This make it an interesting piece in the final step of the workflow but you need to be comfortable working from the command line of your computer.
Opportunities for Improvement
With any tool or workflow, there is always possibilities for improvement:
- I wish I could kick off a routine that exported all of the current files from Google Docs as text files directly to Dropbox and then kick off the Leanpub routine to generate the book files. I looked into Zapier but there functionality with Google Docs is limited.
- The design layout choices from Leanpub could be better. You can create custom themes but the only choices are what fonts you want to use for the header and the body. A little more work is needed for matching fonts, font sizes, and line spacing. I reached out to Leanpub and they have been receptive looking closer at those defaults to improve the overall design.
There are a few others, but they are even smaller issues than the ones above.
I’d love to hear other people’s experiences with Leanpub or other platforms that have helped them simplfy their book publishing workflow.