The Greatest Software Product of All Time: A Developer’s Appreciation of Adobe InDesign
Q: Is Adobe InDesign the greatest software of all time?
A: Yes.
I am perhaps biased by the type of work I do, yet I thought I would explain the things I love about InDesign, as I think its characteristics are relevant to software quality across a wide variety of domains, as well as the notion of quality itself.
My job is automating document generation. Since the early 1990s, I have led document automation projects, such as generating hundreds of provider directories from millions of records of data as a completely automated process, in numerous contexts. I have done this across almost all document types that lend themselves to automation: directories, catalogs, financial statements, postcards, school handbooks, tests, photo books, itineraries … you name it. I am thankful that InDesign came along in time to make this a somewhat straightforward process.
Our lucky choice
When Alissa Whitney and I founded Silicon Publishing in 2000, we had to figure out the right tool for this sort of work, as we couldn’t afford the industrial-strength, server-based tool (Xyvision) that we used at our previous company. We considered everything we could find that was within our budget: QuarkXPress, FrameMaker, PageMaker, FOP, PDF libraries, reporting engines, Fusion Pro (then called DL Formatter), and the new one … Adobe InDesign.
Until we met InDesign, we found one thing that all page composition tools had in common: limits. There were two main ways one could approach data-generated page composition:
Flowing in markup, which we would generate by wrapping data from a database with appropriate tags, to describe either the basic formatting of the text (QuarkXPress, PageMaker, and FrameMaker all provided markup to describe “stories” of text that would flow through a document template, adding pages as needed) or in some cases the complete description of the document — including templates, master pages, text flows, everything — at the time FrameMaker’s MIF was unique in offering a text-based export that would re-import and completely recreate the document from just that markup.
Automating the application as you iterate through the data, using a plugin or script to automate the process of building the document from scratch. Plugin development is much more involved than scripting: while scripting can often be done by a designer, plugin development tends to require a true software developer.
Regardless of which tool and automation approach we chose, some things just plain could not be done. In many cases, features of the application available through the user interface were simply not exposed to automation. Generally, markup could take you so far, scripting could take you so far, and C++ could get you almost anywhere after a ton of effort and learning; however, we would find dead ends even with C++.
For example, you might not know what in-flow continued headers are, but you’ve probably seen them if you ever perused a directory of physicians and had noticed something like this at the top of a page:
It might look really simple but, trust me, it’s quite a big deal. As you flow data through templates, you won’t know in advance precisely which record will start on each page, as some records might have more information and flow to multiple lines. Therefore, you can’t just feed markup; you have to work with either script or a plugin.
Depending on what the (QuarkXPress, PageMaker, FrameMaker) developers had decided about exposure to automation, you may or may not have access to the information that tells you, as the text flows, where you are in terms of the position on the page. It might be necessary to make two passes through the document — flowing the entire text with the first pass, then with the second moving through it programmatically, inserting the proper headers page by page, each time reflowing the remainder of the content.
Welcome to the subtle, obscure world of data-generated typesetting!
Other desktop options as of InDesign 1.0
Consider the three desktop alternatives to InDesign we had at the time (2000-2001) and their degree of extensibility:
- QuarkXPress: Could be automated via AppleScript (if you were on a Mac: nothing comparable on Windows) or via C++ plugins, and it had a markup format (XPress tags) that would let you flow text into the application. The plugins required true expertise, not in terms of coding difficulty but in understanding the object model: almost everything was there somewhere, but not necessarily documented or presented in an organized fashion. Scripting was difficult as not everything in the document model was exposed, and AppleScript itself is less than ideal in some ways. Overall, QuarkXPress appeared to be the most viable option, yet we weren’t sure we wanted to be married to the Mac platform and we had heard strange things about the company: not exactly loved in terms of licensing and customer relations, for example.
- PageMaker: PageMaker’s extensibility model allowed us to use external scripting languages such as Visual Basic or even the Excel Macro language, but through just two entry points: we could basically send an action to the program or get information back, but it wasn’t a sufficiently robust approach to automation to do the things we had hoped to do. PageMaker’s “PMScript” was a form of automation, but it was clunky because it was focused on automating the user interface, not the sort of direct document composition one would like. PageMaker also had a tagged text format something like RTF, but that was very limited.
- FrameMaker: Had a scripting language from a third party (Framescript), two forms of markup (MML for the simple; MIF for the ambitious), and C plugins (which arrived in version 4: note that extensibility is usually an afterthought). The plugins provided almost complete automation, generally attaining the goal of “anything you can do in the UI, you can do with coding” yet there were certain strange limits. Framescript did not expose the entire object model of the application, so scripting was not a robust enough option for us.
In all three cases, the glass was half empty or half full; at least significant, expensive effort had gone into exposing these to automation, which is probably the reason I only speak of these three desktop options and not the 20-something alternatives that didn’t come close. It was impressive how far these applications could go, given that manual use was their main purpose and extensibility was an afterthought.
Extensibility was not our only requirement: we had to consider other dimensions of the desktop products, weighing the tradeoffs of the flow-centric model of FrameMaker (good for long documents that have consistent formatting) vs. the more page-centric, graphically robust model of QuarkXPress (more what we wanted). But extensibility and exposure to automation were primary in our decision-making process. It wasn’t that we had big dreams for the future, it was just that we had been spoiled by tools that had greater fundamental automation capability than the three desktop tools we first considered.
We had experience with server-based, industrial-strength tools, such as Xyvision, which by comparison had more fine-grained control than any of these desktop apps provided for automation, along with a composition model somewhere between QuarkXPress and FrameMaker (flow-based yet with substantial override capability). Xyvision suffered, though, from existing only as such an industrial tool. There was something wonderful about using literally the same tool that desktop designers around the world used. Oh, and desktop tools were what we could afford.
We tried a new product — Adobe InDesign 1.0 — expecting to find limits in terms of UI features available to scripting-based automation. We were rather stunned because, after months, we could find none; twelve years later, there is no change …
The reality was and still is that, with InDesign, unlike all other options, we found that anything we could dream, we could do.
This is not to say that InDesign 1.0 did not have other limits; actually, it had extreme limits! It was very much a 1.0 product at the time, as it didn’t even provide bulleted lists, tables, or a host of other features. Also, it was far from being a server application; you had to watch the UI slowly write out a page at a time! However, the exposure to automation had no limits that we could find. This was quite exciting.
The amazing capability of InDesign for automation
Back in the Xyvision days, Alissa and I were programmers and had a contract with the typesetters (or “Xy operators” — something between designers and programmers). We had to get the correct markup in the correct sequence, their system would flow this, and their “macros” would perform the niceties, such as in-flow continued headers, balanced columns, and indexing.
With InDesign, we took a very different approach — more the way you would work in a C++ plugin (the second option in the figure above). We put the data in memory and iterated through it, building each page object by object, with VBScript or AppleScript, and then ExtendScript (a cross-platform flavor of JavaScript) when it arrived. We found that this technique was fairly infinitely powerful.
Thinking of the future back in 2001
We made the decision to embrace InDesign for automated composition with the expectation that three things would happen eventually:
- The remaining core app features comparable to, say, Quark would be implemented. (It was no secret that this was the direction at the time, and Adobe was well positioned to do so.)
- There would one day be a server version. (This was less obvious, but we had faith that it was inevitable.)
- Processing speed of hardware and performance optimization of the application would eventually give us throughput required of real-world applications. (We prayed that this would be in server form.)
We started building solution after solution based on automating InDesign desktop. In some cases, jobs would have to run overnight (e.g., 100 variants of a 60-page directory), but the output was perfect, and we could accommodate very complex requests.
While InDesign was sparse on composition features initially (especially long document features), these came rapidly over subsequent versions; even in the beginning, the core type engine was wonderful. We just had to work to fake things like tables and bullets. We ended up hacking crude versions of many of the features that Adobe was to add to the product, staying just one step ahead and feeling a little stupid “predeveloping” stuff, only to toss it out with each version.
From the beginning, our output was better than we had attained with Xyvision, although jobs took on the order of 100 times as long to process. Puzzled printers would ask us why they would wait 24 hours for stuff that had been nearly instantaneous at our old company. We doubted this would last forever, and we built solutions that ran for years and produced millions of pages of output, even coming out of desktop InDesign.
How did InDesign automation get so great?
InDesign was apparently designed by some very smart people. We got to know many of them as we’ve gotten so serious about extending it, and Olav Kvern, Rich Gartland, and others told me how it went down.
As they tell it, success came as a result of both policy (around the process of development) and architecture, and ultimately the development of INX forced the issue such that the nature of the architecture itself would mandate adherence to making all of InDesign extensible.This is a rare case where the initial product vision included extensibility.
When developers set out to design automation features for their application, they typically consider five to 10 features that they think users might want to automate, implement five of them, and consider it “done.” This doesn’t work because the developers can’t imagine all of the ways in which people will use their software. QuarkXPress and PageMaker were difficult to automate because their scripting models did not support every feature; workflow automation was usually incomplete or reliant on awkward workarounds.
Peter Boctor, the original designer of InDesign’s scripting architecture, didn’t take that approach. From the outset, InDesign scripting was designed to be complete: Anything that could change the InDesign document database had to support scripting.
The scripting model was also designed to be cross platform; the same objects, properties, and methods would be exposed to the platform-standard scripting languages (AppleScript and VBScript).
ExtendScript support, which gave InDesign scripters the ability to write scripts that would run on either platform, came later, but the groundwork — the common object model — was set in InDesign 1.0.
InDesign is made up of hundreds of plugins; there is no “core” application. Initially, scripting was supported via a single plugin; in later versions, every plugin that could make changes to the document was required to support scripting. This also means that InDesign scripting is extensible (i.e., when you add a new plugin, the scripting interfaces for that plugin appear in the InDesign object model).
When you export a document INX (the precursor to IDML), InDesign traverses the scripting object model of a document and writes it to disk as XML. As soon as INX was added as a feature, it meant that plugin developers had to implement scripting in their plugins; if they didn’t, their data would not be saved. In the internal politics of the InDesign engineering team, INX provided the ideal argument for deeper integration of scripting. Plugin development teams couldn’t call their plugin “done” until it could support INX; to support INX, they had to fully support scripting.
We owe a great deal to Peter Boctor’s original vision, and to Jonathan Brown for his work in expanding and improving InDesign’s scripting architecture.
The plugins we create fit right in among the other product features as first-class citizens — they are, after all, just additional components like the plugins that make up the product. Yet we rarely have to write plugins. The scripting functionality is awesome, and enjoys comparable performance in many case, yet scripts are far easier to write and maintain.
This was absolutely fantastic to us, and we managed to build a business on top of it. Around the world, many organizations have used InDesign to do amazing things with document automation — not just the database publishing on which we’re focused, but also scripts that work in the desktop UI, saving designers from otherwise tedious tasks. InDesign extensibility has gotten better and better over the past 13 years.
Welcome, InDesign Server!
As we were embarassed that our jobs had to run for hours or even overnight, we spent much of our time in the years from 2000 to 2005 begging Adobe for a server form of the product.
Now that I know more about Adobe, I realize how stupid we were, and I apologize to all the Acrobat sales reps, Go Live marketing managers, cafeteria workers, and consultants manning trade show booths, all of who had to endure my pleas for an InDesign Server. (They were remarkably polite, as was everyone at Adobe.) I knew that a server would be the best thing in the universe.
When I finally heard (from an Acrobat sales rep, ironically) that a server version was in beta, I called to beg, “Please, let us in the beta. I am perhaps the only person in the world who literally dreams of an InDesign Server!” They politely let us into the beta, and I flew up to Seattle to meet Whitney McCleary, one of the most amazing people behind InDesign.
I stayed up all night in anticipation. When I found myself in her office, I thought I was dreaming. Was this the right company? I had never in my life seen so many Quark documents assembled in one place. Manuals, newsletters, magazines, ads … everything. I think they had figured out every single thing that Quark had of any value and put it in the product.
Fortunately, InDesign was far better exposed to automation and, by this time (CS2 release time), they had just about gotten there with every last feature. I felt sorry for poor old QuarkXPress, but I chalked it up to karma for how they treated their customers and ended up with this brilliant team of people (in a well-funded company, which happened to have Photoshop, Illustrator, and the PostScript/PDF monopolies) tasked with doing page composition one better.
As Whitney and I walked to Peet’s Coffee & Tea, I mentioned how great it was for them that, with InDesign Server, they had the best software product in the history of the universe, and it certainly would become ubiquitous for server-side composition, which would be huge for Adobe.
Whitney said, “I don’t want to get your hopes up. We decided to release the technology to developers, but we decided not to support it. We are going to leave the support and solution development to companies like yours. Perhaps you would like to become a solution provider?”
“Perhaps?!”
We had been InDesign Server solution providers for five years already; it had just been a matter of abusing the poor desktop to do server tasks. Many companies had (and some still have) “desktop farms” of Adobe products to do automated imaging and page composition. However, with the official, respectable, server version, we really started growing. Ever since the server came out, the scale of our activity has increased each year. Our team now includes some of the legends that built InDesign itself.
What can you do with InDesign and InDesign Server today?
As of 2013, I think it is safe to say that InDesign and InDesign Server can accomplish document automation more powerfully than any alternative by a significant order of magnitude.
Solutions we have worked on include:
- Shutterfly’s “Custom Path” application that lets you build photobooks online that are rendered with InDesign Server.
- The Royal Caribbean cruise booklet generator, that composes thousands of personalized cruise booklets per day with InDesign Server.
- Numerous catalog automation programs with InDesign desktop and server, ranging from 1,000-page black and white data-intensive parts catalogs to retail catalogs such as Macy’s where design is intense on each page.
- Automating online editing and/or batch composition of a vast range of document types across a wide range of industries: real estate, insurance, healthcare, financial, education, government, retail, manufacturing… it feels like we have done every document type at least once.
Consider some of the qualities one would look for in a document-automation or data-generated publishing solution:
- Quality of output: Hands down, InDesign has the highest quality of output in terms of graphics, layout, and typography.
- Throughput: This is something of a new concept to InDesign or InDesign Server, yet by the miraculous evolution of servers themselves, coupled with the merciful (“all the cores you can put on a single server are the same price”) pricing model of InDesign Server, there are few if any barriers to throughput.
- Ease of template design: The sad truth is that InDesign has killed its competitors on the desktop, so all template designers would prefer to work in InDesign. While many other systems (i.e., Darwin,FusionPro, GMC, and XMPie) claim to use InDesign as the starting point for design, they don’t really use it for all output; thus, only InDesign Server can truly provide a solution where the data flowing through the template that you set up is rendered with the same tool with which you set up the template.
Unlike a product such as Word, that owns the market yet is nobody’s best friend, InDesign actually is loved by those who use it. It does have a learning curve, but once designers understand how to work in InDesign to create documents, the only limit becomes their imagination. And in our world, it is the same story for automation.
Why is InDesign exceptional software?
I have not addressed the most basic reasons why InDesign is considered exceptional by more typical users. Clearly, one great thing about it is its core functionality as a desktop tool for composing documents: graphics, text, paragraph composition, object styles, optical kerning... Yes, it is better than QuarkXPress, PageMaker, and FrameMaker for those who use it in the majority of use cases (its lack of functionality for XML authoring is the one thing keeping FrameMaker alive). Even if you never write or run a script, InDesign is an awesome product.
To me, that fundamental quality would put it in the “first-class” category if the extensibility and exposure to automation were comparable to its peers, but would not elevate it to the level of “best-in-class software.” The way it was designed with extensibility in mind from the start, at its very foundation, religiously enforced from the beginning of the product’s creation to the present, puts it in a class by itself.
If you are writing software and want to create something impactful and enduring, consider the ways people may want to extend it. If you are thinking long term, beyond the “minimum viable product” level so popular these days, imagine how great it could be if your product becomes the core engine or a core component for things you would never think of, by virtue of its exposure to automation and general extensibility. Adobe InDesign sets a great example.
Thanks to all who helped me write this, especially Olav Martin Kvern for giving the inside story of how InDesign became so great.
