W3C and IDPF: Better Together?
Many people have offered their perspectives on the potential merger between W3C and IDPF that was announced last week. Peter Brantley is optimistic, hoping that a meeting of minds between books and the web can realize his dream of books in browsers. Nate Hoffelder’s trolling and fear-mongering doesn’t merit a response. But I’m quite interested in the pessimistic view, as articulated by Baldur Bjarnason. In this article I’ll try to respond to some of his points, but I also want to talk about my experiences with both groups.
My Life in Standards
I am both an insider and an outsider. I now work for the “big five” trade publisher Hachette, but I started working with ebooks fifteen years ago, and have made thousands over the years. Of course, there were things about EPUB that were maddening, but that’s just the way it was.
I vaguely knew that actual humans made the specs, but I didn’t think much about it. But eventually the company I worked for joined the IDPF, and I joined a conference call or two out of curiosity. It was totally intimidating, and I understood nothing.
A few years later, fixed layout was all the rage, and we (by now I worked for Hachette) were all in a panic. Apple had presented the world with a fait accompli, and Amazon was intent on incompatiblity. A standard was needed, now. I suffered through more conference calls, saying little. I ended up traveling to California to attend the face-to-face meeting where the fixed-layout spec was hammered out. I was the only person who worked for a publisher; everyone else built reading systems. That was my real introduction to standards work. I knew less about tech than anybody in the room, but I did know we had to find a way to make files acceptable to both Apple and Amazon. Showing up is risky—I was volunteered to write the spec itself.
I was perhipherally involved in the development of EPUB3. I still didn’t know what was going on, but I knew that the transition to EPUB2 was a mess, partly because there were no good sample files. Well, that was something I could fix. I picked a public-domain book almost at random, made the world’s first EPUB3, and tried to get the validation tools to work before they had been incorporated into EpubCheck.
Then EPUB3 came out, and nobody used it.
A year or two later, the W3C came knocking at the door of the publishing industry. They had a workshop on ebooks in New York. I did a talk on things we couldn’t do in EPUB3, mostly because CSS didn’t know much about books. Daniel Glazman also talked about the various inconsistencies and insanities in EPUB3, which inspired all sorts of interesting ideas.
In the meantime I was helping Hachette make print books with HTML and CSS. The W3C held another workshop, this time on print publishing. There I learned about the new Digital Publishing Interest Group, and was volunteered to go to China the next month for some giant meeting called TPAC. And, almost as an afterthought, I joined the CSS Working Group.
Now THAT was scary. I was the only publisher in the room, and the only person who didn’t write C++, or hasn’t written code for a browser engine. But I did know a few things about books and typography, and was willing to work. I appeared in the middle of a huge controversy, and it was overwhelming, but the group was incredibly supportive, and now I’m editing a bunch of CSS specs. Something I pushed for was even implemented! It has been one of the great experiences of my life.
Now I’m working on both sides of the fence. Although the cultures and histories are very different, I’m more and more convinced that we need all these people to work together.
Showing Up
Standards are made by the people who show up, and not nearly enough people are showing up. We need publishers and designers and engineers, typographers and librarians and entrepreneurs, web folks and book folks and business folks. We certainly need more diversity of gender and color and class.
I hope the merger can bring more people to the table. The future of ebooks could really use some input from browser engineers and web app developers who mostly hang out at W3C. Much of what we want for “portable web publications” can probably be accomplished with existing web technologies, but we need to figure that out together.
Responses to Baldur Bjarnason’s Essay
Standards
The problem is that people have been cheering ebook and digital book standards on for more than a decade and none of the fundamental problems have been addressed. We need more than happy thoughts to solve these problems and we need to be aware of the risks we are facing.
Yes, many people may have been cheering and/or jeering, but few of them have been contributing.
I’m not at all convinced that the increased involvement of the IDPF and publishers in the process will push web-book-related standardisation in a healthy direction.
I’m convinced that the increased involvement of web folks will help.
Everybody involved seems entirely oblivious to the possibility of portable web publications and related specs ending up as unimplemented failures.
We do pay attention to this. I think everyone in technology has worked on projects that failed. We’ve all lived through the glacial pace of implementation of EPUB3. We devote much of our lives to working around unimplemented or buggy features in various ebook reading systems (cough, Kindle, cough). We’ve been in the middle of struggles around CSS Regions, MathML, and so on.
Why shouldn’t the web community try to just improve reading in browsers at their own pace, focusing on their own needs and use cases without having to solve the self-inflicted problems of publishing?
Some of us think their pace is too slow, so we try to help by actively participating, thinking, writing, and building.
Why should the web community be happy about being forced to be involved in — and required to solve — the utter mess that is the ebook space?
Who’s being forced to do anything? I don’t understand. Rhetoric aside, most web folks I know love books and reading. Most browsers vendors have been involved with ebooks. There might even be some money to be made—a fair amount of reading happens on the web, and making the experience better can have benefits far beyond the publishing industry.
It’s not us vs them.
Why do the IDPF and the W3C just assume that everybody will be happy about the proposed merger?
I don’t know of anyone who assumed everyone would be happy. I was not part of the negotiations between W3C and IDPF, but I do know most of the people involved, and they are all familiar with criticism.
Why do people think that it’s a good idea?
Because it’s frustrating to work on the same problem with some of the same people but very different processes. Because ebook folks need help from web folks. Because we want more people at the table, and less duplicated effort. Because IDPF has much to offer W3C when it comes to accessibility. Because we think that books should be first-class citizens of the web.
“Intolerable Compromises”
Extending that DRM system to general markup would be an intolerable compromise.
I’m not aware of anyone seriously proposing EME-based DRM for general markup. Among other things, this would involve re-creating an entire browser stack inside the content decryption module. Ebook reading systems that are not based on browser engines are a useful cautionary tale here.
The portability concept isn’t easily compatible with the web’s current security model
Agreed that this is a core problem. I wonder if we might end up with a profile of HTML et. al. for portable documents, along the lines of Google AMP. No JS except as provided by the host website…
The publishing industry is deep into EPUB. Maintaining compatibility with their EPUB stack is one of the stated goals of many publishers and publishing industry companies. However, compatibility with something as complex and flawed as EPUB is expensive, hard, and fraught with challenges and has no value to the web community.
I’m working simultaneously on EPUB 3.1, which has compatibility constraints, and PWP/BFF, which does not. I’m actively fighting against including some of EPUB’s complexity (such as multiple renditions) in future work. Come help us!
There is an opportunity cost to all of these features. The web stack is flawed and needs improvement on many levels. The work going into fixing ebooks for the publishing industry is often work that could go into fixing pressing issues elsewhere. Not fixing something important because of this would be an intolerable compromise.
I’m troubled by zero-sum arguments like this. I certainly have little insight into Google or Apple’s business strategies. And there may be more overlap between ebooks and those “pressing issues elsewhere” than we realize.
Designing a single portable document format that serves the digital publication needs of everybody involved is, IMO, next to impossible without somebody making major compromises in their use cases. Somebody will inevitably find the necessary compromises intolerable.
Well, at least we could learn what compromises might be necessary before deeming them intolerable. We might be pleasantly surprised. Robin Berjon puts it better than I could:
Web technology is found once again to be awesome and more versatile than people expect.
Process
This led to the WHATWG and its set of specs followed by a reconciliation and harmonisation process of sorts.
One would hope that all involved have learned something from that. I would also note that there was a recent attempt to fork parts of CSS to WHATWG, and it failed.
This has led to an increase in a community-oriented standards process where a community incubates and debates a feature before it graduates to a standards track at the W3C.
Indeed! I think many of us have actually signed the Extensible Web Manifesto. We’re trying to build future ebook formats using existing web technologies as much as possible. We’re learning from the implementation experience of EPUB3. Companies like Vivliostyle are even polyfilling pagination, page templates, and all sorts of advanced features.
The only viable process under these circumstances is the community incubation model. Take the features that the publishing industry needs but might interest browser vendors through the Web Incubator Community Group. In parallel, the W3C should set up an Publication Incubator Community Group, structured in the same way, for specifications that are less interesting to browser vendors but are publishing industry specific and relevant to non-browser vendors.
We don’t need a separate community group—this could be done within the WICG. And working groups are still needed to hatch ideas that have been incubated.
IDPF needs better process. IPDF puts out lots of specs that don’t get implemented. That’s sad, but what’s really sad is that the specs themselves don’t benefit from implementor experience. This is where the traditional W3C process, with an emphasis on implementations and tests, can be valuable.
Theories of Change
Maybe it was a mistake for the publishing industry to base their ebook formats on web tech. Maybe they should have just built on OOXML instead.
Well, they tried using the Digital Talking Book spec from DAISY. That didn’t catch on.
And, in spite of all of publishing’s sins, would you really wish OOXML on us?
Maybe it’s best that web books be just a web thing, done by and for the web community as it attempts to solve its problems, and let publishing go off and solve its problems separately.
I don’t think the problems are so separate. Automotive displays and slide shows need native pagination and page transitions. Having a way to describe metadata about a collection of documents is useful for web apps as well as books. Heck, Presto made for a pretty good ebook reading system out of the box.
We’re not asking for very much. Annotations, service workers, manifests, and schema.org can already do much of what we need. I think our chances of success are much better with the web than without it.
Finally
A merger might not help. It might not make that much of a difference, given IDPF’s existing use of web technologies, and the existing links between the groups. But the status quo isn’t exactly optimal. Combining W3C’s stronger process, greater resources, and longer experience with IDPF’s energy and desire for change could be great. Bringing more people to work together on reading in browsers can make the web better for everyone. Better together? I say yes.
