An Interview with Publishing Industry Expert Thad McIlroy and Author of “An Authoritative Look at Book Publishing Startups”

Published Jan 16, 2017 by Len Epp

Thad McIlroy is an author, speaker, and publishing industry expert who blogs at thefutureofpublishing.com. In this interview, Leanpub co-founder Len Epp talks with Thad about his wide-ranging career in the publishing industry, the state of the publishing industry today, whether self-published authors should submit their books to subscription services, some recent controversies in the book publishing world, and the future of publishing and book publishing startups.

This interview was recorded on January 13, 2017.

The full audio for the interview is here. You can subscribe to this podcast in iTunes or add the following podcast URL directly: http://leanpub.com/podcast.xml.

This interview has been edited for conciseness and clarity.

Len: Hi, this is Len Epp from Leanpub, and in this podcast episode, I’ll be talking with Thad McIlroy. Thad is an author and consultant who writes and blogs at thefutureofpublishing.com. His consulting work and experience reaches into many aspects of publishing, from print and the origins of desktop publishing, to analysis of the latest tech, digital book publishing, and supply chain optimization and other areas.

In addition to consulting, writing books and blogging, Thad is also an accomplished speaker, who has spoken at hundreds of events, from conferences to corporate meetings.

In this interview, we’re going to talk about Thad’s career, his interest in tech, and his recently published report, An Authoritative Look at Book Publishing Startups.

So, thank you Thad for being on our podcast.

Thad: Thank you Len, glad to be on board.

Len: Thanks. In these interviews, I normally like to start by asking people about — what I call their origin story. You’ve got a great one, that ranges widely across the publishing industry, and I was wondering if you could tell us a little bit about how and why you first got interested in publishing, and an overview of your path to where you are today?

Thad: Sure. I can go back centuries — I’ll only get briefly into that. But I came out of, well, a literary family. I mean, not that highbrow a literary family, although ‎Kenneth Grahame is one that I can point back to. That’s fairly literary, but my father was–

Len: The Wind in the Willows, right?

Thad: Yes. We have a very direct lineage. And my great uncle, who is also an author, he spent time with him as a boy. That was Lawrence Hill Grahame, who wrote a number of books for young adults at the turn of the last century, which were best sellers, and award winners at the time, and totally unreadable today.

And then my father was a novelist and had two novels published, he was working on a third one when he died. He did a lot of radio work at the Canadian Broadcasting Corporation, which you well know, the CBC. And so that’s sort of the milieu in which I grew up. My father always said to me, “You’re not the creative one in the family, you’re going to be an engineer. It’s your sister who’ll become a writer.” And she does write, but she’s not published.

But my interest gravitated to book publishing. Where I really started in the industry was as a book seller in Toronto in the ’70s, and working for independent book stores and then working for chain stores. That’s where I fell in love with the book industry, and that’s really what’s launched me forward to today.

Len: And did you grow up around Toronto?

Thad: Yes, I was born in Toronto, and lived there for the first 30 years of my life, then moved to San Francisco.

Len: I’m curious about your first entry into the publishing industry. You worked for a bookseller, and I believe early on you edited a book about a Canadian Prime Minister as well?

Thad: Well, that was after I started my first publishing company. So in the third or fourth year of my book selling career, I had gone down to San Francisco. It was for the annual convention of the American Book Sellers Association, which is now the one called BEA [Note: It appears they have rebranded themselves as BookExpo — eds.].

And in those days, it was a huge hall, where all kinds of publishers, all shapes and sizes, would display their wares. And I was there with a friend at the conference, out of interest. And saw all these small presses with nifty books, where it’s like, “Oh you can’t get those in Canada.” And then it’s like, “Well maybe I could start a little company and distribute some of these small presses?”

And so I did. I started what I called Virgo Press, because my partner and I were both September babies, so Virgos. And we started this as a distribution company out of my bedroom, basically. And then a friend came to me and said, “I’ve got this amazing idea. I think it’d be a big best seller. How to Win Canada’s Lotteries was his idea.

And this was when they were still pretty new. They were [a phenomenon]. It was still something that was fresh and exciting. They didn’t have them every single place. They didn’t have 27 different kinds of tickets you could buy, and so on.

And so we published that book, and it was a bestseller. We wrote it pseudonymously. He, I and one other person wrote various chapters for it. And then I toured on behalf of the book. And so that was my first self-publishing experience. It launched what became Virgo Press, which ended up being a trade publisher of some substance. We ended up with 15 staff, and about 60 tittles published over about a three year, four year period.

And then ran into a situation where the bank decided they didn’t want to be financing us anymore and called in our loan. I couldn’t replace the financing, and went bust. But I had had this wonderful four year experience in my early twenties running a fun, small general trade publishing company out of Toronto.

Len: Wow, I didn’t realize it was that early on in your career, that it was in your twenties that you were founding a company and writing bestsellers already. That’s fascinating.

You mentioned the Book Expo America, the BEA Conference, was originally in San Francisco?

Thad: No, it used to travel around more, before they settled on New York much more recently. And then they tried Chicago last year, and that didn’t work out. In those days, they would move it around, as most conferences used to in those days, where it would be West Coast one year, East Coast one year. Then its centre would be Chicago, usually one year. And they’d just circulate that way. So it just happened that year was a San Francisco year.

Len: For those listening who might not be into the minutia of publishing conferences, there are currently two big conferences in the United States — the BEA conference and the Digital Book World conference, which is actually just coming up in a few days after this interview — and where Thad will be giving a couple of talks [The talks are “5 Tools You’ll Wonder How You Lived Without” on Tuesday, January 17, and “How to Thrive in an Era of Constant Change”on Wednesday, January 18 — eds.]. But the reason I asked about San Francisco is that these conferences tend to be focused around New York, so it’s interesting to know that there was a West Coast presence for the big, big players at a certain time.

I think I read on a bio about you somewhere that you may have been one of the first people to publish a book that was created entirely with desktop publishing software tools, and I was wondering if you could talk a little bit about that experience.

Thad: Sure. That was great, great fun. And I stumbled into it. It was not like I had a vision and thought, “Ah, I see what’s coming here, and I’m going to be the pioneer.” At the time, after my publishing company went bust, I became a freelance journalist, and on the side, with a couple of partners, we started a small book packaging company.

To call it book packaging … from the industry, they’re the people in the space between being an author and being a publisher. Instead of just writing the manuscript, you also contract for all of the editorial design and illustration for all of the books you do. You deliver what we’d call camera-ready copy to the publisher, so it just goes straight to press on their side.

They don’t have any of the developmental expenses, you take that on yourself. So you get a higher royalty, as a result of their lower overhead, and it gives you control of the project where, in many cases, because of the particular nature of the book, you want that kind of control. You don’t want to give that up. So I was working on books of political cartooning — it was fun, I’ve always loved political cartoons, and I edited two books, one on Pierre Elliott Trudeau, and one on John Diefenbaker, where I told their life stories with a focus on their political stories, in cartoons and in words. And so on one page would be a famous cartoon, usually I would have about 30 cartoonists in each of the books, with representative cartoons at each stage of their career matched up against quotes, either from one of those Prime Ministers or about them. It was a really fun way to go through and get a history lesson, without having to read a big long text.

The Diefenbaker one — as American Presidents have their own Presidential libraries, Diefenbaker has his own library and centre in Saskatoon, in the spot where he’s buried. It’s a great institution, if you’re interested in John Diefenbaker and in Canadian history, and that’s where I tracked down the cartoons.

While I was there, the Director of the centre said, “Next year, the embargo on his personal papers is ending, and they’re going to be open to researchers. But very few researchers know that. So Thad, you’ve got an opportunity where you could edit John Diefenbaker’s personal correspondence with his wife, with his brother, with his two wives, with his mother, and do a volume.” So I ended up doing a volume called, Personal Letters of a Public Man.

But most of his letters were handwritten, and in those days, to get something from handwriting to typesetting, meant [you had] to hire at great expense a typesetter, who was willing to take the effort to decipher the handwriting, and put it directly into the typesetting machine. So it was a very expensive per hour cost.

However, that year was the release of “DTP”, [Desktop publishing](https://en.wikipedia.org/wiki/Desktop_publishing. The Macintosh was maybe a year old, the first Mac.

Len: So this was about 1985 or so?

Thad: ’85, yeah. Aldus PageMaker came out, Adobe released the first Adobe fonts that would work on the LaserWriter, on the laser printer. And it could be manipulated through a Mac.

It came to me as a suggestion to get hold of a Mac, which I did, and type up all the letters myself, with the notion that I would then give the disk to the typesetting company, and they would translate it into their machine language for the typesetting.

As it turned out, I didn’t check whether that actually worked. And when I got all these letters typed, there were hundreds of pages, and I went to the typesetting companies we knew in Toronto at the time, they’re like, “We don’t know how to read one of these diskettes. We don’t have any way to translate it. Sorry, but if you want this book published, we’re going to have to do what it was in the original thing. We’re going to have to retype them all on this Linotype system, or this Compugraphic system that they were using.

And it was like, “Oh man, all of that for nothing.” But my art director, who we worked with on this project, said, “I was just reading in Popular Mechanics that you can actually output from Microsoft Word version 1.05 directly to this laser printer. If we do that at 125%, and then condense it, shoot it down to 80%, that’ll tighten up the density of the type. And we can use that instead of typography, instead of a traditional typographic machine.

Len: Wow.

Thad: And so we did, and the book was published. Doubleday was the publisher, and we showed it to the editor and my art director at Doubleday, without telling them how we’d done it. We showed them a page spread, and said, “What do you think? Is it okay — running heads, folios, the typography?” They were, “Yeah, fine, sure.”

And so out came the book, with a little credit on the cover page saying, “This was produced with the LaserWriter, and on Microsoft Word,” because PageMaker actually wouldn’t be out at that time. That was just a few months before it appeared.

And when that was published, Apple heard about it and they called me, they got in touch with me and said, “Do you realize what you’ve done?” And it’s like, “Well, no. We just did this and–” “You’re the first ones who have actually done this. We were hoping that this would happen, but it’s never happened.”

And so we got together, and they ended up connecting me with a company that was going to become Canada’s largest reseller of this equipment and technology. I became the product manager for this line, and so converted from being a journalist, a publisher, into a tech really — but a tech in the publishing industry. So that’s where my career transitioned into where it is today, where all of the work I do is at that intersection of the culture of publishing and the technology of publishing.

Len: That’s a really fascinating story. I’m very surprised to hear it, and it’s great to hear that Apple found out and were aggressive and watchful enough to see this, and reach out to you.

I wanted to ask though, what did Doubleday do when they found out?

Thad: Nothing. They didn’t know what it meant. They were fine with it.

Len: And you ended up in Vancouver at some point, and I wanted to ask how that happened?

Thad: I went to San Francisco when I turned 30 and lived there for 15 years, wanting to be closer to Silicon Valley, and also get away from the damn winters. And it seemed great. The city’s fantastic, and it was so close to all of the technology of publishing. I thought that would be a great opportunity. I’m a dual citizen, because my father was born in New York.

And so that was a great move, and I was there till early in the new millennium. And then my mother got cancer. She lived in Toronto, and I moved back to Toronto to look after her for what ended up being three years. And then found myself like, “When will I go back to San Francisco, or what am I going to do now? I don’t really like Toronto. I got away from here, I don’t really want to be back here.” And a friend had an opportunity with a house in West Vancouver, right on the ocean. And so I moved out to Vancouver, and I’ve been there since. Very much in love with the city.

Len: Thanks for that answer, it’s great to learn how all these things connect from the past to the present.

Thad: You’re in Victoria.

Len: Yeah, that’s right — I’m in Victoria, British Columbia. I’m sort of relatively new here. I’ve learned that all winters in Canada are not bad.

Thad: Right.

Len: Just 99%.

Thad: Where did you move there from?

Len: I moved here from Montreal. I’ve moved around a bit myself as well.

Thad: Plenty of tough winters there.

Len: Yeah, and before that I was in England for a few years, so I’ve seen a little bit. The joke I tell to my American and British friends is that I now live on a Canadian island in the Pacific Ocean. Just something most people haven’t even really heard of.

Thad: Right.

Len: Your latest book is called, Mobile Strategies for Digital Publishing: A Practical Guide to the Evolving Landscape, and it came out, I believe, in 2015?

Thad: Yeah.

Len: I wanted to ask you — we don’t need to go into it in depth, but what are some of the mobile strategies for digital publishing that you write about in the book, and what’s your general opinion about the current state of affairs? I guess that is a big question.

Thad: The core kernel that is most important on this topic is this: that everyone has to recognize that the smartphone is no longer an adjunct, it’s no longer sort of, “Oh I have a smart phone as well.” Or, “I work on a computer, and I have this smartphone for when I go out.” They have to get their mind around the idea that people have smart phones, and then on the side they have tablets and computers. The smartphone is the centre of the universe for an ever-increasing majority of the people that are owners of electronic devices.

And so, for publishers who have always seen the smartphone as some kind of globally adjunct, to be, at best, accommodated. My message is: No, start by thinking about the smartphone. When you’re developing content, you should be thinking, “How is this going to display if someone’s reading it on a smartphone?”

And then you can figure out what it’s going to look like as a printed book. Because you know how to do that, you know how to make printed books. You’ve been doing that throughout your career. But do you know how to make something optimal for a smartphone? Of course they don’t, and they’re still not getting their mind around it fully.

At the time there was this distraction around apps, and we all thought that maybe the way to go, the way to embrace the space was via apps. And that hasn’t turned out to be true. A lot of companies have done some interesting innovative things around that space, but that hasn’t been the answer. And so is it in a browser? Or do we just accept that the default apps — Kindle app, Kobo app — that that is the interface by which books are communicated on smartphones?

Well, as of today — yes. Is that the way it’s going to be over time? I don’t think so. There’s a lot of transition still to take place. But my overall message to book publishers is start with the phone, and work backwards from there.

Len: And do you think publishers are heeding that message?

Thad: No, not at all. Not at all.

Len: And why do you think that is?

Thad: Publishers are — I don’t — there’s a — I’m trying to say it nicely. Publishers are not known to be technology-adept. It’s not been a technology industry, even though it is now a technology industry — which is a big argument I try and push to people. And in one of my presentations next week at DBW, I’m trying to convince the audience, in the same way you should be thinking of mobile first, you should be thinking technology first. You should be thinking of your publishing company as a technology company that also does this interesting artistic craft, these cultural artifacts. But this has to take place via technology.

But publishing companies are not in that mind space at all, to their great detriment, because they’re losing market share to self-published authors who are technologically adept. That’s not the only reason they’re losing market share, but that’s a significant part of it — until they get their minds fully around technology, they’re going to continue to be losing sales, as they’ve been doing over the last few years. And they’re going to continue to be outclassed by newcomers.

Len: That’s really interesting. It reminds me of an experience I had at the BEA, Book Expo America conference in New York, I think it was in 2013, when there was this panel of grand eminences, including the CEO of one of the Big Four, Big Five — do we say Big Four or Big Five publishers now?

Thad: Big Five.

Len: Big Five still, okay.

Thad: Yeah.

Len: I’ve heard both. So, a CEO of one of the Big Five, on one of the “Top 50” lists of the most important people in America kind of thing, was on the panel. And there were the heads of some other organizations. And I remember, the CEO or sort of higher-up of one big company actually picked up an iPad that he had sitting in front of him, and like turned to his right and waved it in the face of this big CEO, and said, “These things are real. It’s not a science experiment. They actually exist.” And there was, nonetheless, this wall, just this wall between him and the CEO of the big publishing company that was not penetrable. It was amazing to just see it. I mean it’s one of the things we all know, but to actually see it happening, in front of a big crowd as well, was pretty amazing to me.

Thad: I can well imagine. Yeah, they still see them as toys. It’s unfortunate, really unfortunate.

Len: It’s interesting what you say about technology as well. I mean, I’ve got this theory that there’s sort of two big cultures in corporate America right now. One is the sort of old time-y one, you might say, where domain-specific expertise is not necessarily required, or might even be frowned upon. And so an executive is supposed to be an executive. They’re supposed to be good at business, and it doesn’t matter what the business is. They have these universal skills that they can use, mostly networking and influence peddling and things like that. Those are important and powerful things.

And on the other hand, you have the domain-specific expertise leaders. A classic example from right now would be Elon Musk. Someone who is actually not afraid of typing and doing work, and literally getting hands-on. He doesn’t just go to the factory to put on a decorative hat and intimidate the workers, but actually really knows what’s going on.

It’s a really interesting question with software eating the world, as Marc Andreessen famously said — is it possible to succeed in business with no domain-specific expertise at the top anymore?”

Thad: No.

Len: You don’t think so?

Thad: It’s not. No, not at all.

To me it’s like — forget it. If your senior management are not technology-adept, if they’re not comfortable and fully aware of what technology can do for them, then they’re not managing the company anywhere near it’s full potential, period. I don’t even want to argue with you about that. And not you — with any of them.

Len: I understand.

Thad: It’s unequivocal at this point, from my point of view. They are losing competitive advantage every day they fail to put that expertise in at the highest levels of the company.

Len: And I imagine that as a consultant with technical expertise, this must be something where there are lots of people out there who I’m sure are aware that this is something they require, and that they need to find that expertise somewhere.

Thad: Yes and no. As someone who’s been consulting now for 30 years, it’s never been so difficult, it’s never been so challenging to get clients because of these problems where the easiest way to deal with the fact that they’re not technology adept, is just to ignore it. And so to ignore it, is to ignore that there’s consultants out there that can help them. I’m very fortunate that I am keeping busy. But it ain’t easy. It’s certainly not like they’re lining up outside my door.

Len: That’s really fascinating. I mean, the years keep ticking away, and the big publishers keep not responding.

I was wondering if you wouldn’t mind talking a little bit about some of the big news that’s been happening in the publishing industry — at least for insiders, the big news over the last, let’s say year and a half or so, about data and the analysis of ebook sales versus print sales?

One thing one hears from one side, is a celebration of declining ebook sales. And one hears from the other side no celebration because, first of all, on the other side, people like ebook sales, as one would think publishers would. But also, there is no decline in ebook sales, is what some people are saying.

I was wondering what your much more informed than my own position on that issue might be?

Thad: It’s a great question, and a complex issue. Let me try and give a really short answer, which is hard for me to do. But I’ll try and do that, otherwise I could sort of head off for 20 minutes on this. But you can then ask for some clarification.

Short answer: Yes, ebook sales are declining for the Big Five and for, as it turns out, a pool of about 1,200 publishers, who are measured regularly by the Association of American Publishers. There is a measurable decline. The factor that seems most certain to be the reason for that decline, is the fact that the prices have gone way up, as a result of a legal thing that went on between Apple and the publishers, and the Department of Competition in the US.

So they gave the publishers the ability to up the prices, and they did. And ludicrously so, in my view. But they did, and coincident with that, the sales have gone way down…. There’s one other issue, but that seems to be the biggest one.

At the same time, self-publishers are growing, growing, growing. They’ve reached some kind of a plateau recently. But the growth has been enormous over the last decade in self-publishing. And so all of that, 97% of self-publishing activity is digital, not print. And so all of where the big publishers are saying, “Our sales in ebooks are down,” while these other people who really are your competition, and you’re then….

Part of what they can’t get their mind around is that, “Well this little person, this little self-publisher, they’re not my competition.” Well, en masse they are your competition, and there are tens of thousands of them. Hundreds and thousands of them. And they are succeeding where you are failing.

And how are they succeeding? Both in pricing, of course, but also in understanding the technology, and where the technology intersects the marketing. They understand that far better than the large publishers.

And so that’s the real story of ebooks. No, they’re not declining overall.

Len: I watched an interview with you on YouTube, it might have been with Joanna Penn… where you talk about — there’s a kind of irony where, Amazon is obviously gigantic, and is a pre-occupation of publishers and self-publishers. But actually books — which it was known for initially — are a very small part overall of what Amazon does, and so the irony is that although books and ebooks, they’re a smaller subset of book publishing generally, are, to the rest of us, a huge industry — the book publishing industry is like 150 billion dollars a year worldwide, it’s bigger than other forms of media — but for Amazon, they’re so big that books are a small part of what they do. And yet, they somehow just effortlessly dominate all these other companies. And these companies have become almost entirely reliant on, or existentially dependent upon Amazon doing its job well.

I was wondering what your position might be — if you were CEO of a Big Five publishing company, or if you had been for the last 10 years, what would you do with respect to Amazon?

Thad: Yeah, that’s the huge elephant in the publishing room. There’s a lot of antipathy towards Amazon, and understandably so. But forget that. Almost all of my clients, Amazon is now their largest single customer, and growing. The numbers suggest that Amazon controls about 75% of ebook distribution in the United States.

So most of my data — I’m very US-focused, even though I’m Canadian-based — my career was built in the US. So you’ll forgive me that my figures generally are US-referenced. In some cases, I’ve also got Canadian data. But generally speaking, as we’re used to the publishing industry, in Canada it pretty much mirrors the structure of the US company, with some very significant differences. But anyway, to use US data is not to greatly mislead about Canada….

Len: We’re based in Canada, but basically our we have the same issue with our audience.

Thad: In the US, [Amazon has about] 75% control of ebooks — about 50% control of all book sales — physical and digital. So it’s like, the contest is over, Amazon won. And you can cry about that all you want. But it’s not going to help you at all. Amazon won.

A colleague of mine, Ted Hill, who’s running the DBW conference next week, has a lovely way of putting it, where he’s able to provoke a response or a thought about a response. “What if Amazon was not merely your biggest customer, what if they were your only customer? How would you run your publishing company if they literally took over the rest of the market?”

Which — in fact, I mean, if you track the trend — at some point, that’s not a completely ludicrous thing. But as a brain exercise, it’s really important to think through if there is only one distributor, and that distributor is Amazon. And we know how they behave. What does that do to publishing?

Well it’s not all negative, because Amazon’s a fabulous marketer. And you’re suggesting in what you say — yeah, books are an insignificant portion of their revenue. But part of what makes Amazon so awesome, awe inspiring, is that markets they don’t even care about from a fiscal perspective, they run them as if that was the only thing they did. And so in book marketing — they continue to be incredibly innovative, incredibly aggressive.

And it’s not getting any easier to work with Amazon in the same room, because you have to pay attention to every little movement they make. All of which are designed to sell more books. But also for Amazon to sell them at the expense of any other company.

They’ve decimated Barnes and Noble. Barnes and Noble, it’s just a very sad last few years for them. One can be critical of Barnes and Noble, and there’s lots of reasons to do so. But if you or I were running Barnes and Noble, we would’ve run it into the ground too, because competing with Amazon is a mug’s game, you just can’t do it.

However, from a publisher’s point of view, it’s like, embrace the beast — you’ve got no choice. And for the future of your company — you’re not single-handedly going to stop them. If anything stops them, it’s going to be a groundswell — a very innovative groundswell that none of us perceive at this point. But it ain’t going to be you trying to single-handedly work against them.

So for all of my clients, I say, “Embrace them wholeheartedly. Put on a big smile, even though you don’t feel it. Because these are the people that are selling your books better than anyone else in the world.”

Len: That’s a great, and in my experience, pretty original answer. It’s just so fascinating — you mentioned Barnes and Noble, and the Apple and the big publishers controversy from before. I mean it’s just — it really is in it’s own way, a kind of fascinating comedy. I remember reading a quote — I think from the president of Barnes and Noble, blaming declining sales on the election in the US, because people are afraid, and they’re just staying at home watching cable all day.

I mean, of all the things…. I think there was an article in Publishers Weekly, where they referred to the price fixing scam that was happening at many levels of the publishing industry as — I think it was “a government imposed price reduction.”

Thad: Yes.

Len: The willfulness of it is the thing that really fascinates me, because it it indicates that underneath, people kind of know what they’re wrong about, and why they’re wrong. But there’s something about what’s happened in the last 20 years to publishing that a certain type of person just can’t face up to. And what’s interesting is that it’s not like some kind of deep economics that you’re watching, or like business strategy. It’s something psychological, even at positions of prominence and responsibility. It’s down to some personalities, and their own preoccupations and, I mean, what to you is like something you accidentally discovered, which was the empowerment that comes from new technology, to other people, was Armageddon Day.

They just see the things that they associate with publishing, which were material. Like your typesetting, and stuff like that falling away and falling away and falling away. And they feel like — I think at least my view is that they feel like we’re losing literature, or we’re losing knowledge, because we’re not doing things on paper anymore, and -

Thad: It’s preposterous.

Len: Yeah, and there’s this really interesting conflation of the subject with the material.

Thad: The artifact. Yeah. We have to get away from the artifact. From the concept of the artifact. We have to look at each medium, however we like to do that, as an enabler. As one more opportunity to get the word out. To bring in new readers, to bring in new opportunities. These things are enablers. They’re not artifacts.

One thing you’re saying there reminds me of something a colleague told me 25 years ago. He said, “The only sustainable, competitive advantage is understanding and adopting technology faster than your competitors.” And the point of that with book publishers is, business is debugged. We know how to run book publishers in the traditional way. There’s nothing left to discover there. There’s nothing you can know that your competitor doesn’t know.

The only thing you can know that your competitor doesn’t know is how to, for example, use metadata more strategically than they do. How to maximize the efficiencies of the EPUB format better than they do. Understanding the supply chain, and how metadata informs the supply chain, faster than they do. Aside from that, you have no competitive strength. And that to me is sort of the summation of where we stand as an industry. Technology is the only thing that’s going to save you, let’s put it that way.

Len: On that subject — moving from the big to the small, I suppose — you very recently — just three days ago — released a report called, An Authoritative Look at Book Publishing Startups in the United States. I was wondering, it’s a very long list of companies that you’ve compiled there, and some of them are failed companies. Some of them are plugging along. Many of them were at least attempting to innovate in one way or another technologically. And I was wondering if you could talk — I mean, to begin with — a little bit about what the origins are of this report, and why you were interested in writing it?

Thad: Sure. About five years ago I was on this panel at Tools of Change on the topic of startups. And I was the contrarian on the panel. And the four other people on the panel were all like, “Thad, you’re just being very, very negative. These are amazing opportunities. These are some amazing companies. And they’re going to flourish. And the book publishing industry is fertile ground for startups. Your negativism is completely incorrectly placed.”

I can be a negativist. And so I was a little bit stung by that. But it got me to thinking, “Well, why don’t I go deeper on this, and really find out what’s going on here?” And it turned out to be quite an interesting rabbit hole. I found 900 companies, and started sort of hoarding, collecting.

Over the next five years — every time I heard about a new startup, I would download the information on that startup, if it came out of a blog post or something in Publishers Weekly. Or someone would tell me about it, I’d go to their website and get their mission statement, and add that to the spreadsheet. It kept growing — 300, 600. I did an interim report when there were 600. Then now it was up over 800, it’s like, “What am I going to do with this?”

Well, I decided in the end — I’ll distribute it freely to the industry so they can get a look at what the startup scene is. And I did some quantitative analysis of the data, to try and understand what areas these startups were working in. What kind of funding they got. Whether they’ve had any mergers. Whether they’ve been acquired. A couple of them had gone public. How many had gone out of business? About a third of them have. So that’s the knot of what’s in this report that came out earlier this week.

Len: Speaking of your negativity on that panel five years ago, I looked at your slide deck related to that talk, and I was curious — have your views changed about book publishing startups in the last five years? How would you characterize the current state of affairs facing startups?

Thad: Good question. As I was doing the report, the cynical side of me was reminded — some of these startups are so goofy. And not only is this startup number 231 goofy — that’s just a number — but then you find out that startup number 427 has got the same idea, and launched six months later. And so we have one goofy idea has not made this company successful, and another company’s coming in with the same goofy idea, and they’re going to try and do it too.

And so there’s a lot of that — as I was building this list, I’d find out about a new one, and I’d think, “Oh a new start up, going to add them to the list. Oh my God, their mission statement is the same as 23 others of these startups, a third of which are already out of business.”

So what I saw going on there, is — there’s a cult of startups, right? — that I’ve pointed out a little bit in the report. But we know it, right? I mean the media — whatever newspaper or website you’re on — as long as that company can say, “We’re a startup, and we’re looking to disrupt this or that,” the media eats it up, because the public eats it up, because there are so many glamorous stories around the magic billions that could be made out of thin air six months after startup.

I understand what the allure of that is. But people have to get down to earth a little bit more, and realize that just because they said they’re a startup, just because they’re enabled by the web, just because Mark Zuckerberg exists, doesn’t mean this is a good idea, or that this company’s going anywhere. So that’s the downside.

On the other hand, what you’ve got is a lot of smart people — committed, willing to put their careers and their livelihood on the line, to try and bring some real innovation to the publishing industry. And that’s a great thing. And that’s something that I keep reminding myself of. That bottom line is like, “Thad, forget about the dumb ones, focus on the interesting ones.”

And so I’m hoping in the months ahead on my blog to profile as many as I can that are the most innovative, the most intriguing. Even if they’re very, very small. There’s some that really do have some nifty ideas, and I’d like to spread the word about the best of them.

Len: One example of a type of startup that was backed by really smart people, and run by really smart people — and had talented staff, is Oyster.

I bring them up, not to pick on them, but because their particular approach was very interesting. It was a subscription-based model. I mean, in the end the end they had a book store, but primarily, their business was aimed at people who they believed would pay a monthly fee to have access to loads of books. There’s other examples. Scribd had something similar with romance books, that sort of went belly up. Oyster’s team — just for those listening — got acquired by Google, I think. Or many of their team, or some people on their team were.

Now the example of a subscription model that appears to be working is Amazon’s Kindle Unlimited, although that’s a very curious piece. I was wondering what your opinion is? Looking back on the last couple of years, do you believe in book subscription models? Do you think they’re something that can succeed for a business? And on the other side, do you think it’s something that a self-published author should put their material into?

Thad: Right. To me, at the time — it’s easy now retrospectively, because they have failed to say why it wasn’t a good idea — but at the time, before they failed, there was a lot of hype surrounding it. And I was thinking, like — subscription model, all the books that you would want to get hold of, isn’t that called the public library? And I think I’ve already got that, and I don’t pay an annual fee, and if my branch doesn’t have it, they have inter-branch loans.

So the only thing that made sense on the subscription model to me was, it’s kind of instant-access, not having to go through the queue at the library, or, for the popular books, you have to wait a couple of months if you want the ebook version, until it becomes available. Or even for the print version, sometimes you’re on a list for a couple of months.

So for instant gratification, a subscription model made sense. Except there too — it fell apart because no matter what, they were going to be missing some of the books you wanted. Beause they– Even if they had the Big Five, which they didn’t — but let’s say all the Big Five agreed to it, that’s only 50% of the trade books published by established companies, leaving out the self-published authors. Only 50% of the books published each year in the United States, come from the Big Five. So it’s another several thousand publishers that you need to sign up if you want the other 50%.

So it was never going to be a place where I could say to myself, “Oh, I’ll just go onto Oyster, because I can get it right away.” I can go onto Amazon, and I can buy it right away. So that’s already available to me, and it’s only a few bucks and I can have it instantly. So the subscription model to me was never a profoundly smart idea.

And what Oyster ran into was a particular problem, whereby they couldn’t get the publishers on board. And so they had to pay way too much for these books, which was not economically feasible, and hastened their demise. Scribd is still in business for various reasons. But virtually all the subscription ones that are on my list have gone out of business. And so to me, that was a combination of the publishers undermining them.

But also, the primary value proposition had not been fully developed. It wasn’t that compelling. I mean, is it our manifest destiny, as Kevin Kelly would argue, that every book is going to be online and available to us on a subscription model eventually? That somehow we’ll figure out the way to have one source. But would it be a public source? I don’t know. Will it be Amazon? Who knows?

But our manifest destiny, or the way it would best work for people is — yes, you could access any book at any time — in the moment that you’re interested in doing it. You don’t have to purchase it. You can have it, borrow it, with some sort of funded subscription model, or whatever it would be. That makes sense to me over time. How we’re going to get from here to there, I don’t know.

Len: And if you were advising a self-published author, or a group of self-published authors, would you recommend to them that they try Amazon’s Kindle Unlimited?

Thad: That’s another tough one. Go ahead.

Len: I was just going to say — yeah, for those who are listening — one of the interesting things about this service is that if you’re a self-published author and you put your book on there, Amazon imposes some restrictions on you, on what you can do with your book, and where else you can sell it, and how much you can sell it for, things like that. But also, as I understand it, the money for a subscription service just goes into a big pool. And an inherent and inescapable part of the process then, is that you have to decide how to divvy up the money, and that calculation is entirely up to Amazon. I think their latest way of doing it is to look at how many pages have been read in your book.

Thad: Yeah.

Len: And then this is something that immediately a lot of enterprising characters began taking advantage of. By figuring out, for example, that Amazon wasn’t actually counting the pages that were looked at. It was just looking at the highest number page that had been looked at.

So what people were doing was, basically, putting up more or less fake books that were hundreds of pages long — let’s say — and then having a link at the beginning of the book to the last page of the book. And then that would be sending the signal to Amazon that someone had read all 580 pages of “cat cat cat cat.”

And so there’s this inherent murkiness to what goes on in a subscription service like that. And you’re also susceptible to change. In the same way that internet companies can be screwed over by Google changing it’s algorithm for search results — like one day you can see your ranking fall dramatically. Similarly, you’re exposed to Amazon just deciding one day to count differently, and divvy up those funds differently.

Thad: Yeah, it’s a hornet’s nest, absolutely.

So let’s think of the downside of it. The biggest downside is they demand exclusivity. You cannot distribute your book outside of Amazon, if you want to be part of Kindle Unlimited. That’s a big restriction. And for certain authors of certain genre fiction, to get that extra boost just through Amazon, can offset the potential loss of sales from other distributors.

But if you’re an author of substance — if I can use that as a broad rubric to suggest the more committed fiction writers, and all of the non-fiction writers who are trying to produce books that really add to the canon — it’s an unacceptable restriction. There’s too many other ways that you will miss being able to connect with readers, if you stick with that restriction. So it works best for genre fiction. And it is a chunk of change.

I mean they put 15, 16 million dollars a month into that fund. So in fact, if you think about it on an annualized basis, it’s upwards of 200 million dollars, of what amounts to royalties that are going to the self-published author community. Economically, it’s very, very significant.

But the bottom line for me as a self-published author is — as well as all these other things — the books I have that I self-published, my books are technical, they’re not exciting mysteries. It’s out of the question that I would use Kindle Unlimited. It would just restrict my audience far too much.

So then, who are the audience for Kindle Unlimited — they do have to pay a subscription fee. In the old days, when there was only romances, where the only really consistent genre was, like Harlequin in those days, they would publish weekly, get a new book every week. There was a defined audience of people who wanted a new romance per week, and they weren’t particularly fussy as to which author. They liked a particular sort of sub-genre within romance.

Well that’s spread now, where there’s a lot of people who feel the same way about certain kinds of mysteries, certain kinds of thrillers. They like to get a book a week, or even more than a book a week. And this fulfills that. There’s enough good stuff in KU, that it’s cheaper to be a subscriber to that, than it would be to buy those books individually. But that’s a pretty narrow use case.

And KU, Kindle Unlimited gets a lot more press than it really needs to. I guess it’s become sort of emblematic of Amazon’s power, and ruthless use of power, and ability to — as you say — change the rules according to their whims, and to be exposed to scammers. So it is a very visible thing. But in terms of, really, its significance as a phenomenon for the book publishing industry, it’s much less than meets the eye.

Len: Looking towards the future, in a recent blog post, you wrote — and I’m going to quote you here: “Central to the future of publishing, is understanding where AI intersects with traditional book publishing.”

I was wondering if you could talk a little bit about where you see artificial intelligence intersecting with traditional book publishing and why you think that might be a powerful feature of the future?

Thad: The starting point for my interest in AI and book publishing — well it actually goes back a few years. Let me try and give a short answer and see how that works.

There’s a category of publishing technology that has to do with expressing the meaning. Rather than just saying — if we have a whole paragraph describing a politician’s education, just the education of that politician, and we’ve got 400 words within a much larger biography of that politician, that talks about her time in high school and college and the education that she gained, well, the abstraction of that is semantic. It would be xyz politician — education, college and secondary education. So there’s that abstract layer, in which we describe the meta meaning of a type of content. And so, when you get at that meta level, you have another layer at which you can process content. Where you don’t have to regurgitate every single word in the book.

And so we can get it to the point– Like if we can abstract it to that level, well then someone who’s interested in the education of politicians can do a search of everything that’s been classified at that semantic level, and be able to much more quickly find [what they’re looking for], because it’s not a whole book about that; they could find all of the sub-sections within all of the published literature to cover that particular topic.

And so it’s a potentially very powerful way of classifying this enormous, uncontrollable, multi-billions of words that have been published, and remain in print — from books that have been published over the centuries, literally now. And this then starts to suggest that there are data mining methods that might increase our access or understanding of published work. That was a starting point for me — trying to understand the semantics and how that would intersect with book publishing.

The next thing that happened is Google, Yahoo and Yandex … in Russia came up with a standard called schema.org, which is based on semantic representations of content, and allows those search engines to understand content better.

If you make the effort to explain what the content is at a semantic level, those search engines can do a better job of locating your content. So that put a big impetus on the publishers to start to get a handle on semantics as well.

So then, the next thing that happened that triggered me, was this book that you would’ve seen on my blog, where I reviewed it in a couple of posts, The Bestseller Code. The Bestseller Code is a fascinating book, where the two authors are both scientists.

It’s not just an exploitative look at some secret little method that they came up with, that’s the magic bestseller code — no, these people have done some very extensive analysis of the patterns of the words in bestselling books, of the sentiments that are aroused within, by those words, types of characters, the plot — all those sorts of things.

And they have come up with a formula, a code — the “Bestseller Code” — that identifies the commonalities in the books that have become New York Times bestsellers. And while they aren’t willing to offer it as a prescriptive — i.e. “Do this, your book will become one of those best sellers” — their interim report is really what the book amounts to, where they’re able to say, “We have in fact come up with a scientific quantification of books using the techniques that we now put under the broad rubric of artificial intelligence.”

Things such as sentiment analysis, text mining — you know what I mean. There are a whole set of secondary technologies that are thrown under the umbrella of artificial intelligence. And so this was the first really big, obvious use of the tools of AI towards a specific outcome.

And certainly as a publisher, you just have to start scratching your head and saying, “Okay, so could I use some of this text mining, sentiment analysis to evaluate incoming manuscripts and be able to make some kind of a qualitative assessment.” Based not just on human reading of that. Maybe you could? These are some of the questions that are posed by that.

So coming back to the question that you’re posing to me. What my feeling is, is that the science of artificial intelligence is coming forward by leaps and bounds. The publishing industry has enormous amounts of data in the form of text. And there is an intersection point between those two things that we’re just beginning to get a look at, but which I think is going to be quite profound in the years to follow.

Len: That’s a really great answer. It was sparking in me thinking about Netflix. Netflix is well known for having highly detailed metadata around its shows. It can be like, this person likes movies that are kind of like Quentin Tarantino movies, which have characters named Joe, who dies in in act three, when something falls on his head. And they could categorize the same show many different ways like that.

What people talk about, is how they’re actually using all that viewer information, both to encourage the discoverability of things that people will want to watch, but actually also to create new shows. They base their decisions for what to do, what shows to choose, and perhaps how they should be written, what should happen to the characters in them, based on this information that they have about people’s viewing habits. And it’s a fascinating future ahead of us, where that kind of data can be used to make decisions, and put it in front of people, and then iterate on it and see — well did that work, or did that not work? You can follow the experiment through and look 10 years ahead to what you want to do.

Thad: Yeah. The Netflix example is a particularly good one to spend some time with. My brief comment on that would be, I was initially quite thrilled to learn about what Netflix has done. And you’ll probably remember the Netflix Prize, where they offered a million dollars to a team of scientists that could improve the accuracy of the recommendation engine by, I think, 10%. So it’d get 10% better at successfully recommending a next viewing product to customers.

There’s a whole separate story that is a fun little story on how it played out. But it points to what you’re talking about. At Netflix, they have a lot of scientists on staff looking at how these things can be solved, towards the ultimate task of — can we develop shows, whole shows — where we plot them, develop the characters — which kind of actors are we going to hire? What’s the plot arc going to be? And can we create successful series on that basis?

House of Cards was the first poster child for that outcome. What I point out to people these days is, get a listing of Netflix series that have launched in the last three to five years, and find out how many of them are still being broadcast. They’ve had a lot of failures.

And but people keep talking about their successes. They’ve had a lot of failures. And this is no slam on Netflix. It’s only a reminder that this technology is still very much in flux, and that people are not robots. So you cannot predict with 100% dependability or anything near that number, using science. But you can get closer to it. And so the smart, the best users of the technology are those who fully understand its limitations.

Len: Thanks for that — on the note that people are not robots, it’s something I very much agree with, and it’s an optimistic thing — although I do like robots, too.

I just wanted to say, thanks very much for a fascinating interview, and for giving me and all of our listeners the time. Good luck at the Digital Book World Conference in New York this week, and all the best from me.

Thad: Thanks so much. I enjoyed chatting with you. You are better informed than anyone I’ve spoken to in the last several years, in terms of preparing for the interview. I’m really pleased, because it just made it a lot more fun that you’d taken the time to look at things and ask such smart questions.

Len: Well, thanks very much — and I’m going to leave that in.

Thad: Okay.

Len: Okay, thanks Thad.

Thad: Great, my pleasure.


Originally published at leanpub.com.