Thoughts on Stack Overflow’s Documentation beta
First off: I was really sad to hear that Stack Overflow’s documentation beta was shutting down. I’m a tech writer and sure, you could see that project as a threat to the profession. But it would have been exciting to see innovation in the field, and I was really curious to see whether or not it would succeed.
Plus, Stack Overflow were in a great position to do something big. SO is a hugely widely-used way to get information, often the first port of call for programming questions. As they say themselves, answers there are basically unofficial documentation already. So I wanted to see if they could use that position to come up with a great solution; I’m sad (but not surprised) that it didn’t work out.
Their solution was basically this: their Documentation site was a repository of user-written code examples, supported by some — not necessarily much — explanation and extra information. Really, it’s a site of examples, not documentation as it’s traditionally thought of (and that might have been a better name — they did consider it.) It’s not very structured — it just has a bunch of topics under a particular tag for the technology documented.
They thought this would solve a few problems, which I’ll talk about below:
- Docs are often an afterthought
- Docs are often written from an internal perspective, so they don’t give developers what they need
- Docs aren’t heavy enough on the examples that help developers get started
SO identified a lot of the problems with the current state of documentation correctly. But the solutions they came up with had some serious flaws.
Docs are often an afterthought
Absolutely. And I’ve seen that first-hand from the inside of companies. This happens for two reasons: firstly, because sorting the docs out doesn’t seem as important as building the thing itself. Not all companies accept that, for developer products, if you don’t document it, that makes it incredibly hard (sometimes impossible) to use. Secondly, because it takes a significant amount of work to make quality, comprehensive docs that are kept up-to-date.
“We’ll get the community to do it” seems like an obvious response to the size of that work. But getting that right in itself is a massive effort. It is possible — for example, Mozilla do a great job with this on MDN — but their success has been based on dedicating money and staff to making it happen. SO actually noted in this in their sunsetting post:
But it was also clear fixing Documentation would require a significantly larger team.
And seeing as most companies suggest “we’ll get the community to do it” as a way to avoid expense, it’s a bit of a non-starter.
Docs are often written from an internal perspective, so they don’t give developers what they need
Again, absolutely. Possibly the most important part of the job of a tech writer is finding out what the user perspective is, so that you write the right docs. If you don’t think about users, what you write most likely isn’t actually useful. It’s why designing content has to be so focused on user needs.
But just because you’re a user, it doesn’t mean that you understand what users in general need. It doesn’t mean you understand users with different use cases to yours, or can reason about the breadth of content that needs covering.
Building the right set of documentation takes a lot of hard thought; insights from users help you towards that, but they’re not a magic shortcut that gets you all the way.
By the way, this is why Q&A is so powerful and useful: users actually tell you what they want to know! This kind of feedback is invaluable as a tech writer, but it’s rare to get it in the volumes you do on SO.
Users have knowledge that could fill the gaps in existing docs
They certainly do: SO answers demonstrate this clearly. But just because they have that knowledge doesn’t make it easy to actually fill the gaps. There are more blockers to that: knowing the right place to put it so that it’s findable, knowing how to explain it well… And the bigger the change, the harder it is to make.
Again, from the sunsetting post:
Our interviews showed even very experienced users of T-SQL felt inadequate to contribute documentation. Users with less Stack Overflow experience tended to be intimidated by the prospect of making even trivial edits.
Docs that are open to contributions are a really nice middle ground on this point. They mean that you can have the advantages of having someone dedicated to working on them, who can make those contributions that people are intimidated by, but also let the community fix the issues that they know how to fix.
Examples help developers get started
SO felt that a major gap in existing documentation was providing enough examples. Developers love examples — all the evidence suggests that, given a page of documentation, they tend to skip straight down to the first code example, and most will just copy-and-paste that and get going. (That’s why you should be really careful with providing example code that shows bad practices or things you shouldn’t do: most developers will just copy it without looking at the big red warnings above.)
And yep, examples are really useful. They’re definitely a shortcut to getting going quickly. But examples aren’t a panacea. They aren’t a magic solution to always getting your user to achieve their task, though they might be good at answering specific questions.
If you have enough examples that they cover all variations, that might go part of the way. But if there are lots of variations, then how do you work out which one you need? If there are loads of examples, how do they fit together? And if they don’t have good supporting documentation, it’s hard to extract the one thing you care about from the stuff you don’t.
Things that are hard in the world of tech writing
As well as that, there are a few known-to-be-hard problems in the technical writing world, which SO’s solution didn’t address.
The other end of the long tail
One of the hardest problems tech writers face is working out what to cover, in what detail. There’s a huge long tail of questions that users will have, and we can’t do everything: we have to compromise, and pick the biggest questions for the largest number of users.
SO is great for this long tail of user questions: it can fill those gaps, and much better than we can — because we have to anticipate those questions, whereas on SO they’ve already been asked.
But dedicated staff have an advantage when it comes to the other end: the common issues, the core workflows. It’s much harder to deal with these big, broad questions with just drive-by fixes; dealing with those often means you need to think about information architecture and what the content design of your docs should be. It’s not as simple as filling in the blanks.
“I can’t find it!”
Another big problem tech writers work on is making it so their users can find the information they need. This is why we write task-based documentation: because it’s focused on what users are trying to get done. But the questions users have are often really specific. This is another thing that works in SO’s favour: when a developer searches for a question, it might well match the phrasing of a question that someone has already asked.
The “I can’t find it” problem requires work on a lot of different solutions. SEO and searchability is one of them: making sure you’re phrasing your docs in the same terms developers use. But it also takes good information architecture, which I can categorically tell you is hard. Finding an intuitive and easy-to-navigate structure for a disparate body of information, that’s logically organised but also based around what users need… I’ve spent the last year working on-and-off on trying to get this right. Generic tags and unstructured topics, like SO Documentation was using, can make it incredibly hard to find what you need.
Maintainability
Another complaint SO mentioned was that documentation gets out of date. This work — making sure docs are updated — is huge, and massively tedious, and if I’m not careful it means I end up having no time to improve our docs or make them more useful for people.
It’s one of the reasons for not having examples for everything. Examples that are broken because they haven’t been updated are frustrating at best and useless at worst. My team has limited resources, so I have to say no to things we can’t commit to maintaining.
There are some ways to improve this — automated testing of code snippets is one we’re working towards right now — but it doesn’t take away the work of fixing what’s broken. So for every new thing you create, you have to make sure it’s worth the effort of maintaining it into the far future.
Conclusions
To wrap up: it’s easy to identify the widespread problems that documentation suffers from, but it’s hard to solve them. Doing docs well involves a lot of effort: from working out what needs to be produced, to maintaining it and iterating based on feedback, to keeping the whole structure comprehensible, navigable and useful. Writing new content is only a small part of that.
Some of that work I wouldn’t want to find ways around — for example, I love working out what will help our users, or finding ways to explain complicated things clearly. But it would be brilliant to see people try to solve these problems in innovative ways. If anyone could save me time on the tedious bits of my job, and leave me time to work on bigger and more interesting problems — that would be great. If you work out how to do it — I want to know!
Appendix
Related posts, if you fancy reading up on the SO Documentation project:
