Applying User Testing to Technical Documentation

Can it help improve documentation?

Me trying desperately to install your project.

If you work in a team with lots of codebases, projects and members you may have come across this:

• 🎉 You’ve started on a new project
• 🏃 You want to install and run the codebase on your computer
• 👀 You look at the README document
• 😰 You can’t get it to install. Ahhh!
• 🗣 You shout for help in a Slack channel or ask your teammate very politely
• 🤝 You get help
• 🚀 You end up getting a project running!

My assumption is without good documentation this will keep happening. Is it wasting time? Is this information stuck in a few developers heads?

Is there was a way to improve documentation through user-testing?

User testing web sites

When creating a product such as a web site, you should ensure what you’re making is usable. You can work out what you need to fix through user testing.

Don’t Make Me Think by Steve Krug has some good fundamentals and so I’m paraphrasing some stuff from here.

You may find the following likely in a user testing session:

1. Before and through-out tell them to say their thoughts out loud as they go so you know what they’re thinking. Record their screen for later. Stream the session over Hangouts/Skype/something so other team members can watch
2. Get an idea of your user’s background and experience: “What do you do? How long do you use the Internet in a day?”. To make them comfortable and to understand how they’ll approach the tasks
3. Get a user’s first impressions of what they are using and what they could use it for: “What do you think this is? What would you do first?”
4. Ask a user to carry out a task: “Find a book you want to buy, or a book you bought recently”
5. Review the results as soon as possible and find potential things to fix

Applying user testing to documentation

If we want to improve our documentation, we could apply this way of testing. The steps are very similar, here’s some examples of how the steps may look like for documentation:

1. Same as before
2. We may want to know their background. Programming languages or experience with a framework.: “Have you worked with the framework Angular before?” “Are you familiar with dependency software like Composer?”
3. First impressions: “What is this project?” “What does it do?”
4. The tasks might be: “Install the project”, “Run the tests”
5. Review the results as soon as possible and find potential things to fix

What we found

After doing a couple of these sessions we found some things to improve:

Copy and pasting

There were a lot of commands to get started. The developers would copy and paste these. Yet the way they were in the document was hard to do so:

First use npm install to install dependencies. Then run npm run dev to run the project in a development environment.

To:

Install the dependencies:

npm install

Run the project in a development environment:

npm run dev

Dependencies and preexisting set-up

When writing the README we were assuming which Node version the user was using. We also assumed developers had certain dependencies for node modules installed.

These parts get overlooked as docs often aren’t written from the start of a project.

Too much documentation

One thing we found was there was too much detail. For example, deployment instructions aren’t needed until later on.

We moved these into separate files and linked to from the main README.

How do you improve your documentation?

This improved our documentation. I haven’t had to explain how to set up that project since fixing the major problems we found.

Do you review your documentation? How do you think this compares with Open Source projects?

Interested to hear what other people do.

Cowboy GIF source: http://www.animationlibrary.com/sc/133/Computer_Programmers/