From Paper Folders to Digital Streams

9 min readJun 7, 2016

The Tragedy of Document Management in Government

A Tragedy

Let’s start with the problem:

Government bodies spend millions on so-called Electronic Document and Records Management (EDRM) technology.
But users don’t use it.
So they can’t find stuff, share stuff, secure stuff, delete stuff, archive stuff, …

Let’s say that last bit again — in 2016, some organisations can’t easily search all their information, some don’t even know what they’ve got.

These “EDRM” products were extremely expensive, from big-IT vendors like HP, Autonomy, and Documentum. I know of ~£35m spent over 7 years at one organisation. Of about 13,000 active users, only about 400 were even trying to use it.

A 3% user adoption is … tragic.

That £83,000 per user could pay for human assistants to do an employee’s filing for them.

Post Mortem

So what went wrong?

Information Management and Information Technology imposed EDRM on users, and told them to

Suffer several clicks and illogical navigation just to create a document.
Adhere to unrealistic document naming schemes.
Manually file documents in structures created a decade ago and fossilised since.
Spend time and mental energy adding metadata to documents … from drop-down of dozens of fields accreted over years … and blocking the user until this was done.
Raise a request for a new folder that takes days to turn around —hours, if you’re lucky.
Wait days to share a document while permissions are approved by someone somewhere, and set by someone else somewhere else
Fight with management to get a 200MB storage quota upgraded.
Worry about prosecution because you don’t trust the search results to be complete or correct.
Remember to go back through your old — out of sight and out of mind — documents and pick out the special ones to declare as records. Whatever “records” are.
…

You get the idea. Nobody can blame users for not doing any of this.

Nobody can blame users for, instead, doing what they do in their own digital lives— use an intuitive online service that stays out of your way, makes content creation and collaboration friction-free … fun even, and liberates their intensely personal choice for how they organise their information.

Back To User Needs

Faced with such disasters — it is critical to go back to basics— users and their needs.

User and their needs will be different for each organisation, and each team within that organisation, and even for some individuals. You should do the user research, and not assume these needs.

User research is not a chore — it’s an opportunity to find elegant solutions that actually work, and avoid wasted investment. Why wouldn’t you do it?

It’s useful to look at some of the common findings from such user research:

Most users are not records managers responsible for “retention and disposal” and the transfer of information to the national archive.
Most users are not in specialist roles requiring a high level of traceability and assurance around the creation of content — like a physicist doing safety calculations for a civil energy nuclear reactor. Even then, the rigour only applies to specific tasks, not everything they do.
Most users just want to get on with it — create content, organise it in a personal way that works for them, and find information that’s relevant to their current thoughts. All with a zero-friction user experience that doesn’t distract them from their thinking.
The users who operate transactional applications — like caseworking — would be happy for the application to constrain their workflow to make it easy (low cognitive load), efficient (no need for workarounds), and safe (avoiding common errors). The applications create and organise transactional documents — not the user.
Users will find workarounds if the tools they are given are painful. We can’t pretend it doesn’t happen. Getting the experience right is critical — because the official tools will always compete with their own personal tools — and their own choice might not be sufficiently secure or compliant with the law.
Some users of your information are outside your organisation, and may live in the future. They include historians, inquiry investigators, police, academic researchers, maybe even social entrepreneurs. Sometimes information management practice becomes disconnected from these users, and becomes a thing itself.

Get out of the users’ way — simple, right?

… so why the worried looks from Information Management?

“Digitalising Paper Processes”

Information Management in many departments grew out of the teams that managed paper. Way back when.

Back when stuff was dictated and typed onto paper, manually labelled, manually indexed and physically stored in some organised way. By assistants.

Back when:

you couldn’t do full text searches on the content.
paper took up space so you had to periodically get rid of it.
storage space was costly.
access meant physically transferring paper, maybe after taking a copy.
every access carried the risk of loss or damage to the only copy.
…

Information Management is trying to bend habits from the paper era to fit today’s digital world — “digitalising paper processes” — that happens a lot, sadly.

Look Before You Leap

Let’s not leap to an answer just yet. The best strategies look around at the landscape, get a bearing and work with — or navigate around — nature’s global forces. They don’t set out to fight them — like pushing water uphill.

1. Data Volumes And Kinds Are Exploding

More and more and more stuff is being created at an ever faster rate. In ever more diverse formats and repositories — from smartphone note-taking apps to Google Docs, from Github to Twitter, from Slack channels to wikis.

Today’s information looks less and less like a “document” — a labelled folder with paper pages in it. And important stuff, key decisions are not all being recorded in such a “document”.

The digitally comfortable have developed their own strategy for working with an apparently unmanageable torrent of information — through social timelines and feeds. It’s the best model out there right now.

2. Storage Costs Are Falling Exponentially

Storage costs today are falling exponentially. Cloud platform suppliers are fiercely competing to slash costs, roughly by 60% every 18 months — even offering unlimited storage.

This week Samsung announced a superfast 512Gb SSD the size of a postage stamp — so your organisation’s entire information will fit in about 40 postage stamps, and costing about as much in a few short years time

3. Search Technology Is Getting Cleverer In Leaps

Search technology is getting cleverer and cleverer. Even if many departments can’t do it, full text searching is considered a prehistoric basic capability.

Today, information retrieval tools can do things like:

find information related to your query, even if your query didn’t mention the exact matching words — finding stuff about medics when you searched for doctors.
automatically identify the subjects and themes in a repository to help you navigate it — do you mean apple fruit or apple computer.
identify noteworthy entities, like names, locations, addresses, currency, dates, organisations, etc.
continuously improve accuracy and relevance by learning from observed user behaviour.
understand natural language queries — smartphones now have scarily intelligent assistants that you can talk to — Siri, Cortana or Google Now.

And all these advanced capabilities are being rapidly democratised and commoditised — through open source and accessible APIs becoming common. They’re not that exclusive anymore.

4. User Expectations For UX Keep Going Up

Your organisation’s tools are in constant competition with products and services your users can use elsewhere in their digital lives.

Those cloud services are fiercely vying for users. Users who have no problem switching to competitors in an instant. That’s why the user experience has to be top-notch. Any annoyance, any friction, and you’ve lost users — which can mean the end of your business.

Training and user manuals are a thing of the past. That’s how usable and intuitive your tools need to be, and can be.

A Way Forward

All this points to:

You can’t predict what future historians will want to investigate anyway, so don’t permanently limit them with subjective and politicised decisions today about what gets archived and what doesn’t.
And future technology will enable future historians to do undreamt of things with what we leave them.

A subtle difference, but this is drastically easier than reviewing everything against a vague notion of what might be worth keeping. It means better quality decisions too.

Working with, and organising, information is intensely personal. Fighting this will always result in people using their own tools and information stores.
Better to keep track of what tools people are using, and what information they’re creating there, than not know they’re doing it.

Search is one of the few widgets users universally want to use.
Imperfect search is is better than the status quo, where findability is disastrous in many organisations.
Machine algorithms for search and classification are repeatable, consistent and transparent — human decisions are not.

No, But …

Information Management, and most enterprises, will initially baulk at this.

So it’s worth revisiting what information management really needs to be — rather than what it has become, with the perception that it is something complex and difficult.

I started some work a while back with the national archives to demystify and clarify what Information Management should be. It looked something like this:

Search — being able to find stuff
Audit — knowing who did what, when, including creating stuff, editing it later
Security — being able to limit access if stuff needs additional protection
Legal — deleting stuff you shouldn’t hang onto anymore, like personal data
Ownership — finding out which person is practically responsible for a piece of information

That’s it — it’s really not that complicated.

And guess what — you don’t need additional expensive EDRM products to ruin that modern, user-friendly, cost-effective, zero-friction, collaboration platform you just adopted.

A Lesson In Approved Truth

The Hillsborough investiations and inquiries are a sad, but powerful, illustration of why traditional Information and Records Management is deeply flawed.

The official records didn’t reflect the truth. There will always be a political dynamic which affects an individual’s decision on what to destroy and what to preserve.

The truth depended very significantly on information that wasn’t formalised as documents — individual notes, personal collections of information, rough sketches, important-at-the-time scribbles. This “disorganised informal information” was the most informative — not the officially approved records.

This example is repeated again and again and again — police, financial regulators, coroners, historians, social workers — all search through the mass of informal information, because that’s where the meaning, the context, the motivation is — not clean officially approved records.

So what’s the point of preserving information — if it isn’t to maximally inform future generations?