Reporters and Data and Robots: Why 2018 will be the year of automation in news
On November 30th I clicked a button to run a dataset on birth registration records through a story template I had written in a natural language generation programme. Seconds later I had hundreds of stories ready, one for each local authority area in the UK. We sent out 35 versions of the story to selected news titles across the country — the world’s first automated local news agency was in business.
In the first month of operations we have generated 10 stories based on UK open data and distributed multiple local versions. These are being published by leading regional and local titles. The usage has ranged from a few paragraphs online through to front page splashes in print, background content for special coverage or the factual ammunition for a leader column.
This experimental work on automation assisted journalism is being done through the RADAR project, a partnership between Urbs Media and the UK’s leading news agency, the Press Association. RADAR, or Reporters and Data and Robots, to give it the full title, is being largely funded by Google’s Digital News Initiative, a fund which supports innovation in the news industry in Europe.
The aim of RADAR is to use the emerging technologies often described as ‘robot reporting’ to help the traditional and financially troubled world of local news. In most countries the strain of shrinking revenues has led to fewer reporters in newsrooms. The basic job of getting out there and reporting an area and its residents has got harder as more journalists are tied to their desks churning copy, often driven by generating clicks not reporting what’s important. So, how to help?
Urbs Media, the business Alan Renwick and I started together, grew from the idea that there were important and interesting stories within the increasingly large amount of open data that was being released in the UK. Hard-pushed reporters lacked the time, and sometimes the skills or confidence to tackle the data.
As soon as we started to mine open data we realised that the value went beyond a national agenda. The granular nature of much of what we were working with could deliver stories for every local news outlet. It made no sense for a reporter in each newsroom to be writing their story from their small portion of the data. It made sense to do this centrally.
We turned to natural language generation (NLG), not a new technology, but one that had had limited application in the field of journalism. Some high profile projects had succeeded in automating stories from financial reports or sports results. But our starting point was different. We wanted to use the technology in a more flexible way.
Previous experiments had involved journalists working with developers and NLG specialists to build templates that could be consistently used across recurring data, such as sports results. We wanted to build a fresh template for each dataset and use the technology in a story-specific way to reveal the various angles and produce multiple versions of the story for local news publishers.
To do this in a daily news cycle the technology needed to be in the hands of the journalist. NLG had to become a part of the tool kit, a way to write a story.
We worked with NLG company Arria on the beta development of a software which was easier and quicker to use. And we developed working methodologies to find stories in open data, to structure data so it would work within NLG and to build templates that can bring out key angles and offer the strongest, most interesting take on the data for any area.
From this process we provide data driven stories in the form of ready to publish copy. This can help journalists in two ways. It can lift the daily load in churning copy — we’ll provide strong, verifiable, localised stories across a range of familiar beats including health, crime, transport and education. This should free up time for reporters to pursue leads, report a court case, attend a council meeting or get out and meet their contacts.
In other cases our stories can be built upon. We provide the grunt work across the numbers and the basic copy with the best story angle. This saves a journalist’s time, but the story can be improved with additional local context and reaction — the sort of work that can be done only by a local reporter.
We are working with a pilot group of titles drawn from all the main local news publishers across the UK. Their feedback is helping us to develop our service. In a month we have learnt a great deal about how to improve our processes and how to tailor the copy we provide, but it is still a work in progress.
The RADAR technical development is continuing. We are building data management tools to organise and interrogate our source material for the specific local news task. We are continuing to work with Arria on the functions of NLG Studio and how it can work best for news. And with our partners, the PA, we are building a smart distribution system which will match the geographical hierarchy of all localised UK data to the relevant news outlet to enable bespoke streams of content.
When that’s done we’ll be adding graphical elements and video to the package of content we send on each story to each customer.
The project is technically and editorially ambitious. We believe that stories delivered through automation will be an essential part of the local news industry in the future.
So will robots soon replace reporters? That’s the fear we hear voiced in some quarters. Our experience to date is that the best results come from the combination of the two. Automation can deliver a mass of copy, but the nose for a story, the context of what is newsworthy and the skill of the writing great copy are still best delivered by a reporter.
Our mantra at Urbs has been ‘written by a human, produced by a robot’. That still holds true, but 2018 will certainly be the year when automation will prove its worth in local journalism.
Gary Rogers, Editor-in-Chief, Urbs Media
