A new statistics service for the Finnish Immigration Service

Towards the end of last fall, we at Lucify started a project with the Finnish Immigration Service to redo the statistics section of their website. The end result was released at the beginning of this month. We wanted to share some of the thinking behind the project in this post, along with what we learned on the way.

There were many reasons why a new statistics service was needed. From the Finnish Immigration Service’s point of view, the drudgery of the previous system — hand-producing a monthly statistics PDF — was time that could have been spent on more meaningful activities. With the surge in asylum seekers during the fall of 2015, everyone at the agency had their hands full.

On the consumer side, we wanted to offer the data in a better format than PDFs. Since data in PDFs is not structured or easily machine-readable, building on top of it is difficult. Additionally, the static nature of PDFs does not use the full potential of the digital medium. By adding interactivity, we would be able to communicate a much broader data set in a more intuitive fashion. Reporters, citizens, and other interested parties would be able to find answers for themselves — answers that, until now, were only available from Finnish Immigration Service employees. Again, more time saved for the agency.

A one-page specification

With these goals in mind, we had a short session with the agency to sketch a high-level overview of what the service should do and what it should roughly look like. The annotated version of the image below ended up being the specification for the project.

The high-level concept for the project

Many people — especially those used to heavy, up-front definition phases — were shocked by how lightweight the specification was. I won’t go into the classic debate between lean software practices and the traditional waterfall process here, but suffice to say: this type of project should not use heavy planning before the data has been gathered and development has started.

Defining the fine details of what a data visualisation should look like before getting access to data is, at best, time spent inefficiently or, at worst, time wasted. Data is required in order to explore the different possible ways of presenting it, while constant feedback from users and stakeholders influence the design. If the outcome is locked into place before either of these happen (via a specifications list in a contract), the end result will be far from ideal.

Design decisions and implementation

Design-wise, we had a few challenges. One was related to anonymity: how could we provide detailed statistics about applications, decisions, gender, age groups and nationalities without giving up too much information about specific individuals?

Together with the Finnish Immigration Service, we decided that showing counts for five people or less is acceptable as long as it happens on one only dimensions. In other words, it is ok to show that one person from Vanuatu applied for citizenship in a certain month as long as there are enough people (over five) applying during the same month. On the other hand, if a certain month has five or fewer applications (or decisions), we do not show demographics.

The service in action

Another design-related challenge was the complex, tree-like hierarchy of decisions and applications used at the agency. The Finnish Immigration Service handles a wide variety of applications, ranging from residence permits and citizenship to international protection. We needed a way to let the user drill down into the types of applications or decisions that they were interested in in an intuitive manner, all the while showing statistics from the currently selected level. We ended up building a sidebar that lets users drill down into these different types of cases, showing the relative amount of cases at each level.

On the technical side, one interesting detail is that this project is 100% HTML, CSS and JavaScript — there is no backend server. This simplifies not only the software architecture, but also removes the need to set up and maintain a new server. Since we needed to offer users the ability to download the visible data in CSV form, we had to generate the CSV contents in the browser and show the user a download prompt. Luckily, this can be done relatively easily in modern browsers.

Impact

The upcoming months will tell us if the project succeeded in its goal of reducing manual work for agency employees, both from generating reports and answering data requests. We’re also excited to see if third parties will use the newly accessible data to build new services or communicate the data in novel ways.

Taking into account the scope of the project, we are very happy with the end result. But to us, what is even more important was being able to take part in a project that makes data about an important topic — immigration — more accessible to the general public. So much of the ongoing debate on the subject is based on rumors, fears, and outright lies, not facts and figures. We hope to see changes by making data easier to get hold of and understand.

So many of the global issues that we currently face are complex problems with no clear solutions. At Lucify, we strongly believe in the power of the digital medium to help us better understand these problems — not only by visualising large amounts of data, but by offering ways to simulate, play around with, and intuitively understand complex systems.