From demo to public pilot — Documenting Starting up Smoothly working practices
Starting up Smoothly is the name of the collaboration between the Finnish Immigration Service (Migri), the Finnish Tax Administration (Vero) and the Finnish Patent and Registration Office (PRH). The goal of this collaboration was to pilot a chatbot network, where three independent chatbots are interconnected through an intelligent layer.
Inland has had a vital role in shaping how this project is run, since we are the coordinators between the three organisations. The project started as a collaboration between Vero and Migri in late March 2018. You can read more about the first phase of the project in this blog post: Starting up smoothly — Connecting government agency information through chatbots
In this blogpost I want to document the second phase of the Starting up smoothly, which spanned from June 2018 — after the public demo — to December 2018 — when we started the public pilot.
Steps to develop Starting up Smoothly in this (2nd) project phase
In summer 2018, PRH joined the project to create their chatbot as part of the network. In order to structure this blog a bit better I group our actions and working practices into five steps:
Step I: Onboarding the third organisation
Step II: Developing three individual chatbots simultaneously
Step III: Defining transfers between the chatbots
Step IV: Gathering user feedback and iterating content
Step V: Piloting the chatbot network
Each step can be understood as a period of time, but also as a number of actions and working practices which were used to reach a certain sub goal. Let’s go through each of the steps one by one:
Step I: Onboarding the third organisation
When?
From June 2018 to October 2018
Why?
The goal was to bring the team from Patent- and Registration Office up to speed in developing their chatbot PatRek.
How and what?
Actually, I will write another blog post documenting how we onboarded the third organisation to Starting up Smoothly. I already have the draft for this post, but it would make this blog post even longer than it already is, and it’s a slightly different topic, so please be a little bit patient with me.
Step II: Developing three individual chatbots simultaneously
When?
This is a bit fuzzy, but I will just say it was during September — October 2018.
Why?
We wanted each individual chatbot to have enough content to bring value to the users independently, before we connected them to the network.
How and what?
In the beginning of this phase we discussed the target audience for Starting up Smoothly again: The service is for foreign entrepreneurs who are still in their home country but want to start a business in Finland. Based on additional user research, done during the onboarding of PRH, we defined these three user personas:
The aim with the user personas was to create fictional representations of the service’s real end users, which had potential touch points with all three organisations.
Each organisation then individually defined the content they wanted to create before going live. We synchronised these plans between all organisations, so that the content of the three organisations makes a useful unity for the users.
Next, each organisation started to build their own content. You can say that we worked on this in parallel and independently from each other. Nevertheless, from this point onwards we worked every Tuesday in the same room. This gave us the chance to see the progress of the other organisations, but also to synch our content early on.
After structuring the content on high-level each of us went into more detail for the content areas that were in scope. We built conversation flows: How does the user go through an entire conversation if they have asked a high-level question, that cannot be answered directly? Instead the bot answers giving different selection options.
Conversation flows help us to see how the user experiences the service, but they also allow us to understand the structure of our intents. You can understand intents as a questions[plural]-answer-pairs. There are many ways to ask a question, that can lead to the same answer. That is what we call an intent. Then each intent is put into a hierarchy of other intents, which is structured from more general questions on the high level to more detailed ones lower down in the hierarchy.
Structuring intents is often done on paper, post-its, whiteboards or by other non-digital means. This helps to focus on understanding the structure and user experience first. We see this particularly important for people, new to building chatbot content. Then we start using the software to build the required intents, enter the chatbot reply texts, as well as test and training data. Test and training data are needed for the advanced AI to predict, learn and test, which user questions should lead to which chatbot replies.
It was part of the working process to constantly test newly built content among the team of all three organisations. Obvious errors were found rather quickly when another person looked at the content and gave feedback.
Step III: Defining transfers between the chatbots
When?
This was an ongoing process starting from September 2018 to the start of the pilot in late November 2018.
Why?
We needed to understand how the chatbots interact with each other. The whole idea of “Starting up Smoothly” was to connect three independent chatbots, so this is where the juice comes into the project.
How and what?
Two types of transfers between the organisations felt obvious to us from the beginning:
Reactive transfers refer to transfers where users ask about content that a different chatbot knows about
Example: A user starts with questions about starting a private trader from the Patent and Registration Office. She then asks for information about her personal taxation, which is a content area covered by Vero’s chatbot. PatRek offers a transfer to Vero based on the user’s input.
Manual transfers refer to handovers where users explicitly ask to talk to a certain chatbot or organisation.
Example: A user has transferred from Migri to Vero. She later remembers that she has another question about immigration, so she asks, “Can I switch to Migri?”.
The third type of transfer was less obvious, but based on our user personas we understood that proactive transfers are where we can really add value to the user experience:
Proactive transfers refer to handovers where the users are proactively given more information than they may expect.
Example: A non-EU foreigner asks detailed information about a residence permit to work in Finland. The chatbot then reminds the user that she will probably need a tax card to work in Finland and suggests transferring to the Tax Administration for more information.
Combining three types of transfers between the chatbots supports users to switch smoothly from one bot to the other, from one domain of knowledge to the next — and back. Being able to switch between the three organisations’ chatbots in three different ways facilitates effortless flow of the user experience.
From technical point of view, manual transfers where introduced first, followed by reactive transfers. Proactive transfers where the last types of handovers to be implemented. They were based on the user needs the three organisations identified together.
Step IV: Gathering user feedback and iterating
When?
From September 2018 to December 2018
Why?
User feedback before the pilot helped to understand how immigrants perceive and use the three chatbots together. In effect it improves content as well as technical capabilities.
How and what?
During the timeframe we ran three rounds of user tests: The first round concentrated only on the PRH chatbot (and could be considered part of onboarding), while the other two rounds tested each organisations’ content as well as the transfers between them. Our best practice is to test with real users of our service as soon as possible but wait until there is enough content for the test user to understand the service concept. One main challenge has been to find the right users to test with. This required pre-planning, which we didn’t always have in place, and some effort put into user recruiting (because as government organisations you cannot pay a recruitment service).
Based on user testing we iterated content: We adjusted wordings to be easier to understand for our end users, we created more content that we had not anticipated but that was important to our users, and we adjusted the algorithms based on which chatbots transferred from one to another.
Step V: Piloting the chatbot network
When?
The pilot is ongoing since 29.11.2018 and we promise to continue it at least until 30.06.2019.
Why?
Piloting the chatbot network before putting it into permanent production is one of the essential parts of our user-centred process. We understand user feedback as one of the crucial success factors for our services. Only by piloting with a limited scope we gain feedback from a bigger group of users before investing a bigger amount of resources. We see the usefulness of the service and understand how much resources it takes to maintain the solution. Piloting can be understood as an extended user testing with a wider audience.
How and what?
We piloted in two phases:
Internal pilot refers to a trial round inside the organisations. Each of the three organisations involved in Starting up Smoothly did the internal pilot in their own way: In Migri and PRH a group of experts tried out the service before going live, while at Vero the whole organisation was able to test the chatbot network for a few days. In our experience the internal pilots helped to get understanding and buy in from the organisations and gave the other colleagues in the organisation a chance to try out the new service before it went live for customers.
Public pilot refers to the publicly available service trial. This is what we mostly refer to when we talk about the pilot of Starting up Smoothly. Now that the public pilot is nearing its end, we will evaluate the learnings and gather more best practices. The management group of the project would like to extend the pilot, which is a good indicator that the experiment was successful and that there is value in the service. Plans to continue the pilot until the end of 2019 are currently being made. We will definitely write about the evaluation of this experiment.
Summing up
I shared these five steps in the second phase of Starting up Smoothly mostly as project documentation. I also hope that reading this blog post others might be inspired by what is possible (in government!) and maybe even learn something about the user-centred co-design process we followed in our work.
Looking back after just over 1 year, this project was super-fast, especially given that it happened within a government context and involved three independent government organisations that work under three different ministries. It took 9 months from only an idea to a public pilot of three organisations. The speed of development was possible because the people involved where dedicated to experiment, to work collaboratively and share their knowledge.
If there is one thing that all of the team members can agree on, I am sure it is this one:
Making things happen was more important than making it perfect. Even though we tried hard to make it as perfect as possible.
author: Suse Miessner
edited by Mariana Salgado
