Using Intercom for user research and diary studies

How do you get user feedback on one of the biggest technological shifts in a generation?

HMRC’s Making Tax Digital plan means that from at least 2020 most businesses, self-employed people and landlords, as well as individual taxpayers, will be required to update HMRC regularly throughout the year using accounting software.

This is a big shift from the current system and the reality of how this will work is still unclear. To explore the challenges ahead, we partnered up with HMRC on a research project to see how small business owners dealt with transitioning to digital accounting. We also wanted to probe about participants’ attitudes towards Making Tax Digital (MTD) and all the anxieties and possible blockers that could affect the MTD initiative.

Here’s how we used the power of communications tools such as Intercom to help to us engage with and monitor the experiences of our participants.

Our approach — questionnaires and diary studies

HMRC provided us with 47 participants who were willing to trial FreeAgent software and chat with us about their attitudes towards the MTD initiative. These participant previously filed their tax returns without an accountant a trial to manage their finances online using FreeAgent.

Questionnaires

Because of the generous number of participants we decided to work with questionnaires throughout the research period in order to collect quantitative and some qualitative data.

We used Typeform to design our questionnaires and sent out three of them during the beta period to all of the participants. These were as follows:

  • a pre-life survey to gain insight into needs, behaviours, and attitudes before participants’ first use of FreeAgent — completed by 32 participants
  • a mid-life survey to monitor their progress with FreeAgent — completed by 12 participants
  • a wrap-up survey to gain insight into what changed for participants during the beta process and what problems they might have encountered — completed by 8 participants

Soon after releasing the first survey we started collecting data. Unfortunately, however, we then encountered some trouble with participants’ engagement and saw a drop off rate in the number of people who filled in each consecutive survey.

Diary studies

We also wanted to observe in more depth how some of the participants went about adopting FreeAgent as a tool for their business admin and accounting. We settled on diary studies to allow the participants to record their experiences over time and in their context.

“A diary study is a research method used to collect qualitative data about user behaviors, activities, and experiences over time. In a diary study, data is self-reported by participants over an extended period of time. During the defined reporting period, study participants are asked to keep a diary and log specific information about activities being studied.” — Nielsen Norman Group

Out of 47 participants three of them were willing to take part in the study, which allowed us to collect qualitative data over time about these user’s behaviours, activities, and experiences in the context of the use of FreeAgent. The data was collected as follows:

  • We used Intercom (an in-app messaging system) to collect diary entries from participants
  • In addition we conducted two telephone interviews with them to gain further insight into their experience

We also conducted some informal chats with freelancers and accountants about how they felt about Making Tax Digital coming into action.

Using Intercom for diary study logs

At FreeAgent we’re already using Intercom to support and monitor our free trialists so we had a great platform to regularly learn about the participants’ progress. Every time the three selected participants logged in to FreeAgent we sent them a message asking them to answer three simple questions:

To set this up, we created a new campaign in Intercom. We manually included our participants in the campaign [the rules we used were “Users enter the campaign if” > “They match any of these rules:” > Email is xxx ]. At this point we also had to exclude the same users from the onboarding campaign we run for all our free trial users in order to prevent sending out conflicting communications.

We then set up the message to fire every time a user started a new web session in the FreeAgent app. At this time there is no option to tie the “send message” event to the “user login” event, which we originally intended, so we had to replicate the “send message” event to fire off with each new web session. The guys from Intercom support were very helpful in answering all our questions about it.

Before we set the campaign in action we also had a call with the participants of the study to conduct a preliminary interview and to brief them on the study requirements. We followed the tips in this Spotless article called ‘6 things to consider when setting up a UX diary study’ to help us communicate our expectations to the participants.

We showed them what they will see when they log in to the app, for example:

“You will see a popup box at the right of your screen (see illustration below), at the bottom of which you can type in your answers”

We also provided them with some examples of the kind of answers we were looking for:

Example 1

A1. I finished off a job and wanted to raise an invoice with the customer
A2. Send an invoice
A3. I had some problem setting up PayPal payments but send the invoice successfully. Easy enough to do

Example 2

A1. I always do my admin on Monday morning
A2. I wanted to log in some expenses
A3. I wasn’t really sure what categories I should have used bit got there at the 
 end somehow. It was a bit confusing

Once we started the campaign, in no time at all we were receiving messages from participants. The great advantage of the method we chose was that we had live access to the message logs, so we could work as we went, analysing the responses and asking new questions.

Analysis of the data

Questionnaire analysis

We found that some of the most useful insights came from the qualitative questions included in all three surveys. These came in the form of short and honest answers. Exporting those answers to a spreadsheet allowed us to quickly pick up any recurring patterns in what our participants were saying

Diary study analysis

We also did a similar analysis with the diary study entries and were able to see commonalities between the three participants.

We got some very interesting data from the diary study, not only in terms of their progress but also the context of use:

“Sorry, got distracted by the football and switched off! Will start again…”

We were also able to observe three stages in adoption of FreeAgent:

  • Learning curve
    Initially new users of FreeAgent are faced with a learning curve and this was no different for Making Tax Digital Beta participants.
  • Self help
    Although initially challenged with the learning curve, the participants of the diary study demonstrated the ability to problem solve and to find help whenever needed. We could see that during the study they were asking particular questions but were also able to find the answers independently. Participants commented positively on the ease of use and simplicity of the FreeAgent’s Knowledge Base videos and articles.
  • Increase in confidence
    With the continued use of FreeAgent virtually all of the participants showed signs of increased confidence in their ability to use FreeAgent.

Insights and conclusion

In addition to having a better understanding of how participants interacted with our software upon first use, we also found a range of anxieties and blockers that they have in regards to adopting the Making Tax Digital plan.

The research showed several interesting findings:

  • Submitting regular updates to HMRC could relieve the end of tax year anxiety and help to improve people’s accounting habits (Many participants believe it will help them to build a better accounting habits and relieve the stress of end of the year returns. However some had concerns around increased workload and stress around quarterly submissions)
  • People feel uncertainty with regards to Making Tax Digital as they feel there is a lack of information and clarity (the majority of participants expressed the opinion that the information provided by HMRC lacks clarity. Some key questions mentioned were how Making Tax Digital plan will affect running their business and what the requirements of Making Tax Digital for their business will be.
  • People feel submitting regular updates to HMRC will increase accounting costs (both for participants who use the services of a professional accountant and accountants alike, there is a fear that more regular submissions will significantly increase their costs)

Following the results of our research, our recommendation to HMRC was:

  • to provide more information and clear guidance on the requirements and impact MTD will have on their business
  • support them in understanding of how they can work better with an accountant and mitigate their costs
  • provide a resource to help small business built on their admin and accounting skills and habits and the use of online tools

At FreeAgent we are already taking steps to address some of the issues mentioned. We’ve got a host of resources about MTD available for small businesses and their accountants, including a free guide explains what MTD is, what its objectives are and how it will affect the way taxpayers report information to HMRC in the future.

To find out more about the findings from the research project, see our guide called ‘Making Tax Digital software pilot: FreeAgent’s recommendations for HMRC’.


Aleks Wruk is a UX researcher at FreeAgent, the UK online accounting software made specifically for freelancers, small business owners and their accountants.

Aleks Wruk, UX researcher

This post is part of a series of blog posts going behind the scenes at FreeAgent, digging into our design and development processes, sharing the lessons we’ve learned and openly discussing the challenges we face. See more details here.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.