Getting to grips with data (1)
At B&NES Carers’ Centre we have so much data. In so many places. Charitylog; MS Forms; Excel; Teams — it’s definitely not that we lack data! But we’re not using it effectively. There’s almost too much of it, we can’t see the wood for the trees. What’s relevant? What should we be using to help us make decisions and ensure we’re delivering the best outcomes for carers?
This challenge of becoming more data informed is one we are starting to solve, through our National Lottery Digital Fund transformation project. Having created a ‘Knowledge Board to highlight the assumptions we were making around data, and undertaken a series of interviews to gain insights on staff perceptions on data, we realised we need a new way of working with information, particularly to evaluate and develop our services. This post outlines our first steps towards this!
Delivering a data pilot
One key insight we’ve gained as we’re transitioning to ‘think digital’, it’s that you don’t run before you can walk. Our traditional model of working would have been to identify the problem, create a solution and then move to full roll it out in the belief it would work. Quite often, things rolled out in this way didn’t work as we’d made assumptions and didn’t fully consider the consequences.
So this time, we took a new approach — a pilot. We worked with just our Connect Team, to better understand the data they need to make decisions & improve their ability to capture and use this data moving forwards.
The ‘data pilot’ ran over a 3 month period to understand the problem, consider solutions and actions we could test and reflect on what did and didn’t work in order make changes. This learning could then be rolled out to the wider team, and shared with you!
Categorising data and collecting ‘just enough’
One of our first challenges was to identify what we mean when we say ‘data’. What does that refer to? It can mean almost anything we learn — figures, graphs, words, pictures…. anything we store to help us understand situations or make decisions.
In the context of using the best information to help us develop and deliver services for carers, we found categorising data using this NPC model helped clarify our thoughts and bring order to the chaos. It considers 5 types of data: user; engagement; feedback; outcomes and impact, in order to make insightful decisions.
Looking at data in this way was not something we had done before. With our Connect programme of cafe’s we’d monitored how many people came to a cafe activity. And if they said they liked it. But we had no idea who those users were or what their characteristics were, except anecdotally. Where did they live? How old were they? Was the same 15 people each time, or were they always different? Did we really understand if they were travelling towards our intended outcome of feeling more connected?
It was disheartening to realise we hadn’t been using data, much of which we already had, to make decisions. But recognising this has helped us to plan a new way of working which is more effective.
It wasn’t all plain sailing though! Once we’d started to categorise data, we excitedly identified ALL the user; engagement; feedback & outcomes data we could collect which would improve our knowledge and decision making. We created a long list. A really long list. We went away for a few weeks to look at how much of this we already had and how we could access/collect what we didn’t already know. But then we stopped and reflected.
We realised it was too much. We’d already identified we had too much data and weren’t using what we had, so why were we trying to collect more?! — Sarah Dixon, Wellbeing Manager
We changed tack, and decided to start small — much like doing a pilot. What did we really need. Right now. What will help us make key decisions? Being specific, and really challenging what we needed and why made collecting and using data feel so much more manageable and meaningful. And, dare I say it, more exciting?! It gives a real purpose to data collection to understand how it will be used. It makes it far more powerful for staff to understand why they are asking certain questions as there is a meaning to it. We settled on just the key pieces of data that would help shape the service. It doesn’t mean we’ll never collect other data, we’ll review regularly to understand what might need to be changed (we very much expect things will).
Deciding which data was key data
Our cafe project is for carers of all ages and is designed to connect carers with one another and their community. We therefore chose only the data we felt we needed to test this: the age of attendees; their location in relation to the cafe (was it local); how many times they attended; if it was useful or valuable to them & if it reduced isolation. Gathering this data gave us insights we were not expecting!
We had more data than we thought
Interestingly, when we first sat in a room we felt as if we we didn’t have the information we needed to make good decisions. We thought we’d need to do a lot of fresh data collection to access what we needed. Yet that information was available to us in our database (we just hadn’t known it!). I guess that shows we really couldn’t see the wood for the trees.
The data confirmed anecdotal evidence, but also showed us things we didn’t know
A lot of the data backed up anecdotal evidence we had, but there was information which surprised us and will be used moving forward to re-design services. For example, although we knew cafe’s were predominantly used by older people, the data showed us that 20–39 yr olds, making up just 6% of the total. We’d never been presented with data which showed how clearly this particular service wasn’t working for this age group.
Additionally, a lot of carers visit cafes just once or twice. We’d never picked this up before, and it made us question the carer journey in to a cafe. Why hadn’t they returned? Was there something we could do about this? As with all things, COVID means our face to face cafe’s are on hold, but this is all data we can use as they restart in 2021.
We needed to distinguish between feedback and outcomes
Finally, we realised we really hadn’t clocked the difference between feedback and outcomes data. There is a clear link between the two (it’s relatively unlikely that you will dislike an activity and still get a positive outcome from it). But they are very distinct. We knew without a shadow of a doubt people enjoy cafe’s. But we weren’t as certain that they felt more connected. Sometimes we asked this, but sometimes we assumed it based on their feedback.
In fact, we realised collecting outcomes data was not as simple as one question in a feedback form. Our cafe’s are designed to keep carers connected. But in order to achieve that people first need to feel they are getting peer support, that they are making friends and meeting like minded people, and have confidence to join in. Simply asking everyone if they are less lonely wouldn’t help us understand of the journey they were on. Working with Think NPC we realised we needed to develop a theory of change for our cafe programme, and improve the questions we were asking based on this. Ideally we needed different questions, asked at different points of a carers interaction with us, to show what had changed through attending a cafe.
So our pilot showed us we could use data more effectively and revealed helpful insights for our cafe programme which we can use moving forward, and a great framework for deciding what data was needed. But it challenged and revealed much more to us than this. In the next post we’ll talk about the culture change we’re starting to see, and where we hope this journey will take us next!
