Piloting Gobee — what we learned from the first pilot of the gamified learning measurement tool

Written in collaboration with War Child Holland, War Child Jordan and NYU Global Ties

Since the beginning of the COVID pandemic we have seen a significant increase in the number of online tools and digital learning programmes being rolled out globally. With this increase comes a growing need to find adaptive ways of measuring the effectiveness of these interventions for children’s learning.

In response to this need, in 2020, War Child Holland, NYU Global TIES and the HEA came together, with the help of Porticus, to develop, test and pilot a gamified learning measurement tool that has the potential to measure both academic and social and emotional learning (SEL) skills. Through a process of collaborative design — which we wrote about in a previous blog — the team created Gobee: an interactive, fun, digital solution, providing real-time assessment data to teachers and programme implementers, which is accessible to learners wherever they are.

About Gobee

Gobee assesses the following aspects of children’s educational development:

• Two academic subjects: maths and reading.
• Two social-emotional learning (SEL) scales: self-efficacy (confidence in one’s ability to complete specific academic tasks) and growth mindset (the belief that one’s abilities can grow with effort). Research has shown that children who believe they can do well are more likely to commit to schoolwork.

The tool also includes:

A dashboard: teachers can access a simple dashboard that shows each child’s results and compares their progress both over time and to other children. The tool therefore provides a real-time formative assessment that allows teachers to learn from children’s errors and modify their instruction for the entire class or for individual children.

An assessment builder: Teachers also have access to an assessment builder that requires little technical knowledge and allows them to create their own assessments based on student needs and curriculum expectations.

Our approach to developing Gobee

From the outset, the Gobee team has taken an iterative, phased approach to developing the tool. This includes a three-stage testing process, consisting of Alpha, Beta and Pilot testing, undertaken in partnership with the War Child team in Jordan. Taking this multi-stage approach has allowed us to focus on refining the functionality and core components of the tool, as well as understanding the needs of users. We wrote about the process and our findings from the Alpha and Beta testing here. By the end of 2021, informed by significant learnings from the Alpha and Beta testing phases, Gobee’s first prototype was piloted. This prototype offered a taster of the academic content planned for the tool and the testing focused on assessing the potential of the tool with its target users — children and teachers.

How did the pilot phase work?

The pilot testing took place in Jordan in December 2021 and January 2022, with children aged 6–12 at home, in schools, in community based organisations and in refugee camps. It was important to the Gobee team that the most vulnerable learners were included in the pilot phase, so that the development of the tool captured their specific needs. This included testing the tool in refugee camp settings as well as the second Education Directorate in Zarqa, which is home to some particularly vulnerable communities. Whilst challenging, this approach gave us the opportunity to see how this tool can fit within an Education in Emergencies (EiE) setting, as well as how it works for the diverse sample we used — whether refugee camps, urban camps, formal education, or MoE schools.

In total 244 children, six education professionals (teachers, headmasters and supervisors) and 116 caregivers participated. All children were assigned to play Gobee, in either a classroom setting (136 children) or from home (108 children). Half the students were assigned to the maths assessment and the other half to the reading assessment. All students were assigned the SEL assessment. A subset of students in the classroom setting was assigned to complete a paper and pencil version of the Early Grade Reading Assessment (EGRA; 44 children) or Early Grade Maths Assessment (EGMA; 50 children).

For those students playing Gobee at home, the parents received instructions via phone and an explainer video about the purposes of Gobee and the pilot testing. A focus group was also conducted with a group of six teachers and school principals asking their opinion on the student tool, the teachers’ approaches to assessments in the classroom, openness to using the assessment builder and the dashboard and their opinions on both.

The testing was carried out during a tough Covid wave in Jordan, which presented a number of additional challenges, including having to change schools, use smaller data-collection groups and add more days to undertake the work. However, these challenges also brought opportunities to work with the community to find solutions to working under these conditions.

What did we learn?

During the pilot phase we found that digital literacy was a challenge for the most vulnerable communities testing the tool. This challenge showed up in a number of ways that are important points of reflection:

1. Hardware: how well a device is received depends on its quality. We used devices available to War Child but also ones that were just available in children’s homes. As a result, we had a diverse sample of devices, which in fact helped us to gain wider insights on what issues to expect when implementing the tool.
2. Software literacy: how familiar participants are with the software also impacts how well it works with the tool. Children who tested the tool were familiar with this type of software through previous exposure but that might not always be the case. This emphasized the need to make the software as child-friendly as possible and ask children if it’s clearly presented.
3. Connectivity: there is relatively good internet connectivity in Jordan, however there are areas with limited connection. We wanted to see how the tool worked in this situation (for example in Zaatari camp where internet speed is limited). Connectivity issues proved to be an even greater challenge for us, when the piloting had to be undertaken entirely remotely during Covid lockdowns.

A further interesting observation from the pilot phase was that there appears to be an increasing sense of technology fatigue, especially after Covid. Before Covid, War Child in Jordan reported experiencing resistance to technology in education. Now, the reverse seems to be true and communities have reported some overwhelm around EdTech. This is why Gobee has prioritised co-creation and community involvement in the tool’s development at every stage, including through fostering local ownership, to ensure it is not ‘just another tool’ teachers and students have to use.

Key insights

Overall, the results of the pilot testing, particularly in terms of the look and feel of the tool, how motivating it is for children to use and the benefits to parents and teachers, were extremely positive. We gained a number of valuable insights, which have helped to shape the tool and how we move forward on our journey to scale:

• Gobee has similar efficacy in measuring academic and social and emotional learning (SEL) skills as traditional assessments, whilst offering additional opportunities for assessment: the scores from Gobee academic tasks (literacy and maths) were closely aligned to other assessments typically used in this context — EGRA and EGMA. As well as confirming that the tool is fit for purpose, this finding also presents an unintended opportunity for Gobee to be used for assessment purposes beyond formative assessment, although additional validation is necessary. We also found that the tool was effective in testing skills which are not entirely possible to assess via traditional paper and pen assessments — for example we were able to assess precise number sense by introducing a number line activity. These opportunities are factors that set Gobee apart from other assessments and validate our choice to take an interactive, game-based approach to this tool.
• Gobee’s SEL assessment was positively received by students as well as by teachers who previously had limited tools to quantify and address their students’ SEL: The children’s feedback on the SEL questions was very positive, with participants clearly indicating that they understood the questions and process of selecting their answer. Teachers also confirmed that the SEL assessment helped them to understand which students may need extra attention by seeing their students’ psychological status, particularly in cases where students could not express what they feel in words. It was exciting for us to see the teachers’ positive response to having the SEL assessment within Gobee, especially in terms of strengthening the connection between social-emotional learning and academic learning.

‘This tool brings social-emotional learning to teachers’ and students’ awareness and makes them realise that they need to take this into consideration. And research shows that social and emotional learning is in some ways related to improving learning’ — Dr Ziad Twissi, Education Technical Adviser, War Child Jordan

• SEL scores on growth mindset and self efficacy showed a positive relation with academic performance: During our testing we found that those children who scored highly in growth mindset and self efficacy, also scored highly in their maths and literacy tests. This is exciting supporting evidence of the correlation between social emotional learning and academic performance that is extremely helpful for advancing the case for the integration of holistic learning in national education systems.

I really like and want us to keep developing the social and emotional learning aspect. I think linking children’s mental health and social and emotional state to their learning process will be insightful for the children, caregivers, and teachers. — Hussein Al Amoudi, Can’t Wait to Learn Project Manager, War Child Jordan

• Gobee positively impacts children’s feelings towards being assessed: A really interesting dimension of Gobee’s appeal to children during the pilot testing, which was highlighted by Educational Supervisor, Dr Esmat Alrosan, was the impact a game based tool has upon the children’s feelings towards being assessed:

‘Assessment’ is different from ‘test’. My goal is to be fair with the child — the test is always scary, whether for the old or the young. The beauty of this tool is that it evaluates the child through electronic games that he likes. — Educational Supervisor, Dr Esmat Alrosan

The children themselves also reported that they ‘felt happy and relaxed’ using the Gobee tool. Gobee’s ability to alleviate some of the stress that children can feel when being assessed, through introducing elements of play and fun, is an important benefit of the tool that further supports the choice to explore a gamified approach.

• Teachers were excited about the potential of the dashboard and assessment builder’s features: In addition to the positive feedback from teachers on the SEL assessment, the teachers interviewed showed significant interest in the possibility of building their own assessments through the assessment builder component of Gobee. Work on the user friendliness of the dashboard and assessment builder were highlighted as priorities for the next phase of development, in addition to specific training on these components for teachers. These activities are planned for Phase 2 of Gobee’s development.

What do we need to improve?

In addition to the positive feedback and results of the pilot, we also learned some important things about what needs to be worked on within the tool, which will be the focus of the next phase of development. For example, we found that digital literacy and type of device had an impact on performance. Children who possessed a device in the home were more efficient in using the Gobee tool during the test than those who didn’t. Overall we also saw differences in performance for number sense and addition, based on what device was used — students using phones performed better than students using tablets. While there were various factors that could have influenced these results — including the age, functionality and software capacity of the device used or the testing condition — the fact that the device showed an effect upon maths performance illustrates to us that there is still more user-experience work for us to do to ensure that maths ability is assessed in equivalent ways across devices.

We also saw differences in performance between home-based and face-to-face conditions during testing, again for number sense and addition. Students using Gobee from home performed better than students who used Gobee in a face-to-face setting. Students in the home setting could be performing better due to having more time, receiving additional adult support, not having as many environmental distractions, device type or a combination of these and other factors. This will be investigated further in the next phase of development

What is next?

HEA’s engagement in the first phase of the Gobee project has provided us with a wealth of insights, as well as lessons learned, on the real-world process of prototyping, testing and piloting an ambitious innovation that addresses the complex needs of teachers and learners in humanitarian contexts. The collaborative approach taken by the Gobee partners, leveraging the expertise of multiple organisations, has also been an exciting process to be a part of and watch evolve.

We are excited to see Gobee entering a second phase of development in 2022–23, led by War Child and NYU-Ties, and are particularly interested to see the team plan for and address components of sustainability in the project design, for future scale.

The next steps within this new phase for the Gobee team will be to:

1) Integrate the full grade 1–3 maths and reading curriculum into the tool, together with the SEL surveys, and improve the user-experience on the teachers dashboard and assessment builder (visual design, ease of navigation, tutorials). This will include running three further testing phases — Alpha, Beta and Pilot;

2) Draft and test the first Gobee training package for teachers;

3) Conduct classroom observations and focus group discussions with teachers to understand current assessment practices and identify opportunities, identifying the gaps Gobee can fill in their classrooms;

4) Identify new partners to join the Gobee consortium, as we refine and scale this exciting new tool. This includes partners who are able to test Gobee with teachers and children in different languages and settings, as well as partners who can assist in developing and testing a MEL version of the dashboard for baseline and endline assessment of programme interventions. The MEL dashboard will specifically address needs for programme evaluation and quality assurance purposes.

If you’re interested to know more and test Gobee in your EiE programme in 2023, please contact Kahina Boulegroun, Kahina.Boulegroun@warchild.nl, at War Child Holland.