How Jungroo uses Engagement Score as a key metric in our platform?

Cibe Hariharan, CTO of Jungroo Learning has come up with a new article about Jungroo's Engagement Score. This is a follow up to the previous article he wrote about Jungroo's approach to adaptive learning.

In this article, we will see how Jungroo's engine uses the Engagement Score as a crucial parameter in its algorithms for adaptive learning.

The flow of the current learning system is as follows.

1. The system starts interactions with the student by asking questions.

2. The questions are in an adaptive manner and changes based on each response from the student in an adaptive way.

3. If the system finds that there is a particular concept that the student does not know, then the remedial and learning contents for the concept are shown.

4. Also, If the system finds that the engagement level of the student with the system is low, then the best possible remediation/learning content would be shown

5. The system moves forward in an adaptive way using the knowledge states of the student.

6. Every single interaction of the student with the system and knowledge state of the students are being captured in real time.

A content is presented to the student in two cases:

Case 1 (Student doesn’t understand a particular concept)

When RL(Reinforcement Learning) algorithm predicts with confidence that student doesn’t understand a particular concept that is presented to the student.

More information on how our RL algorithm predicts with confidence about the student states can be found here:

https://medium.com/@sethuraman.ta/jungroos-approach-to-adaptive-learning-e6fa7293a693

Case 2 (Engagement of the student with the system is very less)

Let us denote engagement score at any point t as Et.

Et 𝜀 [0, 1]

When it’s closer to 0, the student is less engaged. When it’s closer to 1 student has a high engagement.

There are two types of contents for which the engagement score is computed.

1. Question Engagement (Et)

2. Content engagement (VEt )

Question Engagement (Et)

Let m 𝜀 [0, 1] denote the marks or points the student is being awarded for his answer to the question, 𝛾 be the time taken by the student to answer the question, 𝛼 be the careless probability associated with the question and 𝛽 be the guessing probability associated with the question.

U ( m, 𝛾, 𝛼, 𝛽) is the understandability function which denotes how much a student understands the concept looking at the parameters m,𝛾, 𝛼and 𝛽.

Now, let us consider some possible scenarios that can arise

1. Understandability of the concept is very high. A student can find questions less engaging as the question is very easy to answer.

2. Understandability of the concept is very low. A student can find that question less engaging as the student does not know the answers to the questions presented.

Hence it is important to consider time spent on a question as well.

Let PEq (T = 𝛾) denote the probability of student’s engagement with question q given the time spent on the question. This is computed irrespective of understandability.

Et = W1 * U (m, t, 𝛼, 𝛽) + W₂* PEq (T =𝛾 )

Question engagement is the weighted average of understandability U and engagement probability PEq based on the time taken by the student on question q.

Content engagement (VEt )

A Content can be a video/remedial content/activity/slide etc. For the sake of this blog, we will be using content to represent a video.

A Content is shown and a question is asked after the content at time t. Let’s denote the content as Vt, student’s progress on the video as Pt and question as Qt.

The Question Engagement Et has computed for the question Qt . Now, let us consider some scenarios,

1. The Student has skipped most part of the video but has understood the concept.

2. The student might have consumed most part of the video and still not understood the concept.

There can be many such scenarios. Hence it is important for us to understand the percentage of content consumed and the time spent on the remediation. So with only question engagement score Et, these cases can’t be handled. We compute Content engagement VEt as a weighted average of Question Engagement Et and percentage of content Consumed Pt.

VEt = W1 * Et + W2 * Pt

What is Weighted moving average (WMA)

Engagement score at any time t can be computed using the above methods. A single less engaging question or video doesn’t make the student drop out or lose engagement. In order to drop out, it takes a sequence of boring contents or less engaging interactions.

To handle this we take into the account of the sequence of interactions of the student with the system and use a moving average on scores of last k engagements of the student with the system. Moving average weighs all the engagement score equally.

Consider a scenario of last 5 (k = 5) engagement score sequence E = {0.9, 0.9, 0.5, 0.3, 0.2 }.

The simple moving average of these engagement score sequence would be 0.56 which might seem to be an average engagement but as you could see from the sequence that student’s engagement with the system is clearly dropping and is very less.

To tackle this Weighted moving average (WMA) is used because it weighs the recent engagement score more and older engagement scores less.

Also, there is a component of decay where the system assumes that the student can forget some concepts over a period of time and adjusts its adaptive algorithms accordingly.

Conclusion

The Engagement levels of the user with the system is a crucial parameter for us to adapt assessments and content to the student. Jungroo uses the time spent, percentage watched, type of response, the correctness of the questions for a student and the relative performances, interactions of the class amongst other parameters to calculate the engagement score of a student. This is a very crucial parameter to do effective personalisation and adaptivity keeping in focus the engagement and mastery of the student.

For more details about Jungroo and our work please visit our website. You can also reach out to Cibe and Sethu directly.

The previous article on Jungroo’s approach to adaptive learning is also worth looking into if you are interested about our approach to adaptivity in itself.