Skill Trees for adaptive learning
Building tools that allow teachers at Prodigy to iterate faster on student learning experiences
Authored by Raúl Cifuentes and Daniel Nguyen
Gizmo is one of the teams within the Education Platform here at Prodigy. In the past, our team has been responsible for providing tools that allow our Education Specialists to create educational content, though they still relied on developer support to make and release a number of curriculum changes.
With one of our newer tools, known as EdCAT, Prodigy’s Education Specialists are now able to create, manage and release new content completely autonomously.
Over the past few quarters, Gizmo’s composition has been altered and we are now working as a cross domain team: software developers and Education Specialists joined forces to deliver faster and smarter tools and content. Among the great results of this organizational change is the renewed Skill Tree tool. Here is a little bit of its story.
Curriculum-aligned educational content in a Game
An academic curriculum is a set of standards expected to be covered and mastered by a student. Those standards are categorized into strands (general areas of knowledge, such as Geometry, Number Sense, Data & Probability, etc).
At Prodigy, our mission is to help every student in the world to love learning. To this end, Prodigy currently offers Math and English games that help students learn by offering questions that, when answered correctly, help them to progress through the game.
Such questions must be relevant to the particular subject guided by an academic curriculum, an assessment test or a teacher plan and should be presented in a sensible sequence: each question is expected to be relevant and more challenging than the previous.
When the student’s answer is incorrect, the next question should be one that is slightly less challenging, and/or one that helps them revisit prerequisite concepts.
Questions and Skills
Here at Prodigy, we organize our educational content into what we call skills. These are sets of questions following a shared template that are aligned to curriculum standards. Each skill has a difficulty value associated with it and is connected to other skills by “pre- and/or post-requisite” skills.
An example of a skill could be: “Convert fractions to decimals”. A question under that skill could be: “What is the decimal number equivalent of 3/4 ?”
So, the questions, and more precisely, the skills, need to be in an order that helps our systems build and offer educationally sound and adaptive journeys to students.
Prerequisites and difficulty
A sequence of skills is established into chains that branch and merge, and the order of skills in those chains establishes prerequisite relationships between those skills. For example: “Understand a fraction” is a prerequisite for “Write equivalent fractions”.
These skill chains can be seen as a directed graph, for instance:
It is understandable that any skill set covering a particular strand will produce a similar graph. However, when we expand the scope and consider multiple strands, skills under different strands might not be requisites for each other. That scenario produces a group of disconnected graphs.
Prodigy’s games must cover the full curriculum and offer continuous paths of learning and assessment. That means that the ability to move from any skill to somewhere else is required, so disconnected graphs are not enough. A bridge between disconnected subgraphs is especially needed when students have not yet mastered certain concepts.
Consequently Prodigy’s curriculum establishes a difficulty level for each aligned skill. We use it to group skills in similar difficulty buckets, establishing a bridge between skills within two separate directed graphs.
At Prodigy, the combination of difficulty levels and requisite chain for a particular curriculum is known as its skill tree. During normal gameplay, the skill tree is used by our games to determine the questions that are presented to students.
Internal Tools & Skill Tree
In the past, our Education Specialists would curate which skills to include in the curriculum, categorize the skills by difficulty levels and establish their relationships via a Skill Tree tool.
The tool shows skills as boxes connected by lines and laid out such that the vertical swim-lanes represent the difficulty.
Additionally, it offers two tabbed sections:
- Filter Tree, to control the set of skills to show on the mesh diagram and
- Skill View, to establish difficulty and requisites for a particular skill
This tool, while operational, offered some challenges to our Education Specialists:
- It offers a small preview of a skill and these previews are not available for newer skills.
- Considering the amount of skills in a curriculum (in the hundreds), it is easy to end up with a dense entanglement of edge lines that are difficult to read.
- The filters often filter out skills that may have been important depending on how they were originally tagged and organized.
- The skill tree tool exists as a standalone tool, separated from the tool where the curriculum and skill content is managed, so Education Specialists need to switch between multiple tools to understand the curriculum structure and content while building the skill tree.
- The existing standalone skill tree tool has no in-built graph structure validation. This means we may accidentally introduce loops, break difficulty patterns, etc.
- When Education Specialists finish their work, they require developer help to release their changes.
- With existing tooling, a developer would discover these kinds of issues and raise it to the Education Specialists to resolve the issue, increasing the number of handoffs and making it more difficult to have confidence around the quality and correctness of the changes.
EdCAT
EdCAT is an application, created and maintained by Gizmo, where curriculum structure and content are managed. Using this tool, Education Specialists can create skills, and maintain and release curricula that become available for use by our education algorithms working in concert with the game.
We wanted to eliminate the friction caused by the use of a standalone Skill Tree tool, by bringing skill tree work capabilities to EdCAT, and as a result, offer our Education Specialists the ability to release their changes in the skill tree without developer intervention.
Exploring alternatives
Foreseeing that work, we started tangentially exploring some ways to present skill tree information visually. EdCAT is a web tool and although there are a lot of JavaScript based chart libraries out there, when it comes to visualizing graphs, our options are reduced.
Tools like Cytoscape, JIT VisToolkit, mxgraph, etc. offer a great start but showing a basic directed-graph with little to no customization was not going to meet our needs. At a high level we needed to:
- Avoid producing a graph with too many edges (hard to follow),
- Express the difficulty variable in the visualization
- Display multiple skill question previews simultaneously
Showing a simple graph according to the requisite-chain is of course not enough. Here is subset of Ontario skills shown as a graph. There’s many lines and the difficulty level of each is not been yet visually expressed.
To reduce the number of skills presented, some interactivity on the nodes seems like the way to go, so despite being quite old, JIT VisToolkit had some interesting features.
It should not be a problem to express the each skill difficulty using the colour or size channels.
In the past Gizmo built a React component that renders question previews. So that’s good news in regard to the third point in the list. However, the majority of those graph-oriented libraries draw on an HTML Canvas. Rendering SVG subtrees on top of the HTML Canvas shapes? No, thank you! We preferred to embed the SVG component in another SVG. At this point, we considered D3 as an alternative.
Here we see a D3 network diagram showing the prerequisite chain. Again, we should be able to easily adjust colour or sizes as functions of the difficulty.
We also reviewed another internal tool, a skill visualizer, checking if it would make sense to bring it into EdCAT. This one is using a colour channel for strands and the difficulty dimension is being nicely handled by the radial layout
Laying the foundation
Some of the issues with the SkillTree tool that we wanted to solve did not have to do with graph visualization, so we decided to incorporate into EdCAT the critical functionalities that allow us to remove the need of using a secondary tool.
Using simple form controls, we began by allowing our Education Specialists to set prerequisites and difficulty values for each skill. We made some improvements over the previous tools: aggregate counting on each filter option and information about where the skill is being used.
On top of that, the Skill Tree page in EdCAT enforces business rules that prevent unwanted dependency cycles, prerequisites with higher difficulty levels, etc. For legacy skill trees, those created with the previous tool, there is a button to check their correctness and a report of any broken validation. Healthy skill trees are celebrated with confetti animations.
Having solved the critical aspects, we got back to the visualization problem. We developers wanted to give Education Specialists a broader view of prerequisites, where they can see the chain of skills for a particular grade or strand altogether.
We started a spike task for building a prototype of a visualization for skill trees. As part of it, we wrote some D3 code that produces some adjacency matrices and arc diagrams aimed to offer clues about the skill tree as a whole. Although writing D3 code is always fun, it can become time-consuming. The idea of reusing some of the existing force directed network diagrams, (from skill visualizer tool) was gaining force.
Around that time, we held a meeting where our Education Specialists showed us developers how they were planning out the skill tree for a brand new curriculum.
In that session, our Education Specialists Jenn and Daniel were using Miro (the web visual whiteboard tool) to lay out the prerequisite chains and difficulty levels for the skills in the curriculum.
They used it to manually create something visually similar to the layout of skills within the legacy Skill Tree tool where skills were visualized as “cards” containing a screenshot preview of the skill, captioned with the skill ID and its difficulty.
From here, they marked sections of the board to represent grades, and within those grades were sections to organize the skills into individual expectations/standards. The result was a massive virtual whiteboard of the entire curriculum containing nearly 1000 (painstakingly, individually created) skill cards.
This allowed them to copy and paste cards, and freely move them around until they were happy with their arrangement and connections. Being able to simultaneously see image previews of an entire expectation’s worth of skills allowed Daniel and Jenn to identify redundancies or inconsistencies within skill chains and add or remove skills as appropriate. Rather than the current flow of creating single skills and aligning them to a curriculum one-by-one, they are able to align curricula holistically in this way — think of it like scoring music for a film rather than burning a mix CD.
After Miro has served its purpose as a planning tool, our Education Specialists then do the work of inputting this data — adding/removing skills, assigning difficulty values and setting requisites accordingly in EdCAT.
Wow! When the meeting ended it was clear to us developers that more than a visualizer was needed. Rather than a visual version of a static dataset (skill-tree), they needed some kind of editor that helps them to produce the skill tree itself.
With this in mind, we abandoned D3 and found ReactFlow. It’s an open-source library that allows us to layout graphs with custom nodes.
Armed with this tool, we got down to work and were able to produce prototypal graphs where:
- Each node represents a skill and includes a question preview (re-using the existing React component),
- The edges represent the requisite chain
- The horizontal position for every node is dictated by the difficulty level.
- Integration with the EdCAT skill tree page was possible (it reacts to filters and/or skill selection)
- Excellent frontend performance is accomplished by lazily loading the question previews.
- In addition to the zoom controls and mini map provided natively by the library, we included a full-screen mode so that Daniel and Jenn could comfortably view the skill tree on their huge monitors.
Despite not allowing users to move the cards yet, with the integration to the form controls, this prototype is at a stage between a visualizer and an editor. It has received great reception among EdCAT users as it solved the issues initially identified.
So EdCAT and its new skill-tree capabilities allowed our content creators to deliver the new Minnesota and English Common Core curricula faster and confidently. Although there’s space for improvements, the close collaborative work between Education Specialists and developers resulted in this solution and we all feel rewarded. 🥳
Future work
There are two areas we want to explore in the future:
- To allow users to move the skill cards around and assign them difficulty values based on their position. We did not include this functionality yet as there’s some challenges related with our current domain data model, where difficulty is currently scoped to curriculum level. We need first to move from a single absolute difficulty for a skill towards the possibility of having multiple relative difficulties (once per context: strand, grade, etc).
- It would also be interesting to offer Education Specialists the ability to easily scroll across an overview screen that shows multiple sub-skill-trees at once, once per grade and strand, similar to how they had their Miro board set up.
Conclusion
With continuous enhancements to the EdCAT tool, the Gizmo team has provided Education Specialists the autonomy to build education content without dependency on developers.
Armed with those tools, they released new curricula such as Minnesota Academic Standards and English Common Core. With a scalable platform, Prodigy will continue offering great educational content that help students across the world to engage with curriculum-aligned educational content, and love learning!
