Coding tutorials: the good, the bad, and the ugly

Ada Kim, iSchool Ph.D. student

Last year one of our new Ph.D. students at the iSchool, Ada Kim, came to me looking for two things: 1) some advice on how to learn to code, and 2) opportunities to dabble in research on how people learn to code.

At first, I thought these were two entirely different requests. But then I realized: why not use Ada’s experience learning to code to advance our understanding of the strengths and weaknesses of informal learning resources such as online coding tutorials? My lab had just completed a series of papers about Gidget, an educational game that teaches introductory programming concepts through debugging tasks (Lee & Ko 2011, Lee & Ko 2012, Lee et al. 2014, Charters et al. 2014, Lee & Ko 2015), but our work had mostly focused on engagement in coding tutorials, not learning.

Here’s the method Ada and I devised:

1. Ada would adapt foundational principles of learning into a set of design principles relevant to coding tutorials.
2. Ada would complete dozens of the most popular coding tutorials online, learning and evaluating the tutorials against the principles at the same time.
3. We interpret the quality of tutorials across the data set, offering recommendations about how to improve the design of the tutorials from a learning perspective.

The learning principles included the following:

• Content should account for learners’ prior knowledge.
• Organize declarative knowledge, providing a conceptual framework that helps learners integrate it.
• Provide deliberate practice, ensuring mastery of skills through immediate personalized feedback that iteratively improves skill.
• Encouraging self-regulation (reflecting on strategies, monitoring comprehension, etc.).

Ada invested more granular details of these four principles across 30 popular coding tutorials such as Codecademy, Kahn Academy, Python Tutor, TutorialsPoint, Coursera, Code.org’s CodeStudio, LightBot, and many others.

Here’s what she found: most online coding tutorials do not yet achieve many key principles in learning sciences. Future research and development especially needs to better emphasize personalized learning by offering precise, contextualized, immediate feedback and teacher both programming process skills, as well as self-regulation skills.

Two educational games that follow strong learning science principles.

Of all of the tutorials we evaluated, the educational games such as LightBot and Gidget best achieved learning science principles, but even those lacked the personalization necessary for effective learning. The worst tutorials were basically static reference guides like W3Schools and LearnPython, which had minimal personalization, minimal organization, and little to no feedback.

Want to learn more about our study? Here’s the SIGCSE 2017 paper we published describing our results:

Kim, A. and Ko, A.J. (2017). A Pedagogical Analysis of Online Coding Tutorials. ACM Technical Symposium on Computer Science Education (SIGCSE), 321–326.