CMU LearnLab Summer School 2017: Innovation, Understanding, Iteration

I’m driven by the impact of research discoveries. But there’s this frustrating gap between research discovery and practical implementation. I suspect this gap is because researchers and practitioners lack the infrastructure and opportunities to collaborate. And addressing this gap is critical for my current research, where I am developing a formative assessment tool to improve introductory computer science education. So, I went to Carnegie Mellon University to connect with hardworking people thinking about similar challenges and see how the LearnLab was bridging the gap between educational research and practice.
The LearnLab at CMU exists at the critical intersection between educational research and practice. It is a cross-disciplinary laboratory for advancing the practical science of how students learn. To me, it’s a place where experimentation and iteration are interwoven with instructional design. How students think and how technology and pedagogy impact them is at the core of LearnLab’s focus.
I decided to attend LearnLab’s week-long summer school to feed a recent interest and research endeavor in developing an intelligent tutoring system that supports students enrolled in introductory computer science courses. At this summer school, I interacted with professors, students, instructors, and support staff working to advance both their own courses as well as our understanding of how students learn. Half of our time was spent hearing from LearnLab faculty and students about topics including transfer and learning, cognitive task analysis, and educational data mining. The other half was spent in our specific tracks (I was in the intelligent tutoring systems track), working towards a poster and presentation at the end of the week.
This workshop reminded me why I love what I do: The community of educational researchers and instructors is full of genuine people who are driven by their passion to empower others. I discussed with Ken Koedinger (CMU) how to balance the burden and benefit of self-regulation tasks in pedagogy. I had a really engaging conversation with Kalyan Veeramachaneni (MIT) about data science education and how there is an overemphasis on feature engineering and model selection and an underemphasis on the “human” elements: developing prediction questions, identifying relevant data, interpreting results, and making decisions. Vincent Alevan (CMU) shared his perspectives on designing effective educational experiences by focusing first and foremost on what’s hard for students to think about, a mantra I will use to inform my practice moving forward.
Key Takeaways
My key takeaways from the LearnLab summer school:
- You can’t see learning happening! Students don’t know what they know. Much of this knowledge is tacit. Furthermore, instructors suffer from expert blind spot, where they have too much expertise to know about learner difficulties and how to address them. Cognitive task analysis helps decompose what we want to teach into components and understand “hidden skills” to better teach these concepts.
- Educational technology is about using data to drive iterative improvement. If we collect and analyze data that represents the desired learning outcomes, we can identify shortcomings in a course and improve upon them. Relying on hunches is not rigorous. Simply applying learning theories with blind trust is also ill-advised because often times the domain is different and the theory may or may not apply.
- General transfer learning is a fantasy. The notion that the mind is a muscle that is exercised is overly simplistic and wrong. Knowledge is largely domain specific, so transferring knowledge between situations is hard. We can achieve some intermediate transfer, perhaps through teaching general strategies or explicitly supporting self-reflection.
- Adaptivity can happen at different time scales and for different types of knowledge. We often have a very narrow perspective of what it means to be “adaptive” (no doubt influenced by Siri, Alexa, Cortana, Google Home, etc. adapting to our usage). Adaptivity can happen at different time scales (e.g. providing relevant feedback after an incorrect action, updating curriculum for the next course offering). It can also apply to different types of knowledge! (See the table below.)

My Ongoing Research: Supporting code tracing by embodying the computer
For my project, I spent the week working with Hieke Keuning (Open University of the Netherlands) to create example lessons for a tutor that helps people learn to read/trace code. Whereas debuggers and code visualizers had students passively step through code, we want students to “embody the computer” and take an active role when tracing code. The intelligent tutoring system we built provides scaffolding for program tracing by highlighting the lines of code that are being executed and having students update a table of variables as the code executes. In a later lesson (pictured below), students must select the next line of code being executed while still updating the table of variables for each method. This is part of ongoing work that Hieke and I are doing, so please reach out to us if you are interested! GitHub: NAP Tutor
Final Thoughts: Don’t just apply learning science. DO learning science!
The LearnLab summer school helped me understand what it looks like to bridge the gulf between research and practice. There are many barriers to overcome, such as reducing the overhead of creating digital learning tools and the infrastructure to collect data about them. But the benefits of this labor are so promising!
We can create more accessible and effective learning experiences if we use information to test assumptions and improve iteratively.

