I developed the Code Studio in partnership with the Terminal.com engineering team. This product helps beginner programmers write code in the browser to reduce frustration in the early experience for Udacity Nanodegree programs.
Who are our students?
We identified our target students before diving into product development. Based on user research about the Introduction to Programming students, it was safe to assume:
- Students were beginners interested in learning how to code.
- They have little to no formal Computer Science education, but may have been exposed to massive open online courses (MOOCS).
- While a few students were hobbyists, most students had the goal of transitioning into a programming related role or grow in their current position.
Why create the Code Studio?
One of the major setbacks for beginner programmers is the need to set up their environment and download the proper tools. This has often been a roadblock for projects submissions since students were expected to work locally and manually upload them to our reviews page. This experience can be frustrating for beginners and has contributed to the regrettable churn at the onset of our program.
What is the Code Studio?
The Udacity Code Studio is an in-browser integrated development environment (IDE) for Udacity projects. It provides students with a series of instructions adjacent to a text editor and students can seamlessly submit their project to the Udacity reviews team. Students also have access to resources like live chatting with a mentor.
My team spent six months developing this product and testing its effectiveness on student retention in the beginning experience.
- Create Project Flow
On the left, students see their lesson panel which contains project instructions. Students navigate through multiple project lessons sequentially. The text editor is pre-populated with relevant programming files on top right, and an interactive Python shell is at the bottom right. Students can code in the text-editor and their progress is automatically saved.
2. Submit Project User Flow
Once a student has completed all of the project instructions, they click on the “Submit project” button to submit their project to the Udacity reviews team. A pop-up modal prompts students to add additional details, confirm the work is their own, and submit.
3. Get Help User Flow
The “Get Help” button allows students to chat with a mentor. If a mentor is available, students fill out a form to provide context on the problem they are facing. Then they are directed to a chat window.
We conducted 3 rounds of user research to evaluate the effectiveness of the new immersive coding experience on retention.
We gathered feedback from usability testing and found a few pain points in the current implementation.
The Submit Project button was not intuitive:
“I’d expect the Submit project button to be in the bottom right, like every other flow I’ve experienced in life. It’s really weird that it’s a simple link in the top left.”
A few students completely missed submitting the project and left the Code Studio without submitting their work because the button looked like plain text and was in the top right of the screen.
In addition, only 4 out of 10 users were eager to use the “Get Help” button. When asking why users did not access the “Get Help” feature, they responded:
“I wanted to try and figure it out on my own…I wanted to try it myself and exhaust my own options before asking for help to prove I could do it on I own.”
The call to action to “Get Help” to some users meant admitting defeat. We decided to go back to our original designs in light of the research.
Based on the usability testing results, there were some high stakes problems to solve. Students were not utilizing a few of the key features that would result in a poor user experience. For example, if students completed their work without knowing they needed to “Submit” their project, their learning experience would be negatively affected. We decided to make the “Submit Project” UI element more visible to students through relocating the button. We also used a brighter color on the button to highlight the importance of this call to action.
Finally, we changed the “Get Help” button to “Resources” since “get help” did not resonate with many independent and self-sufficient learners. The “resources” button was more neutral and encouraged more students to explore this feature.
After design iterations, we conducted an A/B experiment with beta testers. We randomly split testers into two groups and both groups received the same lesson content. However, the experiment group had access to the Code Studio for completing projects and the control group was given instructions for downloading tools to work on projects locally.
Our hypothesis was to see an increase in project submission by 33% between the control group (traditional project submissions) to experimental group (Code Studio projects).
What were the results?
After 24 hours of enrollment, there were 36 Code Studio projects submitted in the experimental group and only 4 projects submitted through in the control group.
After running the test for two weeks, our final results showed 45.3% increase of passing project submissions and a 25% total increase of all project submissions in the experimental group compared to the control group.
More importantly, our students were really excited about the Code Studio:
Students felt successful completing projects without the friction of downloading tools.
“It felt good to complete this first task. I actually feel like this is doable. Thank you!” -beta tester in experiment group
There was enough evidence from our research to show that the Code Studio was a step in the right direction. Looking back, I would have liked to spend more energy on refining the problem we were trying to solve. While we were addressing our numbers related to retention, there was an underlying problem:
How do we give beginner programmers confidence within their earliest experiences at Udacity so they feel excited about continuing their learning journey with us?
🚀 However, the results of the A/B test contributed to Udacity’s recent acquisition of Terminal.com for a tighter integration of the Code Studio 🚀
Interested in learning more? Feel free to reach out firstname.lastname@example.org if you had any questions or feedback.