Increasing Online Course Completion Rate: a UX Research Study (part 1)
As part of the UI/UX Design Job Connector program at Purwadhika Startup and Coding School, we worked together on a group final project and my role was that of a UX Researcher. Here are some of our research findings.
Background
MOOC (Massive Open Online Course) has been around for quite a while. Eight years ago, the New York Times dubbed 2012 the year of the MOOC. Coursera and Udacity were already big names of the industry. Virtually every big U.S. universities raced to partner up with online course providers. Online courses offer equal opportunity for every person in every corner of the world to learn every kind of skill imaginable, from Harvard’s Computer Science class to forensic consultant certification.
However, a 2019 study by researchers at MIT pointed out that MOOCs aren’t working as well as it should be. As quoted by Inside Higher Ed:
Among all MOOC participants, 3.13 percent completed their courses in 2017–18, down from about 4 percent the two previous years and nearly 6 percent in 2014–15.
And there is evidence that this is not just a fad, but a worrying trend: this research showed data from 5.6 million online learners in 12.6 million course registrations from 2013 to 2018.
This leads us to the situation of MOOCs in Indonesia. Online courses are also quite prevalent nowadays, usually in the version of a website or mobile app. With more than 15 million learners and 300,000 teachers registered on its platform, Ruangguru is currently one of the biggest names in the education tech industry. Zenius Education lags slightly behind with more than 11 million users in 2019.
Although there is no exact statistical analysis proving that the same problem might also exist in Indonesia, it is fair to wonder if local online learners might be having the same problems as their international peers. With the rising number of users and thus the rising importance of online learning in Indonesia, we believe that we can conduct a research to solve this problem.
Research Object
As part of the project brief, we also picked an app to be revamped. We chose to improve Skill Academy, a new learning service for professionals launched in October 2019 and built by the brains behind Ruangguru. This research was conducted in February 2019, and the app might look entirely different by the time you read this case study.
Research Scope
Problem
In short, the completion rate of online courses is very low.
Hypothesis
People tend to not finish their courses due to a lack of motivation.
Research Goals
- Find out why people tend to leave their online courses unfinished.
- Get insights about increasing learning motivation.
- Identify solutions that have potential to be profitable for business.
- Find out about users’ opinion of the existing application.
Research Plan
We separated our research into two phases:
- Usability Research, in which we find out the pain and gain points of the existing application.
- User-centered Research, where we actually find out why online course users tend to not finish their course. Here we explore solutions to increase the users’ completion rate.
We decided to conduct two research phases with different objectives but ultimately the same goal: to improve the Skill Academy app. At the time of research, the app was still in its infancy and we believed that we can try polish many of its existing features and user interface, which was not directly related to our initial hypothesis. Therefore, we conducted the usability research in the first sprint and the user-centered research starting from the second sprint.
Usability Research
We use two methods here: guerilla usability testing (specifically using simplified thinking aloud technique) and comparative analysis with direct and indirect competitors.
Guerilla Usability Testing (Simplified Thinking Aloud)
We picked five people to think out loud while using the existing app. The “simplified” part of the technique’s name came from the fact that traditional Thinking Aloud method requires researchers to record videos of the tests before further analyzing them, whereas we only took notes of their thoughts. The simplified approach is enough because we just wanted to find out what usability problems people generally encounter, as described by Jakob Nielsen.
We asked our fellow UI/UX Design students to try using core features of the app, such as browsing for classes, buying classes, and watching lesson videos. The loose participant criteria were people who were tech-savvy, had an acceptable degree of knowledge about user experience principles, and were familiar with online learning services.
At a glance, the pain points relate to elements of user interface and not especially to the application’s usage flow. It is possible that the overall flow of the application do not need much revision, while the aesthetics might be in for a revamp.
Comparative Analysis
We compared the core features of Skill Academy to Zenius Education and Udemy. In a sense, these two products are both direct and indirect competitors for Skill Academy.
Zenius Education is a well-known Indonesian online learning services provider that caters to compulsory education students (up to twelfth grade). They do not provide services for professionals and is directly competing with Ruangguru. As of December 2019, they also opened all access to 80,000 lesson materials for free.
Udemy is a worldwide online learning platform aimed at professionals as well as students. They provide a plethora of learning materials with all sorts of topics, but do not tailor their offerings specifically to the Indonesian market. This might soon be changing, however, with the news of official expansion to Indonesia in 2019.
To date, Udemy is by far the most complete of the three, with additional features like lesson wishlist, Q&A forum, and shopping cart. Zenius also provides the option to download lesson videos, while Skill Academy rewards class completion with a certificate.
Summary
From the research findings, we gathered a lot of small and big improvement opportunities. We tried to gather all of them in one picture:
Going through all of them will take much of your reading time! Here’s three examples of important insights that we decided to act upon:
- Promotion banners on the home page were located on the bottom part of the page, and they are easy to miss (Disclaimer: it really happened to us: one of our team members paid the full price of one class because he failed to notice a 40% discount code!). We decided to move the banner to the top part of the page.
- At this time, we can only pay each lesson individually. Therefore, when users want to join two or three lessons at once, there were no option to pay them all at once. We decided to include a shopping cart option where users can pay in one batch.
- Full-screen videos cannot be watched in landscape mode, either by design or by default. We thought to showcase how videos should be able to play in landscape mode.
Conclusion
There were a lot of insights gleaned from usability testing and comparing the existing app to other competitors. We hope that we demonstrated how those insights were then summarized and turned into an action plan — the basis of our design decisions.
That’s it for the first part of this research. The second part covers user-centered research where we find out ways to increase the completion rate of online courses. Thank you for reading until the end!
— Serpong, where it rains but fortunately never floods.
Want to learn more?
If you’d like to…
· get an industry-recognized Course Certificate in Usability Testing
· advance your career
· learn all the details of Usability Testing
· get easy-to-use templates
· learn how to properly quantify the usability of a system/service/product/app/etc
· learn how to communicate the result to your management
… then consider taking the online course Conducting Usability Testing.
If, on the other hand, you want to brush up on the basics of UX and Usability, then consider to take the online course on User Experience. Good luck on your learning journey!
