User Research Study: Why do Designlab Students Complete The Course at a Sub-Optimal Pace?

This article covers survey methodology, research results, issue prioritization and recommendations.

Background

Designlab is an intensive online UX/UI Design Bootcamp that pairs students with mentors who are established Designers.

Prompt

Conduct a user research study on a topic of my choosing.

Topic: Designlab

Analyze the pace at which students were completing the Designlab program. Anecdotical evidence indicated that students were progressing at a slower than desired pace and took longer to complete assignments than the course suggested time.

Objective

Understand two aspects:

  1. Why students are completing the Designlab course at a slower than optimal pace.
  2. How to improve the pace students move through the course, without sacrificing the quality of learning.

Research Plan

  1. Construct a survey to gain a high-level understanding of the factors that affect a student’s course pace.
  2. Survey both Designlab students and mentors.
  3. Analyze the student and mentor response.
  4. Compare responses with data provided by Designlab showing how many hours of coursework students completed each week.
  5. Provide key insights and a prioritized list of recommendations.

Survey Design

Student Survey Questions

The survey results aimed to determine why students were completing Designlab at a sub-optimal pace. In considering the responses from a MECE perspective (mutually exclusive collectively exhaustive perspective), reasons were categorized as either “Internal” or “External.”

  • Internal reasons included issues that students had specifically with the course content.
  • External reasons included issues such as motivation or unplanned personal events that took away from course time.

Survey questions 1 and 2 were meant to gain background context on each student which could potentially reveal behavior patterns.

Question 3a was used to establish a baseline of students’ expectations.

Question 3b was used to compare with 3a to get students to quantify their external issues. Question 4 was intended to have students then explain these issues.

Question 3c was used to compare with 3a to get students to quantify their internal issues. Question 5 was intended to have students then explain these issues.

Given the “warm connection” with the audience of this survey, I chose freeform responses instead of multiple choice because response rate would not be a large concern.


Mentor Survey Questions

It was important to collect mentor feedback to identify perspectives on obstacles that were slowing students’ progress that may not be obvious to the students themselves.

Question 1 was intended to ensure that mentors’ responses supported the students’ responses.

Question 2 was intended to gain any additional insights that the students may have never brought up directly.

Analyzing Survey Results

Framing the Problem

I was able to collect responses from 12/16 students and 10/16 mentors.

To make sense of the data, I first analyzed four key pieces of student data:

  1. Self Reported Planned Hours of Work Per Week
  2. Self Reported Actual Hours of Work Completed Per Week
  3. Self Reported Course Hours Completed Per Week
  4. Actual Data of Student’s Completed Course Hours Per Week Received from Designlab

These averages provided a starting point for framing the problem and visualizing the drop-off funnel. Students start with an ambitious goal of dedicating a daily average of 3+ hours to Designlab, but they work fewer hours than they had planned. That equated to a smaller number of Course Hours in a week, and, interestingly, Designlab data indicated that that number wasn’t even accurate; it was much lower.

A 58% drop off between the “Actual Amount of Course Hours Completed(Self Reported)” and “Actual Amount of Course Hours Completed (Based on Data)” seemed unusually high. I re-examined the data removing samples that had unusually high ranges or missing data.

This resulted in a funnel that was much more evenly distributed.

The data still painted the same picture. Students set out in the week planning to log 21 hours, but actually logged just under 15. By week’s end, the students believed that time equated to just over 9 course hours completed. In reality, though, it was just under 6 hours, according to Designlab data.

Graphic A (later used in calculations)

This even distribution in drop-off made it difficult to determine which area of the funnel to focus on first.

My next step involved framing the reasons students gave into three top-level categories:

  1. Non-course related factors (Outside factors directly unrelated to Designlab)
  2. Course-related factors (Assignments and course work taking longer than planned)
  3. Awareness (A lack of awareness of overall course progress)

Cleaning Up Data

I took every mentor and student response/reason, consolidated similar responses, and summarized them in the table below.

Bolded answers indicate that the specific reason was mentioned at least three times by students and/or mentors.

Graphic B (later used in calculations)

What became clear to me is that there is no “typical student.” There were a combination of factors at play causing students to move at a slower pace than they had intended.

Prioritizing Recommendations

I listed out the various problems that were slowing students down and included potential solutions. For each problem/solution pair, I included:

  • The number of times that the problem was mentioned in the survey responses
  • The difficulty involved with implementing my proposed solution
  • A weighted contribution score
Formula for Weighted Contribution Score

To determine the priority of recommendations, I calculated a final score for each problem-solution pair:

Top 5 Issues and Recommendations

Issue #1: Uncontrollable Outside Influences (External Non-Course Related Factors)

Any time that a student or mentor mentioned something external (e.g. work, school, family) that got in the way of Designlab, this was classified as an external, non-course related factor.

For the purpose of this recommendation, we should think of this category as largely uncontrollable factors that students cannot control or change.

Recommendation: Require mandated checkpoints/minimum hours of completed course work per week.

Recommendation Timeline: Future Cohorts

It would be unfair to impose these types of recommendations on the current batch of students since they were promised course flexibility. Designlab should be more selective when choosing future students that can guarantee a certain minimum hour commitment per week.

Issue #2: Making Refinements to Assignments

Students spent an excessive amount of time refining their assignments before submitting it for review because of two main reasons:

  1. Students are perfectionists and/or just not yet satisfied with the quality of their work.
  2. Students feel embarrassed submitting their work at its current stage for review by their mentor and other students.

Recommendation: Encourage students to submit early AND submit often. Possible solutions include:

  • “Re-framing” what a first submission should represent
  • Educating students about the value of iterations and early feedback
  • Re-tooling the “submission” flow to include a “submit to draft” option
  • Force students to submit a half-way draft
  • Allow for private submissions

Recommendation Timeline: Immediate

Issue #3: Lack of Awareness with Course Progress

Students are unaware of their current place in the course relative to the larger timeline.

There is a 28% decline between Actual Course Hours Completed (Self-Reported) vs Actual Course Hours Completed (Based on Data).

The only status report that students currently receive is a summary at the end of the week that shows the total number of Course Hours Completed. This makes it impossible to make adjustments during the current week.

Recommendation:

  • Place clear markers throughout the course for each 20-hour block that has passed; this will align with the suggested timeline for mentor appointments.
  • Provide students a simple real-time dashboard tool in the Designlab site header/footer/sidebar.
Suggested UI Design of a time tracking dashboard

Recommendation Timeline: Immediate

Issue #4: Motivation (External Non-Course Related Factors)

There is no external pressure on students to complete the course and no penalties for progressing through the course slowly.

Recommendation:

  • Financial Incentive — Offer some sort of course discount to complete the course by a certain date.
  • Social Pressure — Offer to pair students up with similar schedules and set “check points” throughout the course. No student is allowed to pass a check point unless both are finished
  • Enforce Small Daily Habits — Have students perform a daily, small Designlab activity that takes less than 5 minutes.

Recommendation Timeline: Immediate

Issue #5: Lengthy Reading Assignments

Certain reading assignments are extremely long, making them difficult to complete.

Recommendation:

  1. Re-arrange relevant assignments between the reading assignments so that students can take a break from reading and engage a different part of their brain.
Re-arrange assignments to be at the end of each topic instead of being bulked together at the end of the module

2. Cut down on long reading assignments from the User Centered Design Book. Identify other resources that state similar ideas in a more concise format.

Recommendation Timeline: Immediate

Additional Recommendations

Assignments are perceived to be needing large blocks of time to work on

Students reported working on Designlab on average of 4.4 times a week, for 4.7 hours each session.

Designlab should explore making content and assignments more accessible from a mobile device to enable students to have additional “short study sessions”.

One potential MVP would be allowing students to easily send themselves articles and videos that they can review when they have a few spare minutes.

Unfamiliarity with Tools

It is very time-consuming to learn tools such as Photoshop, AI, and Sketch without prior exposure. Designlab can make this easier by sending out instructions as pre-course work or display tool-learning resources upfront on its website.

Unclear Assignment Prompts

Currently, students go to the “explore” section and search for examples of other students’ assignment submissions. It would save time to link directly to past submissions from the assignment prompt page.

Assignments Genuinely Require More Time Than Estimated

Survey students and ask them how long an assignment actually takes them to complete; then, adjust accordingly.

Conclusion

I set out to research why students struggled to move efficiently through Designlab and to identify ways to improve the program. In conducting the research, my biggest takeaway is that it’s difficult to push people to progress through a course when there’s no external pressure on them. That’s truly the largest issue.

Another key takeaway is the importance of incorporating factual data. It was unclear how to prioritize the identified issues because the responses were extremely scattered. By later incorporating statistics from Designlab and comparing with survey results, I was able to concretely form and prioritize my recommendations.


Update 11/29/15: Designlab recently made a blog post taking action on several of my top recommendations regarding changing the structure of the course timing and adding clearer weekly progress indicators.