Our early designs underwent weekly in-team critique. However, during our first usability study over the Winter break, we uncovered more issues than anticipated.
To better understand how users interact with Filmify, we decided to place a greater emphasis on UX research.
Iterative usability testing
All design wireframes were evaluated at various stages: low-fidelity, high-fidelity, and before handoff to development. Each weekly sprint, we determined a batch of wireframes ready to prototype and test.
Major pages tested included:
- Landing
- Education
- Film details
- Browse/search
- Analysis
- Recommendations
- Our Algorithm
We also occasionally tested specific designs, such as our film hover states, Diversity Score meters, and cast/crew breakdown graphs.
These weekly batched studies allowed us to rapidly gather feedback, iterate, and send back for further testing. Ultimately, each page underwent 3–6 studies (with 4–6 users each study) before handoff to developers.
Participants
We tested with Filmify’s representative users, which are people who watch films often and care about films, for reasons more than entertainment (film enthusiasts). Target users also have a Letterboxd account, so depending on the feature being tested, ownership of a Letterboxd account was included as an additional criterion.
We grew a database of over 100 test participants, keeping names and contact information confidential.
Process
Each weekly sprint involved an end-to-end research process:
- Developing research objectives
- Preparing an interview script
- Preparing prototypes
- Recruiting and scheduling participants
- Conducting 4–6 moderated tests
- Analyzing findings
- Presenting and recommending concrete action items to the team
This intensive process every 7 days required early and constant preparation.
Methodology
We conducted 1:1 moderated usability tests with 4–6 participants every sprint. Tests utilized a mixed-method study design:
- Open-ended interview questions, to learn more about user background with film watching and awareness of the state of diversity within Hollywood.
- Think-aloud usability tasks, to place users in realistic scenarios and assess their comprehension, interest, etc.
- Retrospective feedback.
For the final prototype, in addition to the moderated tests, we conducted 10 unmoderated tests, as a time-efficient way to engage more participants. This was done using a free usability research tool due to budget constraints. All participants were asked to complete a System Usability Scale (SUS) follow-up survey.
Impact
UX research challenged assumptions and influenced many design and product decisions. For example:
- Signed-in users expected the Filmify logo in the top left to lead to the browse page, prompting us to update the navigation.
- First-time users’ behavior led to changes in the placement of key features, such as Diversity Scores.
Improvements were validated through increasingly positive feedback and decreasing issues in each round.
Conclusion
Regular usability testing has been vital to Filmify’s success. We were fortunate to have dedicated UX research this semester, in sync with design and development!
Stay tuned on our Medium page for more weekly sneak peeks of our design and development process behind the scenes.
To get a notification when we launch, join the new waitlist on our site! We hope to see you there! ✌️
