Better Matchmaking on OkCupid

— A UX Case Story

Introduction

Using the user-centered design (UCD) process, I designed new features for the OkCupid dating app. The goal was to improve the quality and accuracy of the product’s matchmaking. Below is the story of the design project, plot twists along the way, and an interactive prototype.

Only have a few minutes? Jump to Results & Conclusion.

Tools used: Balsamiq, InVision, OptimalSort, TypeForm, Skype, Trello, Dropbox Paper, (lots of) real paper.

Team / Role: yours truly playing user researcher, interaction designer and product manager.

Constraints: limited access to users, limited time, and (arguably) too large of scope for one person.

Table of Contents

1. Part 1: The Problem
2. Understanding the Users — initial survey, interviews, empathy maps, personas
3. Usability Analysis — heuristic analysis on current product and competitors
4. New Opportunities— MVP feature definition, user stories
5. Constructing the Mental Model — card sorting, information architecture, site map, user flows
6. Detailed Design —doodles, sketches, wireframes
7. Lessons from the First Cut — feedback and lessons from initial designs
8. Putting it all Together—final usability testing with prototype, style guide
9. Results & Conclusion — summary of results, retrospective and next steps

Part 1: The Problem

It all started innocently enough.

Looking at my OkCupid Match Question inventory, I noticed my own dissatisfaction with the interface…

• There’s a dropdown of subcategories, but no overview of each subcategory.
• Count of items in this list?
• Is there a sort order?

We don’t know.

Even the number of pages is not clear until you scroll to the bottom

We also see the interface forces me to paginate 10 questions at a time…which gets tedious real quick when you have an inventory of over 600 items and no way to search them.

Here’s the whole page with the middle cut out:

I looked at the mobile OkCupid App. It didn’t seem much better.

Again, no sort order or list count:

Given all that, I smelled an opportunity for a UI redesign project.

Part 2: Understanding the Users

Before diving deep into sketches and wireframes, I wanted to better understand the users and their sentiments around the current interface, and determine the greatest opportunities for improving the matchmaking system. I surveyed 26 people and conducted 7 phone interviews with real users of OkCupid.

High Level Research Questions

1. What are users’ biggest pain points with using Match Questions?
2. When has OkCupid’s match system failed users? E.g. it said you were compatible but you weren’t.
3. How can Match Questions (and the Match Engine at large) deliver twice the value it currently does?

I targeted users who are:

1. both male and female, and
2. who are over 25 years of age. Why? Enough life experience to know what they want in a dating partner, to whom a sophisticated matching algorithm like OkCupid’s would be more valuable.

The Screener Survey

I ran an online survey using Typeform.

26 people responded.

Survey Highlights:

Not a surprise, but good to confirm there are more mobile users than desktop:

Varying degrees of investment in Match Questions:

Validation that the site works and people are meeting up for dates (not a lost cause)

Most Surprising Finding from the Survey

“I like it. I think it’s really really good. Highly accurate.”

“Crap”

“A little indifferent. The guy who is now my boyfriend was actually a lower match percentage than most.”

“It’s alright I guess.”

“It’s nonsense”

“It’s good at weeding out obvious incompatibilities e.g. racist = low match, but sometimes I’ll have a 95% match where I read their answers and wonder how the hell they must have weighted their questions for us to score that well together.”

A very wide range of sentiments!

(…What could be the cause of such variation?)

User Interviews

To understand why people’s sentiments varied so much, and also to discover other possible improvements, I interviewed seven real users about their experience with the app.

I prepared 30+ interview questions focused on the matchmaking aspect of OkCupid.

A few of the questions:

Q: How much science and predictability do you think there is behind romantic compatibility?

Q: Have you read the article on how the match algorithm works?

Q: If I said by answering 1000 match questions OkCupid can find you the love of your life (i.e. likely the best match you’ll ever find in the world), how confident would you be that would happen?

Q: How do you feel about the consistency of results based on Match %? How would you rate consistency, on scale of 1–10? If not consistent, how do you then go about managing screening the results, if at all?

Q: How did you get into OkCupid in the first place? Was there some particular feature / selling point? Or something someone said about it?

Q: What do you typically look for when determining whether you want to engage a person (on the site)? In other words, how do you screen a profile?

Q: As you met more people in dating (not just through dating apps), did you notice a change in what you wanted in a dating partner? Or did your preferences largely stay the same? Tell me more about that. (what changed, If anything?)

Q: What do you find most difficult or frustrating about using OkCupid’s Match Questions

Findings from Interviews

Not surprising Findings:

• Assumption confirmed — People DO look for intangible qualities in a match. Every person I interviewed enjoyed being able to read in-depth profile essays. Examples include grammar, intellect, lifestyle similarities, values, drive, ability to hold a conversation. One respondent explicitly did not want to date someone in her own profession because she’s “tired of talking about it all the time”.
• When asked to rate their confidence in OkCupid delivering the love of their life (i.e. an extremely good match), the highest rating recorded was 6.5 out of 10 (lowest was 4). Users perceived gaps in the capabilities of the matching algorithm, such as factoring physical appearance, or personality chemistry. In order for users to raise their confidence in Match Questions, I suspect the model would have to change, suggesting deep enhancements to the product itself (see section below on Opportunities for Improvement)

Surprising Findings:

• Five of the seven people I interviewed did not know how the match algorithm works, though it “made sense” when I explained it to them.
• Geographic proximity was consistently mentioned as a major factor in the quality of a match.
• Regardless of what stage of their dating career they were in, each person I spoke to was highly interested in finding people of high compatibility.
• Those that understood the match algorithm seemed to have more confidence in its results.
• Despite gap in perceived efficacy and accuracy of the matching system, every user I spoke to DID EVENTUALLY NOTICE a correlation between Match % and real compatibility, even if they didn’t notice when they first started using the app.
• Even those that ultimately abandoned the app did put significant effort into using the matching system to find good matches.

Strengths of the Current Experience

From the perspective of users, OkCupid has several strengths that stand out among their contemporaries.

Correlation between Match % and real compatibility. Many users discovered a strong correlation between Match % reported by the application and the level of compatibility when meeting the person in real life. Some discovered this correlation sooner than others, which may suggest an opportunity to improve discovery or learnability. Overall, this is validation that it does work (at least to an extent).

Deep, detailed user profiles. OkCupid provides access to deep, detailed profiles, allowing the user to learn a lot about a profile before meeting the person or even engaging them online. This alleviated a lot of the work of initially getting to know someone and provided some confidence that further efforts were well-invested. Overall, users felt that content and style of writing was a strong indicator of a person’s real life persona, so the written essay profile became a common area for scrutiny. This is also a feature that separates OkCupid from many of its competitors (free dating apps such as Tinder, Bumble, Coffee Meets Bagel), who do not support longer written profiles.

Rich and flexible search capabilities. OkCupid gives users many filters through which to search for potential matches. Filters range from how much a user smokes or drinks to whether they want kids, to their ethnicity and level of education.

While these search filters are standard in many dating apps, OkCupid is still one of the few free apps that offer this degree of flexibility and options.

Large user pool. OkCupid boasts a large pool of active profiles, especially in large metropolitan areas. This greatly increases the chance of finding a specific kind of profile.

Opportunities for Improvement

1) Match Quality. While correlation was strong and consistent, some users do complain about matches still not being good enough. One user said the algorithm got 70% of the way there, but it wasn’t good enough. After several lackluster dates, she got tired of using the app and quit altogether. This suggests room for improvement in providing higher quality matches.

In that same vein, there is also some doubt in the accuracy of the Match %. A 95% match isn’t always better than a 70% match, for example. Users report false positives. Those users often got in the habit of doing further validation and “checking OkCupid’s work”.

Thus, while the correlation between Match % and actual compatibility is palpable, users would appreciate improvements in accuracy and quality.

Una Drake, a professional dating coach, suggests much of the poor experience stems from a lack of understanding of how the match algorithm works, which leads to common pitfalls in using it.

The evidence makes clear that users are not equally effective at using the current self-serve model.

2) Tedium. Multiple users complained about the tedium of sifting through profiles to find a good candidate. Some attributed it to the product. Some attributed it to themselves (e.g. being “too picky”). Regardless, reading through profile after profile got tiring for some users.

3) Relevance of Match Questions. One user expressed difficulty in refining her Match Questions. After using the app and engaging other users, she wanted to modify some previous questions to filter out some kinds of people. While she was able to change prior answers, she wasn’t able to find new questions she wanted to add. “They weren’t asking everything I wanted to answer. I wish I could write my own questions.”

Another user called the Match Questions “stupid”, despite her paying close attention to the Match % while using the app.

Personas

Users I encountered vary in many dimensions

• Degree of stigma associated with dating apps
• Age / Stage of life they’re in
• Gender
• Desire for children

But the distinction most relevant to my design effort is amount of experience using the app.

Ultimately I chose to focus on two personas:

1. Someone who was a novice to the site. To add another dimension, this person is not looking for a long-term relationship.
2. Someone who was experienced with the site, more familiar with the matching system. This person is also looking primarily for a marriage partner.

Empathy Maps:

Conclusion

Clear gaps in the product are emerging. I should keep in mind its current strengths and not lose sight of them as I redesign parts of the product.

Part 3: Usability Analysis

To understand the competitive landscape, as well as current UX conventions in my project’s domain (matchmaking features in online dating sites), I surveyed and analyzed four similar products:

• eHarmony, a subscription-based online dating site focused on finding marriage partners,
• Match.com, another subscription-based online dating site,
• OkCupid, a free or paid online dating site, and
• Pandora, a free or paid music curation and playback service.

Why Pandora?

While not a dating service, Pandora was chosen because it has a few similar features:

• An invisible “smart” algorithm behind the scenes curating “matches” for the user’s feed(s)
• Collects a high volume of user preferences to tune “match” results
• Has a search engine that draws from a large marketplace
• Discovery — facilitating the user to discover something new and interesting
• Captures “likes” and “dislikes” on individual items
• Explicates WHY an item was curated for the user

Top Three NNG Heuristics

Out of the 10 Heuristics for User Interface Design by Nielsen Norman Group, I chose three I considered most relevant to my project:

NNG Heuristic #1: Visibility of system status.

Dating apps are operationally complex: users need to manage an inbox, maintain a public profile, search for and screen match candidates, and then some! Throw in tuning of match criteria, and you’ve got a complex operation to handle through a UI. With many moving parts, the visibility of system status becomes all the more critical.

NNG Heuristic #10: Help and documentation.

OkCupid’s matchmaking algorithm is driven entirely by user input. If the user makes a mistake, their results with the app will suffer. This means help, guidance, and support are critical in the user’s success.

NNG Heuristic #8: Aesthetic and minimalist design.

With the complexity of using a dating site, any effort to simplify the user experience can have a positive impact on the user experience, as well as set us apart from competitors.

Scope

The scope of my analysis includes any part of the user experience related to matchmaking:

• Initial match preference intake
• Display of match queue
• Sorting and filtering of said match queue
• Presentation and interactions with individual match profiles
• Editing and modification of match criteria & preferences

Notable Findings:

eHarmony has arguably the best visual display of information on its individual profiles. Well-organized and hits the sweet spot between too much and not enough information. This is aided by the product’s narrow focus on customers looking for marriage. It also had the most mature and comprehensive help documentation and customer knowledge-base.

eHarmony’s compatibility bar graph is also unique among its peers:

Pandora gave excellent guidance to first time users, providing a solid introduction to the platform.

• Browse and Search options for creating the first station:

• Upon launching the first station, its features are explained:

• Pandora also highlights WHY a particular SONG was chosen for a station:

Match.com does the best job visually highlighting matching criteria on the profile page itself.

Rating Summary

OkCupid was rated high in Aesthetic and minimalist design, but low in Visibility of system status and Help and documentation

Strengths:

• Match % is a simple number that lets users quickly compare two profiles side by side.
• Key topics are addressed in the FAQ
• Search Filters are well organized and clear without obstructing its sophistication.

Weaknesses:

• All Help topics are tucked away deep under Settings
• Some help articles are long and would be aided by pictures.
• No indication of how complete your set of Match Questions is.
• Not easy to understand WHY someone was matched with you.

More details in the full report: Heuristics Analysis Report

Conclusion

Each app had an area where it shined, and OkCupid can definitely learn a thing or two from its contemporaries!

Part 4: New Opportunities

From the user research data and usability analysis results, new questions emerged:

1. Can we expose users to the inner workings of the match algorithm (in an intuitive way) so they can easily and naturally self-correct cases of misconfiguration and prevent false positives and false negatives?
2. Can we re-model the organization of user preferences to resemble how users naturally think about their true preferences and desires? Will this lead to more effective match preference tuning (and more customer success)?
3. Can we expand the scope of the match algorithm so match scores are holistic and complete? Would such an improved model for compatibility lead to better matches?

To answer the questions above, I decided to focus on three new features:

1. A new and improved Match Scorecard that displays the composition of each Match % Score, with links to sub-categories to let users adjust and fine-tune.
2. Custom Match Categories that allow users to GROUP, LABEL, and RATE related Match Questions so that 1) users can specify what’s most important to them, 2) users can see all their preferences in one place and tune their preferences easily, and 2) users can customize what’s shown on the Match Scorecard, and in what order.
3. Factor Search Filter Criteria into Match % Scores. That is, allowing the Match % Score to include factors like height, ethnicity, age, distance/location, etc.

User Stories

I used the User Story Mapping technique to flesh out user stories:

As a user, I want to…

(Related to Match Scorecard)

• Drill down on a category to see match questions within
• See Thumbnail of Scorecard on Match Profile
• View distribution of Match % by Category (donut chart?)

(Related to Match Categories)

• Group Questions together into Categories
• Rate importance of an entire Category
• Browse / discover Match Categories to add
• Add a Match Question to a Category
• Create a Match Category and name it
• Create a Match Category from a Match Question

(Related to Weighted Search Criteria)

• Rate importance of Search Criteria to factor them into Match %

See more details in the full report: MVP Features & User Stories Report

Part 5: Constructing the Mental Model

Mapping the Current Mobile Site Map

Here’s the site map of the current mobile app. It shows the five top level navigation links on the home page, and how they drill down to the Full Profile:

Card Sort Sessions

• 28 card labels based on the MVP User Stories and Preliminary Sitemap.
• 4 sessions with different participants, three remotely moderated, one unmoderated.
• Done using OptimalSort

Example: One of the live sessions:

Similarity Matrix produced by OptimalSort:

Some of the Category Names:

How it Works
Add Match Preferences
Organize and Group
Categories
Further customization of match questions
Matching Details
Explore
NO — Reject / filter out

Key Insights:

• The profile is likely the best page to teach the user how matching works.
• Organization / sorting / grouping is a very familiar activity to users and we should heed current conventions.
• Rating preferences by importance is best done when users can see the preferences side by side (and compare).
• Deal-breakers, can’t-stands, and traits users want to avoid in a match should be grouped together as things to screen and filter out.

New Site Map

With a richer mental model from the Card Sort sessions, I brainstormed and added the new features to the site map.

Major changes:

• The Scorecard is accessed through any Full Profile view.
• A new page dedicated to Match Questions, Categories, and all of one’s preferences. Under this page is where users organize, group, sort, rate, and otherwise manage their preferences and importance ratings.
• Search Filters have a new workflow to include the user’s preferences (e.g. height preference) in Match % scores.

User Flows

To pin down more details, I then carved out three critical user flows to focus on:

1. “After looking at a profile and its match scorecard, user decides to adjust the importance of Match Questions related to sexual interests and preferences.”
2. “User factors preferences for religious background into Match %.”
3. “User creates their first Custom Match Category based on intellect and intellectual compatibility.”

These drawings may look crude, but the process forced me to think through the user experience in greater detail:

Part 6: Detailed Design

— Doodles, Sketches, and First Wireframes —

With some high level designs in place, we can finally get down to the details!

1. The Match Scorecard:

Bar graphs on a spine were my favorite because they showed proportion as well as positive and negative values.

2. Custom Match Categories.

Early sketch with category label on each Match Question page:

3. Added “Add to Match %” to Search Filter preferences pages:

Wireframing the User Flows

I used Balsamiq, primarily for its speed and focus on low-fidelity rendering.

User Flow #1:

( note the three pages of the Match Scorecard and Category details in the “Sex” match category )

User Flow #2:

( note the new custom preference form on pages 6–7 )

User Flow #3:

Part 7: Lessons from the First Cut

Initial Feedback

Exhibit A: I received feedback that the match scorecard is potentially overwhelming and confusing for first-time users.

After revisiting the goals for feature, I simplified it to one page showing the most relevant data.

Other Changes:

1) Added graphic header title (so users are clear where they are)

2) Added an “Importance” label to remind user how these scores are tied to their Match Question answers

3) Added a brief overview of each category and its contents

Exhibit B: Feedback: Adding Current Search Filter Criteria (e.g. height preference) as a Match Score Preference should be separated from Search Filters and given its own structure.

1） “Add to Match %” muddles the current workflow for Search Filters.

2） Having another entry point seems disjointed and confuses the user.

Changes:

1. Removed the 2nd entry point for Search Filters in Match Preferences, leaving the page focused on only Match Categories (and renaming it “Match Categories”).
2. Removed changes to Search Filter workflow so there’s only one entry point (inside Match Categories page)
3. Preferences based on Search Filters renamed and moved lower (and renamed to “Common Preferences” to signify that they are a part of Match Categories, while distinguishing them from Search Filter criteria.

Challenges

At first it didn’t feel right to duplicate the same preferences between Search Filters and Match Categories, but after revisiting the information architecture, reviewing the card sort results, and sketching a few more ideas, I became more confident it was the best approach for the user.

Exhibit C: Feedback: Consider making Importance ratings of Match Questions more visible

As they are, each list of Match Questions doesn’t reveal much about them as a collection, and importance ratings, a key component to the matching algorithm, is hidden.

Changes:

1) Match question lists now show how important each question is. Together with the Match Scorecard, this will enlighten users on how Match scoring works (without making them read documentation).

2) Entire Match Categories can now be rated on importance, and with similar effects on Match % Scores.

3) The same importance bars is also applied to the new common preferences.

Other Challenges

Feature Creep. Less of a design challenge, more of a project management pitfall — I found it tempting to start drawing secondary features and stories (e.g. Previews of Match Score changes throughout the flow, Discovery of new Categories and Questions). To address this, I organized my MVP features and user stories in a more prominent spreadsheet to help me stay prioritized and avoid feature bloat.

Part 8: Putting it all Together

Having addressed the issues raised in initial feedback, I prepared a click-through prototype in InVision for usability testing.

You can run the prototype here. It covers the three user flows outlined earlier.

Usability Feedback

I drafted a script and scheduled four moderated usability sessions with four users of OkCupid (two were more experienced users, two were less experienced). The sessions ran from 30 to 60 minutes. I laid out the steps for the three user flows and asked them to go through the steps. I also collected general feedback and recommendations.

The full script and detailed reports on each session can be found here: Research Evaluation

Key Observations & Suggestions

• More thorough on-boarding is needed if I want to fully expose the new features and foster engagement.
• Several users got stuck moving a question between Categories:

• The data visualization used on the Match Scorecard was well received. One user said “I like it! Very intuitive!”

• The “Common Preferences” workflow could be potentially condensed down to two screens (instead of three):

• More prominent displays of deltas / changes to match scores and sub-scores would be useful. This could be done, for example, using arrows, animations, and/or colors. In the image below, we see a transition, but the users misses it if they weren’t paying attention.

• The Importance Bars I introduced were found useful!

Overall, users found it quite intuitive, and shared a positive sentiment.

Style Guide

Finally, time to add some color.

After exploring some alternative colors (gold, purple, brown), I decided my features, as cool as they are, were not worth a total visual overhaul of the product. I stuck to the app’s current color palette.

Results & Conclusion

So how did we do against the original hypotheses?

Hypotheses I:

Can we expose the inner workings of the match algorithm so users can easily correct and prevent cases of false positives and false negatives?

Yes! We addressed this with a myriad of elements:

a) Scorecards show the weight of each category relative to other categories, allowing users to quickly see when a category has too much (or too little) importance.

Before (OkCupid Desktop) and After (new mobile design)

b) The new importance bars (right) make clear the question’s weight in the matching algorithm. It then becomes very natural to sort by importance.

c) When users change the importance rating on any questions, the impact is immediately visible via the scorecard bar in above the list of questions.

In the example below, the user decides two questions are only Somewhat Important to her, rather than Very Important, and we see immediate visual feedback in the scorecard bar. False negative understood and corrected!

Hypothesis II.

Can we re-model the organization of user preferences to resemble how users naturally think about their preferences and true desires? Will this lead to more customer success?

a) Categories themselves can now be rated by importance, enabling further customization for the user. This is closer to how users think of compatibility, i.e categories.

b) A dedicated page for match preferences, free from the distraction of pretty pictures of cute faces and hot bodies. This helps the user pause, focus, reflect, and tune their preferences:

c) Having the match scorecard list mirror the list of match categories helps users naturally connect the dots (i.e. how they organize their categories is how they show up in scorecards and other reports)

Hypothesis III.

Can we expand the scope of the match algorithm so match scores are holistic and complete? Would such an improved model for compatibility lead to better matches?

Yes, I think we made progress here as well.

a) We can now see factors like Distance and Child-rearing views alongside more traditional categories like Religion and Sex topics.

b) Specialized question cards (left) can be created for match factors like Looks, Distance, and others that don’t naturally fit into the traditional multiple choice format (right).

Retrospective

Did I reach success metrics?

No, until this is implemented in production (and we measure how quickly users reach their goals), I can’t say whether it works or not.

Did I work through the RIGHT problem?

I believe so. The gap was apparent from the user research. Quality matchmaking lies at the core of this product, and any improvements we make to it will enhance its value proposition.

What would I do better if I could do it again?

Expose my ideas to more people and earlier in the process. Scheduling regular meetings with users would be a good routine to ensure I don’t stray far from what’s important to them.

For the card sort, I would also aim for shorter and more granular labels, and probably conduct multiple card sorts. These new features would have benefited from additional exploration on the information architecture.

Overall, I think I went in the right direction, and these features hold a lot of promise.

What’s Next

I’ve gone as far as I probably should on my own. There’s a great opportunity here to dramatically improve the product as a matchmaking platform.

For this to be released, the next steps would involve more usability sessions to flesh out edge cases, design the onboarding process, and then build a prototype that uses real user data.

If you work at OkCupid, I’d love to hear your opinion.

If you’re a fellow OkCupid user and want to see these designs implemented, please contact OkCupid, then contact me.