The Complete Guide to Measuring Lostness
Many people tend to think they don’t have enough time. Not enough time to read, not enough time to sleep, not enough time to eat properly, and not enough time to run necessary errands. People spend good chunks of their lives looking for stuff. They look for car keys, Worcestershire sauce in the grocery store, baggage information in an airline website because they can’t find their suitcases, and where their bank decided to hide the check deposit function on its mobile app. Sometimes, they go back and forth through different (and similar) paths, desperately trying to find what they’re looking for. Not being able to find stuff in products such as websites and apps is a cause for delayed task completion, failure to complete those tasks, and feelings of frustration and dissatisfaction.
Say hello to the lostness metric.
The lostness metric
The lostness metric* is a measure of efficiency using a digital product or service. It tells you how lost people are when they use the product. Lostness scores range from zero to one.
A high score (closer to 1) means that people are very lost and having trouble finding what they need. A low score (closer to 0) means that people find what they want relatively easy. At lostness scores of about 0.4 and up, it is very clear a person is lost even if you sit next to him and watch what he does.
The lostness metric can only be measured in a meaningful way if you know what users want to do. This means that measuring it by looking at analytics data or logs is useless because you have no idea what people wanted to do, or which task they wanted to complete. The best way to measure lostness is in a usability test where participants are asked to complete a task. This way, you know exactly what they are trying to do, what the optimal path to do it is, and how lost they are.
The lostness formula
The lostness metric is calculated based on a formula shown below where:
- L is lostness.
- N is the number of different pages visited while performing a task.
- S is the total number of pages visited while performing the task, counting revisits to the same page.
- R is the minimum (optimum) number of pages that must be visited to complete a task.
Example: Using the lostness formula
If the formula scares you, have no fear. Let’s go over an example to explain the N, S, and R parameters in it. Later, all you need to do is track them and then enter them into a spreadsheet, which will automagically (yay) give you the lostness score.
Our example involves finding product page C1 in the website shown below. The shortest, most effective path to get to this page from the home page is through category page C. This is the R parameter in the lostness formula. In this case, since the minimum number of pages required to complete the task is three, R equals 3.
Let’s assume you are looking at what Dana did while completing the task.
Dana took eight steps to get to product page C1:
- Home page
- Category page A
- Back to home page
- Category page B
- Product page B2
- Back to category page B
- Category page C
- Product page C1
This is also S in the lostness formula, which is the total number of pages Dana visited while performing the task, counting revisits to the same page, so S equals 8.
Dana visited these pages while completing the task:
- Home page
- Category page A
- Category page B
- Product page B2
- Category page C
- Product page C1
In the lostness formula, this is N, which is the number of different pages Dana visited while performing a task. So N equals 6.
This allows you to calculate the lostness score for Dana in this task (see below), which is 0.56. A high score (closer to one) means that people are very lost and having trouble finding what they need. A low score (closer to zero) means that people find what they want relatively easy. At lostness scores of about 0.40 and up, it is very clear a person is lost when you sit next to him or her and watch what happens.
Track lostness
Use the lostness matrix when you observe a usability test or when you look at the path that each study participant took to complete a task. In this matrix, P1 to P10 are the numbers of participants in your test, so each row represents one participant session. In each participant session, five tasks are completed. For each task, log the N (number of different pages visited) and S (the total number of pages visited counting revisits to the same page). You can log the R (the minimum number of pages that must be visited) before or after the test sessions. The R will be the same for every participant in each task (i.e., there will be an R for task 1, R for task 2, and so on).
Important: If this will be the first usability test you are moderating, I stringly recommend you record the session and count lostness data after the fact. instead, focus on the participant, take great notes, and ask questions.
Calculate lostness
Use the lostness spreadsheet (after making your own copy) to add the lostness values you collected during a study. The spreadsheet is set for 5 participants who complete 5 tasks. Feel free to add or remove participants and tasks to match your study setup. As you enter the N, R, and S parameters into the spreadsheet, lostness scores are immediately calculated per participant and task.
Make changes and re-evaluate
Is your product’s lostness score too high? This is evidence that should make you take action. Make changes to your information structure, navigation, hierarchy, labels, element size, shape and location, as well as layout. After you do, re-evaluate with the lostness metric tracking to validate (or invalidate) that changes are working well for users. That’s evidence-based design at its best.
Practice!
Step 1: Watch this video of a participant attempting to complete a task.
Step 2: Track lostness as the participant completes the task and calculate the lostness score for it.
Step 3: Watch this annotated video to verify that you calculated the right lostness score.
Was that fun or what?
Tomer Sharon is the author of Validating Product Ideas Through Lean User Research. Get a 20% discount when you purchase the book directly at Rosenfeld Media while using the code tomernews.
Sign-up for Tomer’s Lean User Research newsletter to get access to more resources on the topic. The newsletter will appear in your inbox no more than once a month.
* Pauline A. Smith; Towards a practical measure of hypertext usability, Interacting with Computers, Volume 8, Issue 4, 1 December 1996, Pages 365 — 381.
