Blog Update #4
Blog Update #4a — Revised goal(s) of experiment:
Our experiment goals have shifted since blog update 3 to focus on the differences between our prototype and the pre-existing UBC lost and found database, a competitor system. Since the main finding from our field study was that the current process was inefficient, one of the primary focuses of our research is efficiency. We will compare the time it takes to search both our prototype and the existing database to find a certain lost item in order to evaluate and compare the efficiency of both systems.
Another critical concern that arose from our field study was the idea of security as it relates to ensuring lost items are returned to their proper owners. Therefore, we hope to assess if our prototype provides users with a greater sense of safety and security when compared to the UBC lost and found database.
Our revised goals are:
- Which interface, the Lost & Found Prototype or the UBC lost and found database, is more efficient for finding a lost item?
- Does the Lost & Found Prototype give users a greater sense of security when compared to the UBC lost and found database?
Blog Update #4b Experimental Method
Participants
We will be recruiting participants who are UBC students and who are routinely in and around the computer science building. We will use convenience sampling for the recruitment approach. Team members will ask friends if they are willing to participate in the experiment. The minimum target for the number of participants is 10.
Conditions
We will be comparing our prototype to the UBC lost and found database in terms of the time it takes to find an item and the satisfaction levels of each interface.
These conditions are based on the following short descriptions of each interface:
- The UBC lost and found database enables users to scroll and filter through a list of lost items by item type and location. The user is able to identify where the item was found and is provided with a brief (1–4 word) description of the item. The interface has minimal user interaction.
- Our interface enables users to scroll through a list of lost items, search for an item and filter either alphabetically or by date. The user is presented with a picture of the lost item with a description. The user is then able to claim an item by answering a security question.
Tasks
The participants will be asked to walk through a list of goals on each interface, with the same end goal — to find the appropriate lost item. The participants will have 3 tasks for each interface — to search for the lost item using the search/keyword bar, search for a lost item through the filtering options and look for an item that does not exist. All three tasks will be within subjects.
Task 1: Search Bar Method
Lost & Found Prototype
- Login with CWL
- Search for lost watch using search bar
- Claim the watch
UBC lost and found database:
- Search for UBC lost and found database
- Search for lost watch using keyword bar
Task 2: Filter Options Method
Lost & Found Prototype:
- Use filter functions to find the lost watch
- Claim the watch
UBC lost and found database:
- Use filter functions to find the lost watch
Task 3: Security Method
This task is included to address the security concerns presented through data from field studies. The UBC lost and found database does not provide specific security measures in case of wrongfully claimed items. The Lost & Found prototype will aim to address this concern by providing a log of items that have been claimed, and the opportunity for users to report a wrongfully claimed item. Therefore, participants will look for an item that does not exist on either interface database to identify whether the security concern can be addressed.
Lost & Found Prototype:
- Find an item that has been claimed by someone else (ie. does not exist in the database)
UBC lost and found database:
- Find an item that is no longer in the database
Refer to the appendix for a detailed descriptions of the tasks the participants will be asked to do, as well as scenario prompts to allow for simulating a real lost and found situation. Since participants are not actually losing any of their items, this would be hard to model in the experiment, but adding a scenario prompt should help to allow for a closer real world problem.
Formal Design
We will be conducting 3 separate within-subject experiments to correlate with the three different tasks the participants will be asked to do.
Procedure
- Participants will be walked through the context of our prototype
- Participants will sign a consent form in order for us to integrate their results to findings from our experiment.
- Participants will be asked to commence tasks 1, 2, and 3 with random assignment to which interface they will use first, as well as random assignment to the order of task completion to allows for counterbalancing.
- Participants will be verbally walked through the series of tasks they are meant to follow as they progress by the first experimenter.
- Participants will be timed as they walk through the individual tasks by the second experimenter.
- Participants will be asked to complete a 10 minute questionnaire on their satisfaction with both interfaces.
Apparatus
Users will be set up in the HCI lab in ICICS X360 at a round table. They will have a phone open and ready for them to use. The landing page screen with the button to login with their CWL will be open for our Lost & Found prototype and the chrome explorer will be open to allow the participant to start their task.
Variables
Independent:
- Interface (Lost & Found or UBC lost and found Database): Participants will be asked to complete similar tasks through successive steps altered to fit the functionality of each interface.
- Task type (Search bar or filter option): Participants will be asked to find a lost item using both the search bar options and the filtering options available on both interfaces. These task types will be within subjects.
Dependent:
- Time of completion: This will measure the amount of time it takes for the participant to find an item given a specific task. A team member will record the start and stop times for each task and participant, and record the times.
- User satisfaction: The user satisfaction will be determined through a likert scale that participants will be asked to fill out. This will help with understanding how satisfied the users were for each interface. Participants will fill out a short questionnaire indicating their satisfaction based on their ability to navigate the interface and how user friendly they feel the interface is. This will be filled out after the completion of the experiment.
The order of tasks and the order of interfaces the participant will use will be controlled by the research team to counterbalance skewed effects that may arise due to learnability and transferability. Similarly, the order of tasks will be counterbalanced.
Hypotheses
Time of Completion: (task 1 and 2)
- H1: The use of our Lost & Found prototype will allow users to find lost items more quickly in task 1 compared to using the UBC lost and found database.
- Null1: There will be no speed difference between the two interface types.
- H2: The use of our Lost & Found prototype will allow users to find lost items more quickly in task 2 compared to using the UBC lost and found database.
- Null2: There will be no speed difference between the two interface types.
Security Satisfaction
- H3:The use of our Lost & Found prototype will have higher satisfaction with security in task 3 compared to using the UBC lost and found database.
- Null3: There will be no difference in satisfaction with security between the two interface types used.
Our 3 separate experiments are designed to test the 3 hypotheses listed above. These predictions are based on previous research done.
When participants were asked to use the UBC lost and found database, their primary challenges were that they struggled with efficiency of the current systems. Our prototype is designed to specifically streamline this process, and if successfully done so, it would support our hypotheses regarding increasing speed in completion of our first 2 tasks.
In addition to efficiency, there is an overarching concern of security within the existing system. The UBC lost and found database is currently accessible to the general public, which means that anyone is able to view the catalog of items in the lost and found and potentially claim them. Therefore, our prototype added specific security verifications and claimed items history log to hopefully address this issue. Our goal is then to understand if these additional processes are helpful to our users, or if they would find it tedious to have to log in etc. This will effectively be tested with our third hypothesis, gauging overall satisfaction, more specifically regarding security.
Statistical Analysis
We will be conducting 3 separate ANOVA experiments as all the tasks are independent of one another. All experiments are within subjects.
Limitations
- Users will not be searching for an item they have actually lost. Thus, having users recognize their item from the picture and determining whether it is in fact theirs cannot be simulated. We will attempt to fix this by providing users with context about who they are and what they are looking for.
- The database of items in the prototype is small, reducing search time.
- Prototype is fairly horizontal — only a single linear path is supported, and this path might not necessarily be the same path our user intended to take.
- The prototype is optimized for a mobile app, and not for the web. The UBC lost and found database is optimized for the web (and actually does not display well when opened on a mobile phone device). For this reason, we will display the mobile app on the web. Although this nuisance variable does not directly limit our experiment, it is worth being aware of.
Blog Update #4c Supplemental experiment materials:
Links
