Robinhood Design Evaluation: A comparison of data visualization evaluation methods

Project Overview

In this project, I conducted a review of three design evaluation methods used for evaluating data visualization and interface design in general. The three practices were reviewed by employing them to assess the data dashboard of Robinhood, a micro-savings and investing app. Robinhood uses a variety of information visualization on its investing pages, which were interesting to evaluate. Their levels of severity defined flaws to highlight the urgency of the problem. Two proposals were made for fixing. After the evaluation, I conducted a comparative analysis of the three methods and subsequently evaluated each one of them. Future work is needed to conduct a more thorough evaluation and compare more evaluation methods to highlight the best data visualization evaluation method.

Background

Design evaluation is an essential method in user experience design and research. We conduct evaluations to identify design and usability flaws to fix them. A fundamental benefit of design evaluation is its discounted approach and rapidity of execution. There are several commonly used evaluation methods, but the researcher also creates new ones through a combination of existing heuristics, literature review, etc. While most of them fit the digital design domain, not all can effectively identify or communicate issues.

In information visualization, evaluation methods and critiquing usability and design efficiency mostly rely on theories and methodologies that predate devices they have been used on. On mobile phones, the array of evaluation methods sometimes fails to address fundamental interaction flaws compared to evaluating applications or websites on desktops. One way to learn more about the evaluation of information visualization made for a mobile application involves looking at data dashboards on microsaving and investing in mobiles that employ all the InfoVis and mobile app features.

A review of Evaluation methods

Usability and design evaluation has, over the years, become a go-to method for design professionals and students alike. Though beneficial in a user experience design, it can be detrimental if not implemented in the right context at the right time (Greenberg & Buxton, 2008). Being one of the earliest and popular methods of assessing interface design effectiveness, several researchers have proposed and implemented heuristics and benchmarks by which an interface should be measured. Nielsen’s ten heuristic principles and Shneiderman’s “Eight Golden Rules of Interface Design” are the most notable. Most new heuristics are either created from existing ones or based on the product or feature being reviewed. Even with its array of benefits, this discount’s overarching problem stems from three issues, heuristics, context, and evaluators. For instance, the heuristics cannot fully describe the problem identified and the urgency needed to address it. Also, the context of use, which varies from the evaluator’s perspective, may affect the evaluation’s outcome compared to empirical testing. The evaluator’s experiences can also affect the evaluation in general (Forsell & Johansson, 2010). Below are some strengths and weaknesses of design evaluation.

Strengths

1. Cheap and easy to implement. With limited resources, a design team can quickly evaluate designs with existing heuristic standards to measure design success.
2. It can be implemented along with empirical usability testing.
3. Due to its rapid format, evaluators can identify issues very early and establish their impact on the product.

Weakness

1. There can be a mismatch between the heuristic standard and the feature, product, or system, overlooking a problem or raising a false alarm.
2. Evaluation, in general, mostly fails to address the context of the use.
3. Evaluators from different, widely different backgrounds and experiences can affect the results of the critique.
4. Evaluators can fail to address critical issues like ethics, culture, race, and its impact on the product.
5. When conducted early on in the design process, it can weed out great ideas since they might not meet specific standards early on.

Design for the fintech industry

Financial technology (Fintech) has reshaped and disrupted the finance space across the globe. Microsaving and investment apps like Robinhood, Cash App, Coinbase, Acorns, and Stash have exposed millions of people, particularly millennials, to the traditional financial industry. Through mobile technologies’ ubiquity, traditionally complicated activities like lending and borrowing, savings, stock trading, money transfers, etc., have been simplified, breaking the steep learning curve and entry barriers (Webel, 2020). This simplicity is also a result of a general approach of catering the financial industry to millennial users who are design-savvy and have inexperienced with the traditional financial system. FinTech within the likes of microsaving and mobile investment services have been leveraging simplified data dashboards to communicate with their users. To understand stock market trends, spending trends, portfolio growth, simple data visualizations, and data dashboard, they are used for quick and easy understanding, making them particularly significant for today’s fintech growth.

For this project, I will be evaluating Robinhood. Robinhood is a free-trading app that lets investors trade stocks, options, exchange-traded funds, and cryptocurrency without paying commissions or fees. The company offers a mobile app and website that provide people the ability to invest in stocks, ETFs, and options through Robinhood Financial and crypto trading through Robinhood Crypto. You can download the app onto platforms like iOS, Android, Windows. The app’s core features include stock trading, Cryptocurrency trading, Money transfer, and Cash Management.

Methodology

Evaluation Goal

This evaluation primarily uses three popular evaluation methods to evaluate the data dashboard of Robinhood, a mobile investing app. Considering the project’s timeline and the app’s core functions, I selected two pages for the evaluation — the homepage and the stock detail page.

Evaluation Strategy

This project’s primary strategy was to select three different notable heuristics and use them for the evaluation. Rather than focusing on the evaluation, the focus was on the strength and weaknesses of each method. The first method was the Top 10 heuristics gathered by Forsell & Johanson; the next is Shneiderman’s Visual Information Seeking Mantra, and Four Data Visualization Heuristics Facilitate Reflection in Personal Informatics.

Scenario

I created a scenario under which the evaluation would follow. The tasks for the review were generated based on the scenario.

A user picks their mobile phone to check the overall gains in a day on the trading platform. The first page they access is the homepage. On the homepage, the user wants to view or drill down into a stock/asset’s details after seeing that it’s increasing or losing value.

Task (View Portfolio details)

View the homepage, check the portfolio’s overall trends, review the stock details, and check the performance over time.

The Evaluation Methods

1. Ten information visualization heuristics gathered by Forsell and Johanson

This heuristic method is a combination of a selection of high-level heuristics from four different evaluation methods (1. Ergonomic Criteria for Hierarchical Information Visualization Techniques, (Luzzardi et al., 2004) 2. Ten Usability Heuristics (Nielsen, 2005) () 3. Perceptual and Cognitive Heuristics, 4. Ergonomic Criteria for Interactive Systems((Scapin & Bastien, 1997) ) that has the most comprehensive coverage to explain all problems. The heuristics are broadly curated to address various usability issues in a product or feature that uses information visualization.

Information coding- Perception of information is directly dependent on the mapping of data elements to visual objects. This should be enhanced by using realistic characteristics/techniques or the use of additional symbols.

Minimal actions — Concerns workload concerning the number of steps necessary to accomplish a goal or a task.

Flexibility- Flexibility is reflected in the number of possible ways of achieving a given goal. It refers to the means available to customization to consider working strategies, habits, and task requirements.

Orientation and help- Functions like support to control details, redo/undo actions, and represent additional information.

Spatial organization- Concerns users’ orientation in the information space, the distribution of elements in the layout, precision, legibility, efficiency in space usage, and distortion of visual elements.

Consistency- Refers to how design choices are maintained in similar contexts and are different when applied to other contexts.

Recognition rather than recall- The user should not have to memorize a lot of information to carry out tasks.

Prompting- Refers to all means to know all alternatives when several actions are possible depending on the contexts.

Remove the extraneous- Concerns whether any extra information can be a distraction and take the eye away from seeing the data or making comparisons.

Data set reduction- Concerns provided features for reducing a data set, their efficiency, and ease of use.

2. Four Data Visualization Heuristics to Facilitate Reflection in Personal Informatics. (Cuttone et al., 2014)

This data visualization heuristics is one of the few evaluation methods that focus on mobile. Personal Informative, another mobile data visualization experience, has become ubiquitous because of mobile devices’ rise. The rationale for the selection of this method is its focus on mobile and wearable devices.

1. Make Data Interpretable at a Glance — The user must gain answers from the data visualization with minimal effort and time.
2. Enable Exploration of Patterns in Time Series Data- The user should have the ability to explore global trends and periodic trends.
3. Enable Discovery of Trends in Multiple Data Streams- Allow users to analyze or compare multivariate data at a time
4. Turn-Key Metrics into Affordances for Action- Interactions within the visualization should be leveraged to create new insights.

3. Shneiderman’s “Visual Information-Seeking Mantra.” (North et al., 1997)

Shneiderman’s mantra, one of the most widely used data visualization evaluation standards. It is perhaps the starting point for all data visualization for screens. The rationale for its selection is because It focuses on the most basic functionalities and heuristics of a good data visualization.

1. Overview first — The user should be able to gain an overview of the visualization.
2. Zoom and filter — There should be the ability to Zoom in and filter out items of interest.
3. Details on demand — A user should have the ability to select an item or group and get points when needed.
4. Relate -The user should be able to find relationships between items.
5. Extract — Allow the user to extract subcollection or query parameters.

Severity Ranking System

The severity ranking system follows Neilsen’s proposed severity scale. It is defined based on three core factors, the frequency, the impact, and the persistence of the problem. They were combined on a single rating scale between 0–4.

0 — The problem doesn’t seem to be a usability issue

1- A superficial or cosmetic issue that doesn’t require an immediate fix

2 — Minor violation of the heuristic, but a challenge for users. It is of low priority

3 — Major problem which occurs frequently and persistently and is a high priority

4 — Catastrophic issue that needs urgent attention before it gets to a user

Findings and recommendation

Method: Shneiderman’s “Visual Information-Seeking Mantra

Findings 1. Inability to check stock trend beyond five years

Method: Shneiderman’s “Visual Information-Seeking Mantra

Violation: Zoom and filter

Severity: 3

Robinhood is meant to be a straightforward investing app targeting novice investors and millennials. While its simplification makes it easy to use, it misses specific details relevant to understanding a stock’s past performance. The time-series filter is limited to just five years. While this is a common trend in most mobile investment apps, it limits insight a user can gain about a stock’s past performance.

Recommendation 1

To address this, I recommend adding a ’10 years’ or ‘All-time’ performance filter to give users more insights into the stock’s past performance.

Four Data Visualization Heuristics to Facilitate Reflection in Personal Informatics.

Findings 1. Limited dates for data exploration

Violation: Enable Exploration of Patterns in Time Series Data

Severity: 3

While users can explore some data about a specific stock, limiting the date to five years doesn’t help the user understand the stock’s previous performance.

Recommendation 1

To address this, I recommend adding a ’10 years’ or ‘All-time’ performance filter to give users more insights into the stock’s past performance.

Data Visualization Evaluation through the 3 Waves of HCI

Data visualization has been used to represent different kinds of data on other mediums throughout the years. The financial sector mainly uses data visualization in almost all of its dealings since it deals with large amounts of data daily. Its popularity grew along with the computers and digital screens in general. Its popularity can also be associated with innovation in the graphical user interface (GUI). HCI as a field

Data Visualization Evaluation through the 3 Waves of HCI

Data visualization used across multiple industries and mediums has grown over the past decades, even though it has been around for several centuries. This is as a result of the innovation in computers and graphical user interface (GUI). This growth is similar for HCI, which can have data visualization under its umbrella.

Traveling back in time to evaluate data visualization across the three time periods would reveal some of the issues, concepts, theories, and approaches that influenced the HCI we practice today.

First Wave (1970–1980s)

Background — The first wave of HCI was influenced by the foundations of engineering. The focus was on human factors and ergonomics since most early practitioners transitioned from aerospace and mechanical engineering fields. Methods were created to fit strict guidelines in a way to understand how humans processed information from computers. It looked at the limitation of users with computer-based tasks.

Evaluation — Given that the interaction was seen as a man-machine coupling, the evaluation will focus mostly on how the user or viewer can interact with a certain part of the visualization. To evaluate the visualizations around this period, the focus could be on understanding how a user is generally about to interact with a computer’s visualization. Given that quantitative metrics were heavily relied upon at this point, predictive modeling and methods like GOMS would have been used to understand the number of time users spent completing specific tasks. For instance, we can evaluate by comparing the time to completing a Robinhood desktop and mobile app. We can also assess using the two gulfs — execution and evaluation. The gulf of execution involves disparity between the user perception of the needed skill to achieve a task and the actual skill required to accomplish that task. And the gulf of evaluation involves the user’s perception of the system’s state and the exact condition. Data visualization uses markers, text, and color encodings et. c. as the visualization to show the user the task at hand or the system’s current state. Using these two gulfs will allow us to understand how users could navigate instruction within the system to achieve their goals or even assess if they understand the instruction.

Limitation: A general limitation of the methods mentioned above is how they are mostly limited in explaining the underlying reasons why a user can or cannot complete a task. Also, methods like GOMS, which relies on an expert’s initial review of the system to predict the best completion time, fail to consider the years of experience the expert has gained over the years to complete the task.

Second Wave (Mid 1980’s)

Background — The second wave of HCI was arguably the most pivotal of all the waves. HCI had influences from social science and anthropology in an era when personal computers and software were also growing. This was a period where the focus was placed on the user’s mental state and how it processes information.

Evaluation — If this evaluation were to be conducted at the second wave, it would draw from the numerous cognitive theories espoused around these times. This era saw more methods that promoted understanding the context and the user’s mental processes, naturalistic and empirical studies. Theories like distributed cognition, situated cognition, and extended cognition all form part of the theories that emerged and can be drawn to understand how visualization should work. A typical evaluation would be for an expert to observe and review how a user interacted with the visualization. Interviews can be conducted to understand their contextual use of features within the application. Selected experts can also conduct evaluation methods like heuristic evaluation. Heuristic evaluation emerged around this era. It was pioneered by Jacob Nielsen and Rolf Molich in the 1990s. The set of heuristics cuts across a wide range of digital products.

Limitation — An attempt to understand human cognition and the context they operate always opens up different interpretations. While most theories and concepts promoted observing and understanding users from the natural environment, methods like heuristic evaluation, which is not conducted in the users’ setting, do not fully show how an average user would use the application.

Third Wave ( 2000’s — )

Background — The starting from the early 2000s was termed as the third wave of HCI. This period saw a significant influence from critical theory in HCI. It highlights how the field became critical with issues like culture, ethics, human values to guide HCI. Methods like critical design, reflective design, participatory, and co-design became popular.

Evaluation — We would draw from the theories and concepts that emerge in this period to evaluate instead of methods. Questions that arise when conducting an evaluation would broadly focus on how the design supports the users, empowers, and elevates their human value. Questions like, is the design inclusive? Is the user at a detriment for giving away his privacy rights? etc. We might ask how the visualization empowers the user to make the right financial decisions, present misleading information, etc. The outcome of such an evaluation would be a list of critical issues that have to be addressed from a broad approach, starting with how the organization in question views human values.

Limitations — Concepts and theories within the 3rd wave are centers are social issues that most organizations either neglect or don’t equip themselves to answer, which sometimes makes it difficult to pursue and achieve. Contemporary methods like participatory design and co-design can help organizations, employers, and users address these social issues.

Conclusion

The three methods used in this evaluation successfully revealed the design and usability successes and flaws of the Robinhood app. Considering the features within the task and scenario on which the evaluation was conducted, Shneiderman’s “Visual Information-Seeking Mantra and Four Data Visualization Heuristics to Facilitate Reflection in Personal Informatics best identified the severest issues of all. Shneiderman’s mantra, which predates both evaluation and arguably the most fundamental data visualization evaluation method, was convenient for me to evaluate with.

After the evaluation, I view the Ten information visualization heuristics more as a product interface design evaluation method but not necessarily a data visualization method. Overall, these three methods are an excellent combination to evaluate a fintech product like Robinhood.

Reference

1. Cuttone, A., Petersen, M. K., & Larsen, J. E. (2014). Four data visualization heuristics to facilitate reflection in personal informatics. International Conference on Universal Access in Human-Computer Interaction, 541–552.

2. Forsell, C., & Johansson, J. (2010). An heuristic set for evaluation in information visualization. Proceedings of the International Conference on Advanced Visual Interfaces, 199–206.

3. Greenberg, S., & Buxton, B. (2008). Usability evaluation considered harmful (some of the time). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 111–120.

4. Luzzardi, P., Freitas, C., Cava, R., Duarte, G., & Vasconcelos, M. (2004). An extended set of ergonomic criteria for information visualization techniques. Proceedings of the Seventh IASTED International Conference on Computer Graphics And Imaging (Cgim-2004), Kauai, 236–241.

5. Nielsen, J. (2005). Ten usability heuristics. http://www. nngroup. com/articles/ten-usability-heuristics/(acc-essed ….

6. North, C., Shneiderman, B., & Plaisant, C. (1997). Visual Information Seeking in Digital Image Libraries: The Visible Human Explorer. Information in Images.

7. Scapin, D. L., & Bastien, J. C. (1997). Ergonomic criteria for evaluating the ergonomic quality of interactive systems. Behaviour & Information Technology, 16(4–5), 220–231.

8. Webel, B. (2020). Fintech: Overview of innovation financial technology and selected policy issues. In Fintech: Overview of innovation financial technology and selected policy issues ([Library of Congress public edition].). Washington, DC: Congressional Research Service. https://purl.fdlp.gov/GPO/gpo141846