When Perceptual Proxies Attack
Tricking the visual system to understand its inner workings.
Visual data comparison is a crucial step in data visualization. Instead of making these comparisons quantitatively, as a computer would, it is believed that the human visual system uses so-called perceptual proxies: “shortcuts” used to extract the required information from a visual shape. Frequently, these proxies represent straightforward visual properties such as area, outline, color, etc. Gaining useful insights on these proxies can suggest guidelines on ways to optimize a visual graphic to match these proxies in order to minimize the perceptual error. Since these proxies differ depending on the type of visualization, design, data organization, and so on, they must be assessed on a case-by-case basis.
Overview
In a recent IEEE VIS paper, Ondov et al. evaluate the choice of perceptual proxies for two visual comparison tasks — finding the largest mean (MaxMean) and the largest range (MaxRange) between two different bar charts — by performing two experiments. Before we get into the experiments, let’s understand a key terminology: adversarial dataset. These datasets for a particular proxy basically try to deceive the visual system into favoring the incorrect answer, thus revealing how our vision works. For example, consider the “longest bar” proxy for highest mean, where the shortcut is to simply choose the bar chart that has the longest bar as a proxy for the highest mean. An adversarial dataset for “longest bar” would have a longer bar for the chart that has the smaller arithmetic mean. In other words, if the visual system uses that proxy, the answer would be incorrect.
The first experiment in this paper was theory-driven and relied on the belief that we understand perceptual proxies. The second experiment was data-driven, where instead of tuning the datasets, they were randomly perturbed, starting with the same statistical features and using participants’ choices for subsequent lineups to locate data that was progressively more deceptive.
In both tests, horizontal bar charts with equal thickness and the revealing hues orange and blue were used. The two max mean and max range tasks are global in scope, in that they require the participant to survey the entire visualization rather than individual items. For the procedure, the platform briefly displayed a side-by-side lineup of two data series that were represented as bar charts. Following that, participants had to click the icon that represented the bar chart they thought had the bigger mean or range. Let’s dive deep into the experiments now.
Creating adversarial examples
The first experiment started with a list of educated guesses for what proxies our visual system might use, generated adversarial datasets optimized for them using simulated annealing, and then tested these datasets with human participants. The goal of this experiment was to find evidence that participants could be using perceptual proxies in visual comparison tasks, and to understand how participants used different proxies differently. The figure below demonstrates the four proxies selected for each task.
The two hypotheses for Experiment 1 were:
- Adversarially manipulating perceptual proxies will mislead participants to be worse at making a visual comparison; and
- Individuals will be affected by such manipulations differently.
To quantify the effect of proxies, a measurement called titer was used. It essentially scales the task difficulty in different trials for both the bar charts. In simple terms, a larger titer value to identify the correct answer meant that the participants were more likely to be deceived by the adversarial examples, implying their usage of those proxies. Alternatively, smaller titer values showed that participants may not be deceived by authors’ manipulation of proxies.
The analysis was divided into two steps. At a high level, authors applied Bayesian logistic regression to participants’ responses to estimate titer threshold values for each proxy, along with deriving the measurement error of the participants’ thresholds. They utilized these parameters in a Bayesian linear regression to estimate the effects of each proxy on participants’ perceptions. Utilizing the results, the authors found evidence to support the first hypothesis: participants are likely misled by some of the manipulated proxies. This experiment concluded that an average participant might be deceived by the centroid proxy and therefore might be using that proxy to estimate MaxMean. The authors also found that slope and slope range are likely to have smaller titer thresholds, suggesting that an average participant is more likely to select against these two proxies. They also found that, on average, most participants are consistent with themselves across all conditions.
Human-guided optimization
In the above experiment, we had to guess what proxies people might be using. However, what if we just allowed the visual system to show us what an adversarial chart looks like? That’s what the second experiment addresses. It considers the human perception of the summary statistic to be a black-box function that the authors seek to optimize. The two hypotheses for Experiment 2 were:
- Optimized charts will display identifiable characteristics corresponding to the proposed proxies.
- Optimized charts will be adversarial, appearing to have larger summary statistics versus random charts with the same statistics.
Aspects such as tasks, participants, apparatus, and kinds of proxies in Experiment 2 were the same as in Experiment 1, with procedural and evaluation metric differences. This experiment starts with two charts with the identical summary statistic and iteratively builds adversarial examples depending on participant responses. Although further examination of the results of both experiments is needed, they provide complementary evidence that the centroid proxy is at least on the right track.
Takeaways
In conclusion, both experiments had a common goal of determining whether the participants employed perceptual proxies in the two tasks, despite the fact that they tackled the same issue from different perspectives. They provide intriguing evidence that unstructured perceptual response data can be used to observe perceptual proxy theories from vision research. The researchers discovered that even when a virtualization is designed for a specific task, it is still possible for participants to be misled by the data. Nonetheless, this publication establishes a foundation and methodological framework on which future research can be built upon.
References
- Brian Ondov, Fumeng Yang, Matthew Kay, Niklas Elmqvist, Steven Franconeri. Revealing Perceptual Proxies with Adversarial Examples. IEEE Transactions on Visualization & Computer Graphics, 2020.
