With more and more technologies attempting to create better interfaces by emulating our day-to-day interactions, it won’t be long before we have complete hands-free interactions with technology. Bowen Yu and his coauthors at NYU developed a system that takes questions in the form of everyday speech and turns them into data visualizations and representations.
Although we have a variety of tools for visualizing data, most of them need you to either explore the interface and find out how to get what you want, or you must turn your thoughts into a language that the system understands which then offers what you want. In instances when we’re brainstorming and analyzing data, however, these processes tend to become hurdles. This study, on the other hand, solves the problem by giving you the freedom to speak your mind while the system visualizes your query for you.
This study’s natural language interface is based on VisFlow, a dataflow visualization system (DFVS). DFVS is a general-purpose visualization toolkit that allows the user to map a dataflow diagram that consists of system modules that process and visualize data. When an analyst wants to analyze the information, all they have to do is type out what they want to see and how they want to see it.
For instance, the researchers provide an example where a natural language query is made to visualize certain attributes in a dataset pertaining to cars.
“ Visualize mpg, horsepower, and origin of the selected cars from My Chart in a parallel coordinates plot ”
When the above query is typed in, the system starts to analyze and pick up cues from it.
The system then pulls out the “special utterances,” which are basically nouns and verbs that assist the system to comprehend what the user is trying to visualize and how they want to visualize it at the first level. The system then converts the tagged grammar into functions and does additional analysis to see if it has all of the information it requires to execute a visual representation.
Why is this hard? Natural language is extremely dynamic and varies greatly from person to person. When people are under a larger cognitive load and trying to communicate their views, this is amplified. As a result, the system must frequently make decisions on its own as to what the user is attempting to look for while parsing the query. The researchers term it the parsing ambiguity, and they solve it by constructing a set of rules that direct the system’s decision-making and learning over time.
What makes FlowSense stand out is its well executed user experience, which reduces the learning curve whilst simultaneously teaching the analyst on how to become a power user. They implemented this in two folds
1: Real time highlight of the special utterances in a query — What this does it that it educates the user as to how the system runs in the backend and also get a realtime feedback as to how their query is being perceived.
2: Auto Completion — When brainstorming and analyzing data, since the brain is already under a high cognitive load, users might not be able to maintain multiple trains of thoughts. However, if the system itself provides sample queries, then it might spark a new idea in the user’s head which will open more tangents to explore.
Currently, the system only works on dataflow diagram editing. In the future, Bowen Yu and their team want to improve the system where it could answer complex analytical problems which results in a series of visualizations and an observation.
References
- Yu, B., & Silva, C. T. (2019). FlowSense: A natural language interface for visual data exploration within a dataflow system. IEEE Transactions on Visualization and Computer Graphics, 26(1), 1–11.
- Conference Presentation : https://vimeo.com/368702648
