Support my work by wishlisting Compicactus on Steam
Inspiration
When I was a kid I was fascinated by chatbots, the idea of being able to have a conversation with a machine sparked my imagination.
At a young age I typed line by line the ELIZA code, copying it from a computer magazine to a CZ Spectrum. I was not very impressed by the result.
The first time I felt like I was actually conversing with a computer was, oddly enough, when configuring software for the Microsoft DOS operating system, it looked something like this:
I don’t remember what the name of the program was, but I perfectly remember the feeling of talking to the machine. I remember suspecting that the machine understood me. All this without the need for me to type words on the keyboard.
The program commented on my choices, I responded, and together we got the configuration done. I never felt that connection with a program again.
In order to recreate this experience, I designed a new way of interacting with the machines.
Current conversational UI does not work
Despite advances in processing speed and connectivity, users of intelligent automated assistants are limited when it comes to entering commands to provide information, execute actions, create custom routines and perform complex queries.
Three areas have advanced technologically: the conversion of sound signals to text, the identification of intent in text, and the recognition of entities in text. Although progress in the first area has been remarkable, the remaining two areas still pose great difficulties, largely due to the inability of modern deep learning models to work at the semantic level efficiently.
Some limitations in natural language processing are: understanding what is the real need of the user when an expression has more than one interpretation, and the acquisition of information poses serious problems due to the large number of lexical units of natural languages.
At the same time, the use of natural language analysis systems through deep learning models requires the collection of large amounts of data and its subsequent processing in corporate servers, out of the user’s control, resulting in most cases in a detriment to the user’s privacy and security, and in a limitation in the quality of personal data that the interface can handle.
These limitations result in an interface that does not contextualize queries, does not store a sufficiently complex internal model of the user and his/her interests, and does not allow the user to customize and program it according to his/her needs. This results in a poor user experience. The user is limited if the interface only allows him to select predetermined commands, or frustrated when he or she is free to articulate via text or voice an unlimited number of commands, only to discover that most result in an error message.
An unconventional solution
There are two ways to solve the problem of communication between humans and computer programs.
- By making the programs understand natural language. This is not yet possible.
- By making humans communicate using formal language (programming code). Something that not everyone can do.
I propose a third solution: Apply user interface design to enable communication that looks like natural language to humans, and looks like formal language to the computer program.
Inspired by the idea of finding this middle ground I developed a new user interface element. A menu that allows you to enter commands to a program without having to type them in full.
The user selects tokens (concepts / words) from a list, and the program suggests the commands that best make use of those tokens. Enabling collaborative communication.
Token Menu
The following proposal solves these limitations by departing from the path of pattern recognition and natural language analysis. Instead, it proposes the use of formal language commands accompanied by a user interface based on the selection of sequences of lexical components that help the user to program these commands in the simplest and most intuitive way possible.
This allows the user to enter more elaborate queries and even program their own routines. By not using expensive natural language processing algorithms all information can be kept on the user’s device, protecting their privacy and allowing them to store and manipulate sensitive information securely.
The Token Menu is able to present the user with a list of commands based on the user’s selection of one or more tokens, allowing the user to enter commands in formal language without the need to know its syntax and without the need to use natural language. Avoiding its problems and limitations.
This significantly expands the range of actions that the interface can allow the user to perform, while preventing the user from selecting a command that results in an error.
The Token Menu allows the user to execute a command without having to type it word for word, without having to fill out a form, select it from a predetermined list of options and without having to use natural language.
The user can traverse the command space more easily, without having to learn a particular syntax, without being constrained by the structure of a form or list, and without being forced to memorize the correct phrases to pronounce. Moreover, it allows the user to discover new commands while searching for known commands.
Examples
You can check this video of a demo made in Godot here.
This is a virtual version of Elon Musk to which you can ask different queries about Tesla, SpaceX and Artificial Intelligence. You can test it at here.
If you have any questions or feedback about the project I invite you to follow and DM me on Twitter: https://twitter.com/EibrielBot
Eibriel
eibriel.eth.limo
