Imagining Doubtful Creative Tools
Thoughts on the intersection between doubt and Machine Learning
Creativity is fuzzy and wobbly, why aren’t creative tools?
Creatives constantly seek new inspirations and tools to push the boundaries of what can be made with their digital tools. Machine learning features can further this boundary-pushing by leveraging the role of doubt and uncertainty in the creative process.
Machine learning uses fuzzy logic that’s similar to human logic and involves confidence scores, numbers that qualify the machine’s confidence of its output based on its probability. While a high confidence score is important in many circumstances — you wouldn’t ship an auto-transcription feature that gets 50 percent of the words wrong — when it comes to the creative process, the concept of what is “right” and “wrong” is subjective.
Limitations, experimentation and stream-of-consciousness thinking are all valuable parts of the creative process. It might be the inexplicable loose connection, cultural relevance, or some obscure failure of a machine learning feature that inspires the user. We don’t need to strive for an unachievable perfect accuracy before introducing machine learning features into everyday creative tools.
After all, neural networks were inspired by neural connections in the human brain, first conceived in Alan Turing’s 1948 report of the concept he called “Unorganized Machine.”
Below are images generated by DeepMind’s BigGAN. The first image is the algorithm’s neutral state without being assigned any category, and each image afterward is assigned an increased understanding of goldfish (0.0–1.0). The image in the middle is poetically represented as a glowing, underwater bubble. We do not know why this is, but the results are stimulating.
What if creative tools leaned into the fuzzy and often unexplainable nature of our creativity and intents?
What would it be like if creative tools were more similar to our brains, and considered our doubts, emotions and fluidity?
Bugs and Failure as Inspiration
Using limitations to create process-oriented art has a long history before computers (e.g. Jackson Pollock and John Cage). Since computers and televisions were invented, entire genres like glitch art and pixel art have emerged that turned technological limitations into aesthetic playgrounds.
In Magnet TV (1965), artist Nam June Paik used magnetic interference to create an abstract pixel sculpture inside a TV by altering its reception. Computational artists have been inspired by failures of computer programs and made art with them for decades. Like Paik, they do so by observing the technology with its limitations, often divorced from its intended purpose — they play with computation as a direct medium.
In the live performance discrete figures (2018) by Rhizomatiks Research and dance company ELEVENPLAY, artist Kyle McDonald used a seq2seq model with dance pose data to train a generative dancer but chose to cut the learning short and keep the generated dance glitchy and uncanny. In a powerful moment in the middle of the performance, the human dancer starts to imitate the half-learned dancer projected in an AR dance duet. The artists describe the project as a “quest for a new palette of movement to foster undiscovered modes of expressive dance that transcend the limits of conventional human subjectivity and emotional expression.” By leaning into an imperfectly trained model, they did so quite successfully.
As creative machine learning is democratized, users can probe at the algorithms and analyze their opaqueness and inaccuracy, interacting with the algorithm on a deeper level and opening up new channels of experimentation and creativity.
Doubt and User Experience
So, how do we begin to design doubt in machine learning into creative tools?
Today, users often come to their computers with uncertainty, and user experiences have evolved to accommodate it. We use Google’s “I’m Feeling Lucky” to learn about something we might have never searched, random name generators to help us come up with unique usernames, and smarter font matching or color palette generators to quickly explore design options. There is a lack of certainty, but because we expect a potentially unexpected or random outcome, the experience is pleasant and rewarding.
We can design for creative machine learning features by setting the right expectations and building robust tools that express doubt and value exploration and fun.
We should consider how the interface affects user expectations and directs actions. For example, the size of the search bar affects the type of search terms entered, the fixed size and rectangular shape of the window frames might convey that the tool is a static and contained piece of software, and a standard “confirm” button on a learned, context-dependent feature might make users expect a robust, personalized result.
Rather than expecting both the tool and user to be certain about their abilities and intents before taking an action, what if we communicate doubt via language and interface aesthetics to build an enriching and mutual creative environment?
- Express doubt through language
Imagine a feature that generates semantic brushes based on what is in a photograph. The feature uses a semantic understanding of the image to see that it includes autumn trees and suggests a photorealistic tree brush to paint with. The brush could have a refinement slider that allows the user to move between unrelated brushes (e.g. skyscrapers and jet planes) and perfectly related brushes (e.g. more autumn trees). This feature would use a 0–100 percent confidence score to indicate its confidence that the suggested output is related to the existing image, but using the technical term “confidence” assumes that the user wants realistic suggestions. Using more open-ended, neutral language like “related” and “unrelated” or even visual representations of the recommendations expands the space for exploration.
Aside from receiving uncertainty from the machine, we might also consider how users can communicate doubt to the machine. Allowing users to communicate their own doubts in the creative process in general can make a tool more accessible. Imagine if a user could program a tool to understand that when they say “bleh,” it means they are uncertain about the current results, prompting the machine to make broader or more surprising suggestions. They might also be able to dictate how precise they need the outputs to be for a particular goal.
A more mutual and communicative relationship can be fostered by leveraging ML alongside machine and user doubt.
2. Express doubt through interface
Kai’s Power Tools from 1992 had unconventional looks and names (Explorer, Goo, Photo Soap, Power Show) that brought users into a surreal world of image editing. It provided an experimental and playful space where your mind could be messy and your digital creative workspace is littered with colors and textures.
The style of a user interface helps users build a mental model of the tool and grasp the tool’s purpose. Our mental models have evolved as interface styles developed throughout the years, from skeuomorphic buttons and shadows to today’s flat, geometric, hyper-productive UI. Not suggesting that it’s time to bring back skeuomorphic interfaces, but the visual aesthetics of an interface must be thoroughly considered when it comes to building intelligent functionality into creative tools.
Imagine a wobbly button that indicates uncertainty, or a window that isn’t rectangular. What if users could specify the shapes of their tools the same way they can choose dark mode and different color schemes?
Instead of confirming or canceling an action, what if users were unsure about their intent during the exploration phase of a project? Instead of using workarounds like manually pasting variations, going through undo-redo cycles, or saving each iteration individually, an intelligent creative tool could be designed from the start to embrace both machine and user uncertainty, expanding the boundaries of the digital creative tool.
3. Express doubt by default
AI Pioneer Stuart Russell is working on redefining what he calls “human-compatible AI,” or AI that embraces human uncertainty. He works specifically with robots, but his ideas translate to machine learned creative algorithms.
His human-compatible AI follow three rules:
1. The robot’s objective is to maximize realization of human values.
2. The robot does not know what those values are.
3. Human behavior provides information about human values.
Russell suggests that human-compatible AI should be uncertain because humans are uncertain. Its values are defined as the human and machine seek to understand each other better. Imagine a creative tool that did the same.
The recent uptick in creative machine learning gives toolmakers many opportunities to explore and innovate. Recognizing the tool as an extension of the user, the “unorganized machine” provides a provoking companion to the fuzzy, intuitive thinking that is a natural part of the creative process. As intelligent creative tools develop, their designers should lean into the uncertainty, rather than shy away from it.
Thanks to Lisa Jamhoury, Patrick Hebron, Archie Bagnall.
Machine Intelligence Design is a team at Adobe Design promoting a user-centered approach to machine learning and artificial intelligence in creative tools, while simultaneously working to establish deep understanding of how these technologies are changing the ways we interact and create.
We are hiring!
