A taxonomy of practices emerging at the intersection of AI and design.

Nadia Piet

Follow

May 14, 2020 · 7 min read

Shaping the world around us

The role of a designer is to intentionally shape the world around us with the resources we have available. That world is increasingly shaped by data sets, algorithms, and pre-trained models.

As Artificial Intelligence(AI)/Machine Learning(ML) is finding its way into every industry and area of life, we can no longer afford to limit the development of these systems to the domain of engineering. As people are widely affected, both positively and negatively, we no longer get to excuse ourselves because it’s ‘too’ technical.

You may ignore technological developments and continue your design practice as-is for now. But if you recognize the influence and responsibility we have as designers or are simply curious to learn what AI and design have to do with one another, you came to the right place.

How do AI and design intersect?

While still in its infancy, the intersection of AI/ML and design is beginning to take shape and establish itself as a field. As organizations are rapidly investing in AI/ML, the need for designers to create holistic human-centered systems is urgent. The other day I even came across the first job opening for “AI designer”, and then another one — so yes, I suppose it’s a real thing now.

In trying to structure my thinking around this field, I wanted to create some sort of mental model of how these separately emerging practices fit together.

Right now, I’m noticing 3 ways designers are working with AI/ML:

1. design with AI (human-machine collaborations with creative output)
2. design for AI (integrating human-centered design practices into the AI dev process)
3. design of AI (interaction and UX design for AI systems)

— each of which I’ll discuss in more detail below.

1) Design with AI — human-machine collaborations with creative output. AI as a design tool or partner.

How can AI assist us in the creative / design process? Design with AI is exploring collaboration between human creativity and computational logic to produce creative output. Creatives have begun exploring the abilities of AI to augment their images, videos, text, music, UI, product design, architecture, and any other format that can be rendered into computable data.

This practice seems most relevant for graphic designers, artists, and creatives.

Some examples:

• generative or parametric design in architecture like Autodesk did to design their new Toronto office
• automating tedious design tasks like AirBnB’s tool for digitalizing wireframes or Netflix’s tool for localizing graphics
• logo and brand asset generation tools that have learned from best practices for example from Brandmark
• using machine learning models such as PoseNet and mapping its prediction to visual output like this collaboration by technologist Maya Man and dancer Bill T. Jones
• Generative Adversarial Networks (GANs) to generate images imitating or warping a certain input style. Try the Ganbreeder yourself
• Vera van der Seyp is one of my favorite AI designers working with generative type and graphics
• Sofia Crespo is an AI artist focused on nature & neural nets

Some resources:
algorithms.design — incredible directory of AI x design projects
Runway ML —the easiest way to get started with creative AI
AI Artist’s AI Art Tools — exhaustive collection of creative AI tools
autonomous.design — set of AI-powered design tools

Some thoughts:
Continuing a similar debate around intelligence, how do we define and measure creativity? Who owns, gets to take credit, and sells the art when it’s made in collaboration with an AI, from an often pre-existing dataset, and open-source model? Should we use computers to imitate best practices or incorporate randomness instead?

2) Design for AI — integrating human-centered design practices into the AI development process. AI as a problem-solver.

What, when, and how should we build AI s? Design for AI is about bringing elements and practitioners from the human-centered design approach into the AI development process. Considering user needs and human implications within engineering decisions will enable us to build systems that are considerate of the context they’re intended for. This requires a close and cross-disciplinary collaboration between designers and engineers.

This practice seems most relevant for design consultants, strategists, and researchers.

Some examples:

• Being able to frame user needs as data exploration queries and machine learning problems
• Informing trade-offs between different algorithms and optimization parameters based on the specific use case (e.g. recall vs precision)
• Prototyping the experience through Wizard of Oz methods to validate user value before investing lots of resources
• Interviewing domain experts for an early understanding of feature selection and decision logic

Some resources:
AI meets Design toolkit — my toolkit for designers to work with AI
Lingua Franca — A Design Language for Human-Centered AI

Some thoughts:
What do designers need to learn about AI/ML? What do engineers need to learn about design? How can we make the most of both designers’ and engineers’ practice? How do we communicate and collaborate to design human(ity)-centered AI?

3) Design of AI —shaping the UX, UI, and user interaction with AI systems. AI as a design material.

How will users interact with the AI systems we built? Design of AI is about designing interactions with adaptive, intelligent, and semi-autonomous systems. Every design material comes with unique opportunities and challenges. In the same way that designing an event poster is different from designing a mobile app, designing AI/ML-driven applications is different to designing mobile apps. To create helpful and holistic experiences, this requires designers to acquaint themselves with the materiality of data and AI/ML, and coin smart solutions.

This practice seems most relevant for UI, UX, and interaction designers.

Some examples:

• Leveraging new types of interactions available such as voice interfaces and computer vision for better user experiences
• Designing interfaces to explain to users (with the right amount of detail) how the system works
• Laying out an ongoing onboarding process that makes sense of changes
• Building in the right user feedback mechanisms both implicit and explicit to help your model learn
• Auditing datasets and models for inclusivity and bias
• Anticipating potential unintended consequences

Some resources:
AI-Driven Design Ebook #4 — Design Challenges in Machine Learning Products — my ebook with AWWWARDS & Adyen about some of the unique design challenges

Some thoughts:
What are helpful mental models for users to have around understanding AI/ML-systems? How do we materialize high-level principles like explainability and user feedback loop into (post-pixel) interfaces? How can we build responsibility and ethics audits into every step of the development process to anticipate consequences and ensure inclusivity?

*) AI x Design — misc.

There are plenty more practices and design challenges surrounding AI/ML. While these may not fit into the categories above, they’re very relevant to us designers and this post wouldn’t be complete without mentioning them.

• Data design
  On the periphery, we catch one line of work that is less about AI/ML and more about the raw material it feeds off: data. This includes data visualization, data design, information design, even database design.
• GUI AI applications
  Building applications where people can view, train, adapt, evaluate, and deploy machine learning models through Graphical User Interfaces instead of code. For example Teachable Machine, autoIBM.
• Data literacy & AI edu
  Developing accessible educational content and tools to improve data literacy and make AI/ML understandable to a wider audience
• … tell me what else is emerging in the comments!

A taxonomy to think and talk about AI x Design

A very young field, this is a first attempt at building some sort of taxonomy to organize our thinking and talking about this topic.

It’s by no means complete or decided — rather, it’s here for you to argue and add onto. I’d love to hear from you with more examples, new categories, and alternate mental models.

Join our community of AI x Design geeks

AI and design have a lot more to do with each other than a first glance may suggest. As a development that is so influential, it’s of crucial importance more designers, social scientists, and everything in between, get involved alongside the engineers building these systems.

That includes you! If you’re feeling in any way triggered, consider this a call. Get lost in some of the resources, ask your engineers if you can help, run off to make some weird AI art. Whatever it is, take that first step.

If you’re (aspire to be) working in the field of AI x Design way, I invite you to join our community of AI x Design practitioners on Slack.

There’s so much to be done and this is just the beginning. I’m excited and honored to see it all unfold and contribute where I can.

I’d love to hear if you learned anything new, and what (if any) next steps you’re planning to take!

TL:DR

There are new practices emerging at the intersection of AI and design. For now, they seem to fall in 3 categories: 1) design with AI (creative human-AI collaboration), 2) design for AI (human-centered design for the AI dev process) and, 3) design of AI (interaction and UX design for AI systems). With plenty more practices popping up at the periphery, this young but rapidly growing field offers exciting opportunities for designers to get involved and make an impact.