As a result, designers need to understand AI and ML as a material to explore and design with. At Adobe, we design tools that amplify the world’s ability to create and communicate. And ML is already playing a huge role in enabling and enhancing that.

Design Lab at Adobe is a team that experiments with emerging technologies, and AI and ML have been on our radars since a few years ago. Having realized its potential, and quirks too, we wanted to introduce it to our fellow designers at the Adobe Design team. We see value in designers building a pragmatic understanding of what AI enables — deeper than thinking of it as a unicorn that can magically do anything — thereby enabling them to have meaningful conversations about design possibilities through AI. We did that with a series of workshops that we affectionately called ML4D.

As with other technologies like Natural User Interfaces, Intelligent Agents, Mixed Reality, and more that we experiment with at the Design Lab, we want designers to see Machine Learning as a material to explore and design with.

There are some excellent resources online that elaborate upon it from a technical perspective. For designers however, our approach has been to strike a balance between not getting lost in too much of technical detail while providing the right amount of depth to enable explorations.

Having done a dipstick test with designers in our team, we realized that everyone wanted to understand it at a different level of detail. So we decided to roll out the program in three phases — Excite, Play, and Build.

Excite

This phase was structured as a two hour session that provided a basic understanding of the domain. We started with an introduction to how ML differs from traditional computer programming. We touched upon how AI is about simulating human intelligence — a concept broader than ML — and ML is a technique that enables computers to learn from data as opposed to being fed explicit instructions.

We used our favorite examples, from very practical use cases, through applications in art, going up to some very wondrous and even conceptual ones. These are examples that got us excited about the field and we introduced them in a manner that shows the evolution of this field and possible directions that it could take.

No introduction to ML should be complete without a discussion on ethics and its implications on our culture. Through examples and anecdotes, we demonstrated how personal biases in choosing or leaving out training data affects the outcomes, and could have disastrous consequences for a brand.

For designers and creatives, ML significantly changes the dynamics of their relationship with their tools. On one hand, it allows the computer to develop a deep symbiotic relationship with the creator. On the other, it raises questions on creative control, hinting towards an inversion of the relationship a creative has with their tools. Our discussions on this extended beyond the 2 hour session and still pop up at our coffee breaks.

Data — Train — Apply

ML and AI are often thought to be blackboxes that can take in anything, and give us directly usable results. While it is true in a lot of cases, we wanted to take a step back with the participants to think about the steps involved in creating a trained model in the first place, which would enable richer understanding in how to tackle new use cases. Another value in this is the ability to then combine traditional processes that designers might already be aware of throughout the cycle.

For instance, Adobe Photoshop is not one huge ML model, but is composed of different components, some driven by AI and ML, and others by traditional algorithms, which work together to allow creative flexibility depending on what the user needs.

We wanted to capture this process in a general recipe of sorts, such as to provide an easy to apply universal framework in thinking of creating new models. Thus, we introduced the three step cycle of getting data, training a model, and applying the results of the model as a data-train-apply framework.

Building Blocks

Our intention for this series of workshops has been to enable designers to think with ML as a material. This approach of thinking of technology as a material guides us to identify, and design with its properties. It frees us to think beyond available examples and use it creatively.

We also acknowledge the fact that there are multiple algorithms that let you perform multiple nuanced actions. Keeping track of them can get overwhelming very quickly. Therefore, instead of introducing specific algorithms, we introduce them through cards that we see as Building Blocks.

These blocks refer to algorithms grouped by what they do, rather than their specific underlying technicalities. That way, they are easier to remember and think with.

These blocks are “If this then that”, “This or that”, “Make like this”, “Turn this into that” and “Group similar stuff”. While we realize that there would be cases that wouldn’t neatly fit into this categorization, our aim was to provide an approachable framework that allows thinking with ML for people with no prior experience. This set of blocks could even grow in number in the future when newer classes of algorithms are developed.

Given this framework, people could now start framing problems they are trying to solve with one of or a combination of these blocks. For instance, if someone was looking to convert their sketches into paintings, they could look into the “turn this into that” category for answers. Then, depending on the input and output data types, they could pick a specific algorithm.

We introduced these building blocks through a bunch of examples that represented them. We broke these example down into the data-train-apply cycle, combined with one or more of the building blocks, to start developing a sense of how to think with these frameworks. The examples were drawn from creative projects and applications, something our audience would easily relate with.

The blocks are slightly abstract by design. That allows for thinking beyond the examples we shared. It also enables imagination of future possibilities, and creative use cases afforded by changing any part of the data-train-apply cycle.

Here’s a quick explanation of the building blocks.

Play

With the abstract thinking concepts in place — get data, train, apply and the building blocks, and having discussed a bunch of exciting projects from these lenses, we were ready to venture into the ‘play’ phase of our ML4D workshop. The play phase was essentially to start looking at how to apply these concepts into practice. For this, we chose resources that are openly available and are extremely friendly to begin with.

Google’s Teachable Machine

The first session of the play phase was with Teachable Machine (TM). It is a web-based tool created by Google which makes creating machine learning models fast, easy, and accessible to everyone. Currently it allows for creating this or that(classification) models only using images (from webcam or otherwise), sounds and poses. What’s great about this tool is that it enables playing with the data-train-apply cycle in a transparent, approachable, step-by-step manner. It highlights various nuances about each step of the process.

In the data collection step we showed that a good dataset should have enough variety in the samples and contain representations of different orientations of the object we are trying to recognize. To help the model distinguish between classes better, it is also necessary to evenly spread the number of training examples per class, ensuring that they contain sufficient samples with ambiguous and overlapping features.

By showing biased datasets captured using the webcam, we also discussed how bias creeps into ML implementations, and how we can avoid it by being aware of it in the first place.

The training step also enabled us to introduce the concept of transfer learning, which TM uses for training. This discussion enabled us to elucidate aspects like layers and weights of a neural network. We were able to discuss various hyper-parameters exposed by TM. This brought forth the notion that model creation is like another craft at its core, and varying these aspects enables vastly individualistic expression using the same underlying algorithms. TM then makes applying the trained model very simple too, using some easy to play with platforms like p5JS.

As an end to end example of using TM, we created a game wherein 2 teams had to move a ball on the screen towards a goal — lots of excited shouting was involved!

In the second half, we enabled designers to experiment with readily available ML models. Our key focus was to show ways of playing around with other building blocks which we had discussed earlier.

We capped the Play phase with a summary of the concepts and frameworks that we had introduced. It was exciting to see our designers come up with use cases for ML in their own products, and even personal art practices. Some of them made a game, and some made music. One of our fellow designers from the Design Lab, Prabhat, even trained a GAN on his own paintings.

While we wait for the next phase of this series, it felt great to see ML being used as a playful design material that allows new forms of expressions and explorations. We had also conducted an abridged version of this workshop at an external conference, where we got similarly exciting vibes.

—

This post was authored by Harshit Agrawal and Nikhil Tailang from the Design Lab. Harshit has exhibited his art created using ML at various galleries. Nikhil has recently been baking bread based on recipes generated using ML. Reach out to us at designlab(at)adobe(dot)com.

For a more formal introduction about us — Design Lab is a team at Adobe that explores emerging technologies and their potential to enable new opportunities in creative thinking and expression. We design tools that amplify the world’s ability to create and communicate.

We often work through design-led research, speculative design projects, art installations, and creative workshops, to enable experiences between people and digital systems — as well as new forms of creative expression.

At an über level, we see technologies as design materials, and we work by understanding their properties, and sometimes even imagining new properties for these materials. The technologies that we explore are situated in a fluid and emergent landscape, and therefore our processes constantly evolve in response to the materials that we are designing with.

We primarily call ourselves designers, but we are trans-disciplinary, and are always ready to take on new roles. Our team currently comprises of new media, interaction and product designers, creative coders, artists, researchers, and design managers.

Who we are

Though we love thinking about artificial sentience, we’ve not reached there quite yet at our lab, and there are real people in the Design Lab team.

Mallika Yelandur is a Program Manager, who drives strategy and outreach for Design Lab. She enjoys bringing definition and process to transform an idea into a value-creating innovation. Equally at ease with left- and right-brained thinking, when she’s not donning the program manager’s hat, she loves collaborating with the team on intense brain-storming sessions and exploring the future of creative tools. When not at work, true to Da Vinci’s quote, she loves creating pieces of art that are never finished, only abandoned.

Prabhat Mahapatra has the title of Design Manager at Design Lab on his visiting card, but instead prefers to be an intern learning from all the awesome folks he works with. He is usually found sketching away his thoughts in his sketchbook or doodling in his head while attending meetings. He loves dabbling in drawing and painting, and through various projects at Design Lab, dreams of building a future where anyone can express themselves using the medium of their choice — be it through art, storytelling, music, dance, or just via their thoughts (why not)! He has a formal background in Industrial Design and has worked in the field of product, graphics and space design, ran his own design studio, taught (and still does when time permits) at design schools, before joining Adobe to look into user experience design for one of industry’s best vector illustration tool, and ultimately co-founding Design Lab. 15 years of making design his playground, his current area of interest is in exploring the future of creative tools of expression using emerging technologies and speculative design processes, and making many many more drawings.

Harshit Agrawal is the latest addition to Design Lab. He comes from a mixed background of design and technology, and loves to work across disciplines to imagine new ways of creative expression. He dabbles between being a tool maker and an artist, and is always excited learning about and experimenting with various kinds of emerging technologies- using them as a means to expand notions of creativity and critically think of their implications to society at large. He likes crafting experiences that bring people closer to various technologies in a fun and engaging manner. At the Design Lab, he’s exploring the rapidly evolving world of machine learning from a creative expression lens.

Nikhil Tailang is one of the founding members of Design Lab. He is a designer and a maker, and constantly seeks new ways to prototype experiences. At work, what excites him is the role of intelligent machines in shaping our tools for creativity, and the ensuing dynamics between the creator and the tool. He finds joy in uncovering underlying structures in complex systems. He is always seeking out his next hobby, but his persistent interests are computer programming, hacking plants, baking breads, and aeroplanes. He likes to observe people and is always up for learning something new. He wishes for a world where people develop a deeply symbiotic relationship with technology. At Design Lab, he works on Natural User Interfaces and Computational Creativity.

Reach out to us at designlab at adobe dot com. We’re always excited to hear new perspectives on how technology will change the way we express ourselves and work.