Assisted Drawing

Exploring Augmented Creativity

4 min readDec 20, 2015

Seeing drawings that were made over 20'000 years ago has a haunting effect. Human history has been profoundly shaped by our ability to create visual content - assisted by tools. This experiment and report explore a new generation of creative tools, that help us draw.

Drawing is one of the oldest forms of human expression. According to Wikipedia it is a “visual art in which a person uses various instruments to mark paper or another medium”. Drawing helps us to communicate. The history of drawing is intertwined with the evolution of tools. From Rocks to Photoshop: Tools help us to change perspective and draw differently.

Experiment: Classifying Drawings

This experiment started with an exploration of the research project “How Do Humans Sketch Objects?” by Mathias Eitz, James Hays and Marc Alexa. The researchers collected 20,000 sketches across 250 categories (741h drawing time) and build a system that classifies drawings in realtime. For the following video, I asked a professional Illustrator to test the system:

A primary aim of this experiment was to explore how humans draw differently, when a machine continuously is trying to guess what is being drawn. To intensifying the user/machine feedback-loop, a Text2Speech system was added. Illustrator: KedamaMi - Music: Oscar Woods.

Entertaining side-note: Youtube’s A.Is classified this video as “copyright infringing in over 150 countries”, due to the used song. It was composed in 1937 by Oscar Woods and is now Public Domain, according to Archive.org.

Experiment: Memorability Prediction

This experiment started with an exploration of the research project “Understanding and Predicting Image Memorability at a Large Scale” by Researchers from MIT’s Artificial Intelligence Laboratory (CSAIL). Their system can predict how memorable or forgettable an image is.

As the current system is trained on photos, performance on drawings is poor. You can try it out with your own photos, by using their online app. This experiment serves as pointer to a future, where machine assistants can tweak drawings (& photos) on-the-fly, to make them more memorable.

Research

Augmenting human skills with machines has a long history. Our ability to draw has continuously been expanded with new techniques. While we are still feeling the aftershocks of tools like Photoshop & AutoCad, A.I assisted drawing tools are emerging that radically redefine creative processes.

Research in “Assisted Drawing” technology dates back to the early 1960ties, where Douglas C. Engelbart famously called for the Augmentation of the Human Intellect. In recent years, research has substantially accelerated: A wide range of experiments are taking place in machine learning, human computer interaction, psychology, design and art.

20 Assisted Drawing Examples

The following is a collection of 20 Assisted Drawing Projects. It is in no way complete and aims simply at sparking a debate about vision & opportunity.

01. Freehand Drawing with Realtime Guidance

02. Reinforcement Learning Drawing Agents

03. Assisted Creativity for Novices

04. Style Transfer

05. Exploring Latent Spaces

06. 3D painting simulation at the bristle level

07. Autocomplete hand-drawn animations

08. Computer Assisted Animation of Line and Paint

09. Animating drawings with face recognition

10. Drawing in Virtual Reality

11. Drawing with Games

12. Handwriting Beautification

13. Robotic Handwriting Assistance

14. Generating Handwriting

15. Live stream of Drawing Robot controlled by Users.

16. Drawing Physical Stuff

17. Memorability Prediction

18. Adaptive Drawing

19. Live stream of Human Drawing in-front of 3 million people.

20. Drawing with Elephants

Thanks to @zachlieberman @graphific @genekogan @hardmaru @kastnerkyle @alexjc @udibr @alecrad @worrydream @KedamaMi @nervous_system @inconvergent @auriea and many more for inspiration!

Final Thoughts

Le Corbusier once said: “I prefer drawing to talking. Drawing is faster, and leaves less room for lies”. The advent of powerful assisted drawing tools is enabling us to draw very differently and explore yet unseen perspectives. The implications of A.Is augmenting human creativity are profound & exiting!