How to draw pictures: Style

Computational Models of Representational Art, Part 3

In two previous blog posts, I described simple models for drawing. So far, these drawings were relatively plain. But, we can expand on the basic techniques to get a really broad and expressive range of artistic rendering styles.

Stroke style

As already shown, we can use different styles of outline strokes, which includes both drawing brush textures, as well as changing the shapes of the outlines. We can even animate them. Here’s an example of applying different kinds of strokes to animations of 3D models:

Animated 3D models with different styles of strokes (skip ahead to 4:15 in the video).

Shading and Hatching

Of course, we might draw inside the shapes. The most popular and well-known non-photorealistic shading algorithm is toon shading, which is widely used in games and animations:

Toon shading in the classic video game Jet Set Radio. Many games and animations use toon shading to give the look of traditional cel animation.

But there are lots of other ways to draw objects. For example, hatching is widely used in pencil and pen illustration. Here are the results of an algorithm that we developed to draw hatching illustrations:

3D hatching, automatically generated from the 3D model on the left.

In order to hatch a surface, we need to determine the hatching directions, as well as their density. In the above illustration, the density is based on a shading: darker regions have cross-hatching. To determine the directions, we notice from looking at real drawings that people often draw lines along the directions in which the surface curves the most and the least (formally, these are called the principal curvature directions). Perceptual psychologists have also found that people tend to perceive hatch lines as curvature directions.

We can also let a user author the rendering style. Here is an animation method that applies the Image Analogies approach to 3D animation:

There are lots of other ways to draw objects, whether watercolor, oil; more or less “impressionistic,” and so on.

Most computer animation and games that build on these techniques are essentially creating new visual artistic styles, mashing-up conventional line drawing and painting with computer graphics algorithms. An example is the lovely Paperman animation, which combines contour tracking and toon shading—driven by an enormous amount of authoring and creativity from by professional animators:

Pictorial design

All of these rendering styles so far are still relatively “rigid:” they carefully draw each line exactly where linear perspective projection says it should be. However, artists draw in many ways that play with conventional linear perspective, or ignore it all together. Even in seemingly-realistic images, artists often manipulate perspective to improve images and to improve portraits. Beyond that, they sometimes play with perspective to exaggerate an effect. Here is a 1914 painting by Giorgio de Chirico, which uses multiple perspectives to give a sense of dissonance and melancholy, and a 3D rendering inspired by it:

Left: 1914 painting by Giorgio de Chirico. Right: 3D rendering by Maneesh Agrawala et al.

The 3D rendering on the right uses different camera projections for the three separate objects, violating conventional linear perspective.

Many other elements of drawing and rendering depend on their appearance in the image. For example, this method for Dr. Seuss-style rendering adapts the density of image elements to image space, and makes sure they always face the viewer:

Dr. Seuss-style rendering (on right) based on the 3D model on the left, by Kowalski et al.

Here is a video of that system in action (albeit low-resolution, and suffering from video transfer artifacts).

Defining Style and the Science of Art

The main motivation for this research has been to create new tools for artists. However, the algorithms I’ve described in these blog posts also provide a generative theory for representational art. The more different styles they can create, starting from a relatively small set of concepts — such as contours, stroke styles, and shading styles—the more they could potentially provide a scientific description of how we draw and paint. Furthermore, these building blocks of strokes, shading, and pictorial styles provide a useful way to categorize and understand artistic style.

Despite all these successes so far, there remain many open research problems. How do we programmatically define the space of styles, and there are still significant new technologies needed? What user interfaces can we use to author styles? Can we combine these techniques with machine learning and style transfer?

For a longer discussion of how these models can help create a science of art, see my NPAR 2004 paper, and, as well, for how they can help us classify and define style, see this 2005 paper by Willats and Durand.

These generative models are only half of the whole story. To really understand the science of art, we also need to study the visual perception of art. I’ll discuss this topic in a future series of blog posts.