Drawing With Sound

Authored by Josh Miele (Smith Kettlewell, @berkeleyblink) and Claire Kearney-Volpe (NYU)

Sound Canvas is a tool that invites users to draw through sight, sound, and gesture. Like each of the Creatability Experiments, this project prioritizes accessibility and universal design by accepting multiple modes of input and output. We are particularly excited about Sound Canvas, because it is intended to support blind and visually-impaired users in drawing through a meaningful system of non-visual feedback. In addition, the Sound Canvas project is the result of user-centered practices, with blind and visually-impaired design, usability, and product-testing experts included at all stages of iterative development.

The Basics/Key Features:

Drawings in Sound Canvas can be made with a keyboard, mouse, or your body, and they can be rendered visually, as well as through an original sonification scheme. Here are some of the key features explained:

Multiple Inputs: In order to be inclusive of a range of users with a range of ability, the experiment accepts multiple modes of input. You can draw with your mouse or trackpad, arrow keys, or map to a point on your body, like your nose.

Sonified Canvas: Drawings are translated into sound in real-time — using a special sonification system (see below). Sounds pan from left-to-right as you draw a line from left to right, and the pitch of the sound goes from high to low as you draw a line from the top to bottom of the canvas.

Exploring with Sound: When your pen is up, you can explore your drawing using horizontal and vertical sonic scrubbers (left/right and up/down arrows). As the arrow keys are used to scrub horizontally, not only does the position of the sound shift using stereo location, but the sound simultaneously represents all the pixels in the column. Since all the pixel tones are played together, the user gets a quick overview of where drawing has already taken place. In addition to moving horizontally, the vertical arrow keys can be used to move up and down the column to hear the vertical location of individual pixels with their associated pitch.

Input/Go-To Position: We included a feature that allows you to go to a specific position on the canvas. You can use the key command Shift + G to move the pen or exploration position to a specific location relative to the top-left corner of the canvas.

Key Commands: The inclusion of keyboard commands for interacting with important UI elements is another basic aspect of universal design. In addition to Shift + G to move to a specific location in the drawing area, Sound Canvas includes the following keyboard commands:

• Mute: Shift + M
• Pause: Shift + P
• Focus Canvas: Shift + C
• Pen down: Shift + U
• Position: Shift + G
• Keyboard Shortcuts: Shift + k

Try it out!

In Sound Canvas, use an input of your choice to draw horizontal, vertical, and angled lines. Once you are comfortable with this, try writing your name! When you are done, you can explore the shape of letters and whitespace in between them by toggling the pen up (Shift + U) and using the arrow keys.

Background and Design Considerations:

Blind people interact with computers using a screen reader — a type of application that provides keyboard commands to move the cursor around the screen, and which simultaneously provides non-visual output of the screen contents using text to speech or refreshable braille on a dynamic display. People with limited vision may also benefit from software screen magnifiers which often include a variety of helpful features such as color inversion and cursor enhancements. These utilities provide excellent access to most text-based applications, but creation and representation of graphics and images can still be a challenge for blind and low-vision users.

As you may imagine, many extremely smart people have given the problem of accessible graphics creation and presentation considerable thought. Creating a tactile (feelable) version of a graphic is one option, but this requires quite a bit of extra hardware and expense. On the other hand, high-quality sound is readily available on almost any computer, making a sound-oriented solution extremely portable and low cost. Like the screen reader, an accessible drawing tool must address two challenges — controlling the pen and providing useful auditory feedback about where marks have already been placed.

Drawing and Review:

As with screen readers, use of keyboard commands provides an excellent set of options for moving a review cursor or drawing tool. Arrow keys provide a clear framework for direction with a specific increment for each keypress, allowing a sort of dead reckoning even without the sonification feedback. Other researchers have also identified the importance of being able to move quickly to specific coordinate locations (Kamel and Landay’s 2000), so we have included the Go-To feature mentioned above. Also included is a “Where Am I” command which announces the current cursor coordinates.

We have included multiple options for controlling the cursor location in the drawing space, but we expect most blind users will opt for the arrow keys. When the drawing space is active and the arrow keys are specified as the drawing mode, the user can move around the space using the arrow keys. When the pen is toggled down (drawing mode) each press of an arrow key moves the drawing point in that direction leaving a mark behind. Sounds provide feedback about the X and Y location of the current pen stroke. If the pen is up (not drawing) then the sound output acts as a review cursor, providing auditory feedback about what has already been drawn in that location of the canvas.

Sound Output:

Sound Canvas uses sonification to provide an auditory representation of where the colored pixels are in the drawing space. Sonification is the use of non-speech sounds to represent quantitative data. As described earlier, we use stereo panning to represent horizontal location and pitch of multiple or individual tones to represent the vertical position of colored pixels.

While we benefit from the work of many auditory displays researchers including Gregory Kramer, Bruce Walker, and many others, the sonification scheme used in Sound Canvas borrows most directly from the Optophone — a sonification scheme demonstrated as early as 1913 to let blind people read printed material with sound. The Stereotoner (an Optophone-like device developed in the 1960s) used a camera with 9 pixels oriented vertically to roll across a printed page. The camera could be zoomed in or out to let it span a single line of text. As the user rolled the camera across the page the sound indicated what the ink was doing, with high tones at the top of the line and low tones at the bottom. For example, a capital L would sound all of the tones at once, followed by a single low tone, while a capital A would start low, rise, split into a middle tone plus a tone that continued to rise and then fall (all while playing the middle tone of the A’s crossbar), and finish with a single falling tone.

While we no longer use this approach to read print, many sonification designers have since applied the mapping of pitch to vertical position for the auditory presentation of graphs and images. Recent examples include Peter Meijer’s vOICe system, and the Ed Summers SAS Graphics Accelerator. One of the authors (Joshua Miele) used a similar approach in SKDtools (2003) — a sonification toolbox for MATLAB.

An innovation of Sound Canvas is its dual exploration mode which plays all pixel tones in the current column simultaneously when moving the cursor horizontally, but which plays only individual pixel tones when moving the cursor vertically. This gives the user the flexibility of scrubbing horizontally to obtain a qualitative sense of the canvas, while making it easy to obtain exact pixel placement information when scrubbing vertically.

Additional Comments:

The accessibility features of this experiment include an innovative sonification tool for both drawing and exploring, “a go-to” position command, and keyboard equivalents for all major functions. We recognize that the drawing function itself is quite rudimentary, leaving many common drawing tools unimplemented, including the ability to layer and label objects, draw lines at an angle, create circles and other polygons, etc. In order to keep things simple and focus on the accessibility features of the interface, we have provided only an etch a sketch-style tool for keyboard-based drawing.

How other people can take the code/components and make their own

We hope other developers will continue the work we have started here. Please visit the Creatability component repository on Github if you are interested in using the components library we have created and used in the project.