Drawing With Sound

Authored by Josh Miele (Smith Kettlewell, @berkeleyblink) and Claire Kearney-Volpe (NYU)

Sound Canvas is a tool that invites users to draw through sight, sound, and gesture. Like each of the Creatability Experiments, this project prioritizes accessibility and universal design by accepting multiple modes of input and output. We are particularly excited about Sound Canvas, because it is intended to support blind and visually-impaired users in drawing through a meaningful system of non-visual feedback. In addition, the Sound Canvas project is the result of user-centered practices, with blind and visually-impaired design, usability, and product-testing experts included at all stages of iterative development.

The Basics/Key Features:

Multiple Inputs: In order to be inclusive of a range of users with a range of ability, the experiment accepts multiple modes of input. You can draw with your mouse or trackpad, arrow keys, or map to a point on your body, like your nose.

Sonified Canvas: Drawings are translated into sound in real-time — using a special sonification system (see below). Sounds pan from left-to-right as you draw a line from left to right, and the pitch of the sound goes from high to low as you draw a line from the top to bottom of the canvas.

Exploring with Sound: When your pen is up, you can explore your drawing using horizontal and vertical sonic scrubbers (left/right and up/down arrows). As the arrow keys are used to scrub horizontally, not only does the position of the sound shift using stereo location, but the sound simultaneously represents all the pixels in the column. Since all the pixel tones are played together, the user gets a quick overview of where drawing has already taken place. In addition to moving horizontally, the vertical arrow keys can be used to move up and down the column to hear the vertical location of individual pixels with their associated pitch.

Input/Go-To Position: We included a feature that allows you to go to a specific position on the canvas. You can use the key command Shift + G to move the pen or exploration position to a specific location relative to the top-left corner of the canvas.

Key Commands: The inclusion of keyboard commands for interacting with important UI elements is another basic aspect of universal design. In addition to Shift + G to move to a specific location in the drawing area, Sound Canvas includes the following keyboard commands:

  • Mute: Shift + M
  • Pause: Shift + P
  • Focus Canvas: Shift + C
  • Pen down: Shift + U
  • Position: Shift + G
  • Keyboard Shortcuts: Shift + k

Try it out!

Background and Design Considerations:

As you may imagine, many extremely smart people have given the problem of accessible graphics creation and presentation considerable thought. Creating a tactile (feelable) version of a graphic is one option, but this requires quite a bit of extra hardware and expense. On the other hand, high-quality sound is readily available on almost any computer, making a sound-oriented solution extremely portable and low cost. Like the screen reader, an accessible drawing tool must address two challenges — controlling the pen and providing useful auditory feedback about where marks have already been placed.

Drawing and Review:

We have included multiple options for controlling the cursor location in the drawing space, but we expect most blind users will opt for the arrow keys. When the drawing space is active and the arrow keys are specified as the drawing mode, the user can move around the space using the arrow keys. When the pen is toggled down (drawing mode) each press of an arrow key moves the drawing point in that direction leaving a mark behind. Sounds provide feedback about the X and Y location of the current pen stroke. If the pen is up (not drawing) then the sound output acts as a review cursor, providing auditory feedback about what has already been drawn in that location of the canvas.

Sound Output:

While we benefit from the work of many auditory displays researchers including Gregory Kramer, Bruce Walker, and many others, the sonification scheme used in Sound Canvas borrows most directly from the Optophone — a sonification scheme demonstrated as early as 1913 to let blind people read printed material with sound. The Stereotoner (an Optophone-like device developed in the 1960s) used a camera with 9 pixels oriented vertically to roll across a printed page. The camera could be zoomed in or out to let it span a single line of text. As the user rolled the camera across the page the sound indicated what the ink was doing, with high tones at the top of the line and low tones at the bottom. For example, a capital L would sound all of the tones at once, followed by a single low tone, while a capital A would start low, rise, split into a middle tone plus a tone that continued to rise and then fall (all while playing the middle tone of the A’s crossbar), and finish with a single falling tone.

While we no longer use this approach to read print, many sonification designers have since applied the mapping of pitch to vertical position for the auditory presentation of graphs and images. Recent examples include Peter Meijer’s vOICe system, and the Ed Summers SAS Graphics Accelerator. One of the authors (Joshua Miele) used a similar approach in SKDtools (2003) — a sonification toolbox for MATLAB.

An innovation of Sound Canvas is its dual exploration mode which plays all pixel tones in the current column simultaneously when moving the cursor horizontally, but which plays only individual pixel tones when moving the cursor vertically. This gives the user the flexibility of scrubbing horizontally to obtain a qualitative sense of the canvas, while making it easy to obtain exact pixel placement information when scrubbing vertically.

Additional Comments:

How other people can take the code/components and make their own

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store