Storytelling with Sound: How to Develop Digital Literacy in Immersive Learning Environments

If you’ve seen the ending scene of Star Wars without John William’s music, you realize that what we hear makes up an immersive experience as much as what we see.

We perceive our physical and digital environment though visuals, words, and sounds. Students at young age learn to express themselves in writing and visual arts, but what about sound and music?

This week, the Facebook Education team invited me to curate a set of examples from teachers and students around the world using SOUND (natural sounds, sound effects, their own voice, or music) in immersive storytelling projects.

To categorize and reflect the different examples, I created the following interactive four-field matrix, and wrote a short description of each examples below. In a nutshell, my main point is this: sound design and production skills are key components of digital literacy, and this area should be included in the curriculum for digital storytelling.

Big thanks again to our teacher community for sharing your examples!

Sound in immersive storytelling: an interactive matrix

ADDING SOUND TO DEDICATED AREAS IN A SCENE

Scenes with sound effects: This is a fun project to start with. Students can use a voice recorder app to create their own sound effects such as steps, traffic sounds, bees, running water, etc., or use existing sound libraries to create an audio-visual experience of any place or situation. This winter story was made with sounds from freesound.org.

Virtual field trips with natural sounds: Here is a similar project using animal sounds. Elementary technology teacher Michelle Eckstein from Peak to Peak Charter school created a virtual tour documenting the different life zones in Colorado. For this image, she added sounds of birds at the Barr Lake. Try find the downy woodpecker or the Canadian geese!

AMBIENT SOUND AND MUSIC

Virtual lessons with music: Literary teacher Monica Samarra uses meditative music in the background of a 360 image to introduce students to American poet Walt Whitman. The music is not specifically related to Whitman, but it creates a unique ambient for reflecting his thoughts.

Crafts and digital storytelling with ambient audio: Primary school teacher Astrid Hulsebosch’s 3rd grade students in Italy built dioramas of early cultures using cardboard boxes, captured them with a 360 camera from inside, and added self-made videos, links, and music that made their creations come to life!

NARRATING PARTS OF A SCENE

Drawings and artwork with recorded sound clips: One of the most popular use cases is enhancing students’ artwork with sounds and narration. In this example, students of Nowaki Asada created an interactive sound map of Manhattan. The sound clips from different parts of the city are recorded with smart phones, and embedded in a photo of their artwork.

Immersive language learning: If you are learning new vocabulary in a foreign language, contextual experience matters. Here is a great example for English-language learners created by Yefei Jin from Nearpod. Students can see how new words are written, and listen how they are pronounced.

Explain and document a project: Students of primary school teacher John Korrasa created a virtual tour of a beehive. Some of the narrations are used to explain a specific area in the image, others to orientate the viewer to look closer or to think about certain questions related to beekeeping.

NARRATING ONE OR MULTIPLE SCENES

Virtual lessons from teachers to students: Teachers can easily create engaging expert orientations to any subject in the curriculum. In this example teacher Tapio Nenonen created a biology lesson recording sound clips about the plants and mushrooms you can find on a forest walk in Finland. (Anyone interested in ZPD & immersive learning, take note.)

Orientation to a virtual learning environment: Karsten Steiner’s student in the Erasmus MoPic Virtual Museum project narrates the starting image of a larger student-created virtual learning space. The purpose of the narration is to introduce the project to the viewer as well as to give practical guidance as to how to navigate in the environment.

Interactive 360 presentation from a field trip: 5th grade students from the Barron Park Elementary school recorded short narrations to a series of 360 images from their field trip to Levi’s Stadium. In this case, five students out of 22 had signed up to work in the sound & audio team to capture sound effects and to record scripted (or improvised) sound clips after the field trip.