Creating A VR Audio/Visual Experience On the Web With A-Frame and Tone.js
A-Frame is a framework for creating virtual reality experiences on the web. With just a link, anyone with a VR headset or VR-enabled phone can immerse themselves in a 3D space. Tone.js is a JavaScript library for creating sounds. Let’s see what happens when we combine the two.
First, we will create our environment. A-frame makes this really easy. With just some basic HTML we can create a whole 3D environment with aframe-environment-component. Below is some markup that will get us started.
<!DOCTYPE html>
<html>
<head>
<title>Basic Scene with Environment - A-Frame</title>
<meta name="description" content="Basic Scene with Environment - A-Frame">
<script src="https://aframe.io/releases/1.0.4/aframe.min.js"> </script>
<script src="https://unpkg.com/aframe-environment-component@1.1.0/dist/aframe-environment-component.min.js"></script>
</head>
<body>
<a-scene environment="preset: starry">
<a-camera>
<a-entity cursor="fuse: true; fuseTimeout: 500"
position="0 0 -1"
geometry="primitive: ring; radiusInner: 0.02; radiusOuter: 0.03"
material="color: black; shader: flat">
</a-entity>
</a-camera>
</a-scene>
</body>
</html>Notice the <a-entity cursor> Inside of our camera. This will allow us to interact with our synth later on. But before we get to that, let’s make sure that our project is loading correctly.
When you open the page, you should see a 3D sky full of stars with a grid on the ground. This was created by aframe-environment-component when we set the environment property with <a-scene environment=”preset: starry”> .
If you’d like to try changing the environment, simply add another preset. As of writing this tutorial there are 16 different preset environments to choose from.
For our synth, I like the idea of it being in space because space is awesome. Let’s make our environment look more like the surface of a planet.
First we’ll remove the grid and add a texture to the ground by changing
<a-scene environment="preset: starry">to
<a-scene environment="preset: starry; grid: none; groundTexture: walkernoise">If you run the page now, our planet is still too dark to see anything on the ground. We can fix that by adding an ambient light to our scene.
<a-entity light="type: ambient; color: #CCC"></a-entity>With the ambient light, your scene should look something like this.
Now that we have our environment, let’s start building our synth!
Creating A Component
A-Frame is built with an entity-component-system that allows us to create components and attach them to entities in our scene.
Let’s create a new file called synth.js to put our component in.
AFRAME.registerComponent('synth', {
schema: {
// Describe the property of the component.
},
init: function () {
// Do something when component first attached.
},
update: function () {
// Do something when component's data is updated.
},
remove: function () {
// Do something the component or its entity is detached.
}, tick: function (time, timeDelta) {
// Do something on every scene tick or frame.
}});
As you can see, A-Frame has some built-in lifecycle methods, this makes it easy to add interactivity to your WebVR projects. Now that we have the base of our component ready, let’s look at how we can make a synthesizer with Tone.js.
Tone.js
Tone.js is a framework for creating interactive music in the browser built on top of the Web Audio API. Creating a synth with tone.js is as simple as including the line var synth = new Tone.Synth().toMaster() but we are going to create an oscillator and add some parameters so it will be easier to customize later on.
const synth = new Tone.Synth({
volume: -15, // -15dB
oscillator: {
type: 'triangle' // oscillator type to triangle wave
},
envelope: {
attack: 0.05, // envelope attack set to 50ms
release: 2 // envelope release set to 2s
}
}).toMaster()Let’s add this code directly above our component in our synth.js file. Now that we have a synth, we need to include a way for our component to trigger it. Remember that <a-entity cursor> that we added to the camera earlier? This cursor has the property fuse="true" . This will enable us to listen for when the cursor is interacting with our entities. Let’s add an event listener to our component for the fuse cursor.
We will create our event listener in the init lifecycle method and we will create a new method called trigger for our tone.js trigger.
...init: function () {
// attaching a listener to our element for the fuse cursor
this.el.addEventListener('fusing', this.trigger.bind(this))
},// Creating a new method to trigger our synthtrigger: function () {
//tone.js function to trigger our synth with our parameters
synth.triggerAttackRelease(this.data.note, this.data.duration)
},...
Including The Synth Component In The Scene
So we have our component created, now we have to include it in our A-Frame scene.
First, let’s include Tone.js as well as our synth component in our html file. Make sure that your synth.js is loaded after Tone.js.
...<script src="https://unpkg.com/tone@13.8.25/build/Tone.js"></script><script src="synth.js"></script>
</head>...
We also need some entities to attach our component to. Let’s add some of A-Frame’s default shapes to include in our scene.
<a-scene>...<a-box synth="note: E4" position="-1 0.5 -3" rotation="0 45 0" color="#4CC3D9"></a-box><a-sphere synth="note: C4" position="0 1.25 -5" radius="1.25" color="#EF2D5E"></a-sphere><a-cylinder synth="note: G4" position="1 0.75 -3" radius="0.5" height="1.5" color="#FFC65D"></a-cylinder>...
Notice the synth property. This is the component we created. ‘Synth’ is the name we registered with AFRAME.registerComponent(‘synth’, {}) and “note” is what we defined in our component’s schema. We also have a “duration” property that we can use to change the length of the note. For example synth="note: E4; duration: 8n" would play an 8th note rather than the default quarter note.
Now if we open our scene in a web browser we should see our shapes and when we move the cursor over them a note should play from our synth component!
Using The Oculus Go Controller
The way our scene works now is with a fixed cursor in the middle of the screen. On a VR headset this would be called “gaze” controls. When you turn your head the cursor will move and look towards an object to aim the cursor. This is a fine experience and works nicely for a lot of projects. But what if we want to control our synth with a VR controller? Waving your arms around to play music sounds like fun, so let’s try modifying our scene to use an Oculus Go controller.
First we need to add a couple of entities to our scene for our controller and raycaster.
...<a-entity oculus-go-controls>
<a-entity laser-controls raycaster="far: 200; interval: 100"></a-entity>...
Here we have our own entity for the Oculus Go controls as well as one for our raycaster, which is set to fire every 100ms.
Next let’s modify our synth component for our Oculus Controls. We will do this by adding the raycaster to our component’s dependencies.
AFRAME.registerComponent('synth', {
dependencies: ['raycaster'],...
Then in our init method we will change the event listener to listen for raycaster-intersection .
init: function () {
this.el.addEventListener('raycaster-intersection', this.trigger.bind(this))
},...
Running the scene in an Oculus Go should now show your controller with laser controls and the notes should play when you move the laser over the shapes.
If you’d like to try it out, you can run it and see the full code here — https://glitch.com/~space-synth-vr
Wrapping Up
Now that we have a basic VR synthesizer scene created, there are tons of possibilities to enhance the experience! We could add more objects to interact with and more synthesizers with effects to our component. We can also add animations to the objects when we interact with them. As our scene grows we will have to keep performance in mind. Luckily A-Frame has a lot of built in features to help with this.
Here are some helpful links
Components — https://aframe.io/docs/1.0.0/core/component.html
Raycaster — https://aframe.io/docs/1.0.0/components/raycaster.html
Interaction and Controls — https://aframe.io/docs/1.0.0/introduction/interactions-and-controllers.html
Tone.js — https://tonejs.github.io/
Project Code — https://glitch.com/~space-synth-vr
Thanks For Reading!
