Creating Music with Flutter
A while back, I fell in love with the tonematrix, a web-based musical instrument inspired by the Tenori-on™. The tonematrix is a 16x16 grid of buttons that each represent a musical note. The rows on the instrument step through the major pentatonic scale, starting with the middle C note. The instrument will loop through the columns of the grid, playing the notes of any selected button within the column. The beauty of the tonematrix is that it can generate pleasing melodies no matter what pattern you create.
Its only problem is Flash. The instrument was built on a now-deprecated platform that is becoming unsupported across all modern browsers. To keep the tonematrix alive, we can try porting it over to Flutter, which should hopefully stay supported for the foreseeable future :)
Building the tonematrix in Flutter presents some fun challenges. First, we need to figure out how to play music from our application. Second, we need to create a grid of buttons that each represent their own musical note. Finally, we’ll need to use a Stream to loop through the grid horizontally, playing the notes from the selected buttons. We’ll need to maintain a constant tempo as we loop through the grid, so performance will be a critical factor.
Let’s get started.
Building the music box
Starting with the grid
The design of the tonematrix is generally straightforward. To start, we’ll keep it simple and create an 8x8 grid. We’ll generate a list of 8 Row widgets that each hold a Column widget of 8 buttons. The buttons will be spread evenly across the grid. Ideally, we’d spend some time ensuring that the grid is responsive, adjusting to the width and height of the given mobile device. For now, we’ll restrict the size of the grid to a 400x400 square. Our grid button needs to hold its isSelected state so that we can highlight each toggled button.
We can now start creating patterns on our music box:
Wiring up the playback functionality
We need to loop through the grid horizontally, playing the buttons column by column. We can’t hog up the main thread with our loop, so we need to use Futures to ensure that our instrument runs asynchronously. The tonematrix has a tempo of 120 bpm, so we can create a periodic Stream subscription that fires a Future event every 125ms. For each event, we can do the following process:
- Find the buttons that have been selected within the current column.
- Play the music notes associated with the selected buttons.
- Highlight each toggled button to indicate that the button has been triggered.
- Move to the next column.
Let’s solve the first step. We need to keep track of which buttons have been selected in our grid. Since we’re currently storing our states within each button, we’ll need to lift our state up. The simplest solution would be to add a callback method parameter in our buttons and store our selected button states in the parent grid widget. However, this approach can easily lead to an anti-pattern called prop-drilling, which causes major headaches when you want to restructure your widget tree.
Instead, we’re going to try and push our buttons state into a global state that can be accessed across our application. We’ll be using the provider package, which provides a relatively simple approach to state management. Flutter provides an excellent guide to getting started with provider, and we’re going to model our application with a similar structure.
We’ll be managing our state with a ChangeNotifier, which can be used to store the selected button state of our grid. Whenever a button is tapped, we can update our state with Provider.of:
Provider.of<GridState>(context).addButton(column, row);Our grid state can look something like this:
Playing some music
With our grid state, we can now trigger our Stream subscription to start playing the notes associated with the selected buttons. We’ll leverage a handy library called flutter_midi to play our notes. All we need to do is load up a MIDI file, and call into playMidiNote. Starting at the C3 octave, we can translate our pentatonic scale from musical notes into MIDI notes. Since there are only 5 notes within the pentatonic scale, we’ll need to step through to the next octave to get enough unique notes for 8x8 grid. We can do this by adding 12 to each of our MIDI notes in our C3 scale. For instance, the C3 note can be translated to 48. Adding 12 gets us to the C4 note, which translates to 60. We can follow this pattern across multiple octaves, so we’ll never run out of notes!
We’ll be using a sine wave sound file to simulate the tonematrix as closely as possible. The Perfect_Sine.sf2 file used for our music grid can be found in a collection of free synth soundfonts.
Creating a light show
With our ChangeNotifier grid state, we can easily notify listeners of any change made to our grid. As we’re looping through our grid playing music column by column, we can create a blinking effect for each triggered button in the current column. We’ll achieve this effect by doing the following:
- Setting the color to a lighter shade.
- Wrapping the widget into an
AnimatedContainerto simulate a button blinking on and off. - Adding a white
BoxShadowto simulate the light spreading away from the button.
Since the state has been lifted up, our grid buttons can be stateless widgets. To listen to changes made in our grid state, we can wrap our grid buttons with a Consumer<GridState> widget. Our button will re-build anytime notifyListeners is called in our grid state.
We have now fully wired up our music box. Let’s see what it can do!
Adding gesture detection
While our 8x8 music box is pretty cool, the original tonematrix uses a 16x16 grid. Let’s try and fit 256 music buttons on our mobile screen:
Wow, those are tiny buttons. Having to press each individual button is going to be a terribly tedious exercise. To make the selection experience a little bit less painful, we’re going to try and use gesture detectors to select multiple buttons with a single swipe.
We’re going to listen to onPanUpdate events, which keeps track of a pressed-down pointer as it moves across the screen. Gesture detection in Flutter will not automatically provide which widget is underneath the pointer, so we’ll need to do some math to determine the position of the pointer relative to the grid. We can trigger Provider.of<Context> call to our application state once we figure out which button widget to toggle.
Since gesture detection is now happening on the parent Grid widget, our buttons can transform into musical “squares”.
We can now drag our finger across the screen to select multiple squares at a time.
Controlling our music box
For our final step, we need some way to control our Stream subscription. We could have it constantly running once we load up our application, but that might get annoying fast! We can leverage our application state again to control whether or not our music box is playing.
We can now have a GridControl widget that holds a play and reset button. The play button will start or pause the Stream subscription, while the reset button will wipe out the selected squares in our grid. We can add an AnimatedIcon to our play button to add some flair.
Our 16x16 music box should now be good to go. Let’s make some music!
Conclusion
We’ve managed to build a pretty cool music box with only simple state management, Stream subscriptions, and a midi player. It comes nowhere to the original tonematrix, but I’m sure we can put in a little bit more elbow grease to get it right. There’s a ton of improvements that can be made:
- Making the grid responsive across all devices. We could also store the
GridDimensionsproperties into anotherChangeNotifierthat can update our grid anytime the size of our device changes. - More playback controls. We could control the tempo of the box, reverse the order of the playback, or change the midi file (any .sf2 file works with flutter_midi).
- Adding web support. flutter_midi currently only works on Android or iOS, so we’d need to figure out how to play midi sounds on a browser. Web performance on the web is a bit finicky, so we’ll need to be careful to keep the application optimized.
You can check out the full code, and I encourage anybody to try and tackle any of the outstanding features listed above. I’ve named the project as flutternome, as I found it closely resembled a monome grid. Feel free to ping me if you run into any issues. Thanks for reading!
