EnsembleVR — Overcoming your Stage Fright with Virtual Reality
What if you could play alongside some of the greatest orchestras in the world?
--
EnsembleVR is a virtual reality application that immerses the player in a realistic concert environment. The game mimics all aspects of a classical performance, from a backstage room to audience applause and even responsive orchestral accompaniment.
Why
One issue that musicians almost universally face is that performing in front of a live audience, on-stage is completely different from practicing alone in your home. Additionally, for musicians whose performances often include accompaniment and or involve other musicians, it’s often difficult or even impossible to play a piece as it was intended — with the full accompaniment.
For example, many classical concertos involve a complete orchestra, but for the casual musician, you’ll likely never have the chance to solo alongside a professional orchestra. Of course, you could simply play a recording of the accompaniment part and play along to that, but thats no where near as satisfying as playing alongside a live accompanist who would speed up, slow down, and follow your expressive nuances.
The goal of this project, therefore, is to attempt to simulate, as realistically as possible, the experience of performing. Check out these demo videos!
- The first video is a complete example of what you’d see and hear if you were using EnsembleVR.
- The second video is a demo of the score following component where we purposely try to mess up our playing to see if it can still accurately follow
Features
Main Hall
The front stage contains the space where the user will spend the most amount of time engaging with the application. Therefore, it’s important to design the frontstage in a way that is both captivating and realistic. The frontstage architecture takes inspiration from Spaulding Auditorium at Dartmouth College. The hall contains colorful structures and cool designs to provide a unique experience for the user. There is also an upper balcony level to support large audience sizes and to make the hall feel more immersive and realistic.
Backstage Room
Every performance begins with the musicians preparing in a backstage room. Preparations typically include cleaning instruments, fixing bow ties, organizing music, and generally trying to ease nerves. In this game, the backstage room is also utilized to choose the piece, such that the user can select from a wide range of pre-loaded pieces.
Virtual Orchestra
A classical concert revolves around an orchestra consisting of strings, brass, woodwinds, and a conductor. To mimic this, we created animated human models for each of these instrument sections so that the player can truly feel like they are playing alongside an orchestra.
Audience Interactions
One of the most difficult aspects of a live performance is the fact that there is a live audience watching you. This is often the cause of stage fright, and consequently, unsatisfactory performances. We wanted to mimic this feeling so that the players could get desensitized to the pressure. Therefore, we created animated audience models that watch your entire performance. Of course, everyone needs some encouragement, so at the end of the performance, the audience will clap and cheer.
Live Accompaniment
A Python application listens to the incoming audio and accurately determines where in the piece the user is as well as their tempo. This is done using a stochastic model combining techniques such as hierarchical hidden markov models and Kalman filters/Linear Quadratic Estimation.
Just as a basic overview, we represent states in the hidden markov model as subdivisions of each note. At each state, there is a certain probability of observing a pitch, and each state also has a probability of transitioning to other states. Therefore, determining location in a score simply amounts to finding the most likely state sequence given previous observations. As for adjusting tempo, a state’s self-loop probabilities dictate how long a note lasts. We know how long we expect to hear a note, so if we hear a note early or late, we can compute the tempo of the previous note. Using this, we apply a Kalman filter to approximate our current tempo and update the appropriate probabilities.
If you want to learn more about the math/algorithms as well as some of the computational optimization tricks behind this part, please check out the Github repo and the corresponding wiki where we explain this in greater detail.
As for generating the accompaniment, we use Unity’s Pitch Shifter plug-in to change a small sample of sounds into a library of musical notes. These notes are played when prompted by the Python application via a Websocket connection.
Fixed Tempo Accompaniment
We also allow the user to practice against a fixed track. In this mode, the user can play a recording of the accompaniment for a piece by simply pressing a button. The user can change the tempo of the recording at anytime, making this ideal for musicians who want to practice keeping up with other members of the ensemble.
Issues We Came Across
There was a lot of research needed when tackling this project since score-following is still an active area of research. A lot of time was spent reading paper after paper and prototyping different methods ranging from Dynamic Time Warp to Reinforcement Learning. A lot of these papers showed promise but were not exactly what we were looking for. Luckily, we were eventually able to find a few papers that did fit our criteria, which allowed us to mix and match ideas from these papers and end up with our current implementation.
The communication between the Python application and the Unity project was also difficult to establish. Once we had it, the headset had trouble with handling the speed at which messages were being processed and sent by the Python side. This is a difficult hurdle to overcome since even a fraction of a second feels “off” to the human ear.
Additionally, many members of the team hadn’t worked with Unity prior this project. This turned a lot of the project into a learning experience, which had its ups and downs.
Impact
We believe that this VR game is a viable tool for musicians to practice performing in front of an audience. This can be expanded to musicians of all backgrounds, from young students starting off on their instruments to experienced professionals who are looking to maintain their performance composure.
The Team
We are a team of 6 Dartmouth students majoring in Computer Science. Moreover, we each have great passions for music. Many of us play instruments, such as the guitar and the violin, and we all love listening to a wide range of musical genres.
- Marshall Peng
- Bryan Shin
- Ryan Hyun
- Soohwan Park
- Myles Holt
- James Lee
Try it for yourself!
If you have an Oculus headset and want to try EnsembleVR for yourself, here is a shareable link to the .apk! For instructions on how to deploy the game, here is a link to the Github repository that includes a README with directions.
EnsembleVR .apk: https://drive.google.com/file/d/1BoVSxiobD31sZLNn6Qb-DimpiuhovuNM/view?usp=sharing
EnsembleVR Github Repository:
EnesmbleVR Score Following Github Repository:
