I’ve always been a fan of HIIT (High Intensity Interval Training) and I prefer not to heat my house in the winter, therefore I’ve dabbled in DDR. After playing most of the songs in StepMania, the open source DDR software, you eventually want to play your own music but still have reasonable choreography for your songs.
Generating DDR step patterns ends up being a pretty straight forward task for an NLP model. In the past I’ve played around using FastAI and NLP to create dialog for Star Trek the Next Generation and a new script for The Room. This was my next foray into NLP and using the newest version of the FastAI library.
NLP works by creating a language model from a corpus of texts. In general, NLP accuracy is good if it is around 30%. That is, if we are able to predict the next word in text 30% of the time then we should be satisfied. DDR is a much simpler proposition as there are ~20 different options for steps so accuracy can be much higher. I was able to achieve accuracy of 50% from my stepmania step files.
Before starting this project I did some reading to see if others had worked on this particular task. I found a really cool paper called “Dance Dance Convolution” by some Masters Students at UCSD. This project actually looked at the spectrograph for an mp3 to predict the location of each beat. They discussed some of the shortcomings of this method in that their model does not generate particularly unique steps (freezes, jumps) but does match the beats.
The model that I created can be described in this way: Can we generate step patterns that look like they are legitimate DDR songs, without knowing what the music is? In the end I was able to generate steps and play some songs, but I have to spend additional effort on getting the stepfile to sync with the music. Which, I view as an entirely separate problem from the NLP portion of this project.
The Classifier is an interesting component of the analysis of the step files. Can we accurately predict the song difficulty from the pattern? For those that have played DDR, it is evident that a song like MAX 300 on beginner difficulty is more difficult than other songs on Hard difficulty. I would actually contend that the machine learning model is the more accurate ground truth and it can tell us which songs are improperly categorized.
Below is an example of the confusion matrix. We were able to achieve classification accuracy of 80%.
The next step in this process is creating the .dwi files that StepMania loads the step patterns from. This part is fairly straight forward but involves pulling BPM information for particular songs and determining the timing for the start of the note. This is my goal for part 2.
Github Code below: