DDR Step Generator — Part 2

Continuing on from part 1 linked below:

DDR Difficulty Classifier with FastAI NLP — Part 1

After training a NLP model that accepts step mania files. I was able to then use the prediction mode of the NLP model such that we can then generate a foot pattern for any arbitrary song.

This exercise was less relevant to the use of deep learning and more going through the effort of creating the .dwi file that step mania expects.

This project was simplified with some libraries I was able to find that can perform beat matching and provide the offset time for the initial gap. Additionally, typical DDR songs are in the 90s to 180s range. As an excercise to the reader, a Herculean effort could be possible if one were to complete a 10+ minute long song. In fact, I would contend that they might just be the strongest person in the world if they can successfully complete Innagadadavida (17 minutes in duration).

Some of the Librosa beat matching and gap finding is shown in the code block below:

def MakeMP3(filePath, shorten = True, shortenLength = 90000):
    """Shorten the mp3 if it is greater than the length specified.  Default is 90 seconds"""
    
    song = AudioSegment.from_mp3(filePath)
    newSong = song
    
    if shorten and (len(song) > shortenLength):
        newSong = song[:shortenLength]    
    
    return newSong
def GetGapTime(beatTimes, gapLength = 200):
    """Gets the first beat after X milliseconds, 200 by default"""
    for beatTime in beatTimes:
        if beatTime > gapLength:
            return beatTime
        
def GetBPM(fileName):
    # Load the audio as a waveform `y`, Store the sampling rate as `sr`
    y, sr = librosa.load(fileName)
    tempo, beatFrames = librosa.beat.beat_track(y=y, sr=sr)

    # Convert the frame indices of beat events into timestamps
    beatTimes = librosa.frames_to_time(beatFrames, sr=sr)
    #Convert to milliseconds
    beatTimes = beatTimes * 1000
    
    gap = GetGapTime(beatTimes)
    bpm = tempo
    
    return (bpm, gap)

The project was pretty successful. I am now able to take a folder of random mp3 and produce some reasonable DDR songs. A sample is shown in the graphic below:

Selection of Music from Library

And as a specific example one can see the types of patterns that it is capable of generating. I left a couple of .dwi files on the github for those that might be interested in trying this out without using the training process.

Single Song and Step Patterns

This project is reasonably successful in generating DDR songs that match the beat of the music. Overall, the patterns are arbitrarily generated without any connection to the song. Some of the songs matched very well to the music. It is essential that the bpm of the song does not change over the course of the song. It is certainly possible to change the bpm through the use of standard librosa functions and can be input into the .dwi format easily but that is beyond my interest in this project. I really was interested in being able to listen to a larger variety of music.

The most interesting idea from this project would be to create an end-to-end model that actually uses a spectrogram of a song to predict the step patterns. Librosa has the ability to produce a spectrogram easily so this would be my next area of study provided I intend to spend more time on this.

Link to the code below:

brentbiseda/DDR_Deep_Learning