Song Recommends Based on Lyrics

Not so long ago I wrote a post that would help detect similar songs based on the lyrics alone. If you are a music lover and really do not have your heart set on a single genre then basing your song pics off of lyrics alone is definitely a different way to look at how you choose your next song. For many applications such as Spotify, songs are recommended based on genre.

Typically genre is based on instruments used, pace/ beats, and tone. If you look at rap music verse rock and roll they are not very dissimilar in types in the language they use but very different methods to wrap their lyrics.
I remember a long time ago now when Johnny Cash passed away so many artists from all genres talked about his music and it always surprised me how many rappers really loved his music and found him as inspiration to their own music. Johnny Cash had a large number of songs describing his life and his struggles. If we really think about this many rap songs are describing the artist's struggles. We also can see this in rock music such as nirvana singing about deep-seated depression and angst.

I believe much of the time songs are recommended based on so many different factors aside from the lyrics. The lyrics are typically the poetry to the soul of the artist and the deep feelings of the song are tied to the lyrics. I felt that it was important to focus on showing that maybe branching out to a new genre may inspire similar feelings to a song from their normal playlist.

For this, we will leverage a content-based recommendation system. A content-based recommendation uses the item's features to make recommendations rather than similarly liked items by a user. In this case, the features will be the lyrics of the song. We will go about doing this in a few different ways to see how our results vary from each methodology. In the first method, we will leverage TFIDF vectorization and then measure the distance between the songs with cosine similarity.

Once we have made our features in vectors we then create a similarity matrix using sklearns pairwise.cosine_similarity function. This will allow us to simply loop through the results and sort each one by the highest similarity value.

When we input say “God’s Plan” by Drake we get

Number 1: What Lovers Do by Maroon 5 Featuring SZA with 0.361 similarity score

Number 2: Cash Shit by Megan Thee Stallion Featuring DaBaby with 0.132 similarity score
Number 3: No Limit by G-Eazy Featuring A$AP Rocky & Cardi B with 0.115 similarity score
Number 4: Got What I Got by Jason Aldean with 0.114 similarity score
Number 5: Powerglide by Rae Sremmurd & Juicy J with 0.103 similarity score

Input song “Wishing Well” by Juice World

Number 1: Party Girl by StaySolidRocky with 0.115 similarity score
Number 2: The Bigger Picture by Lil Baby with 0.103 similarity score
Number 3: Got What I Got by Jason Aldean with 0.101 similarity score
Number 4: Worth It by YK Osiris with 0.096 similarity score
Number 5: New Rules by Dua Lipa with 0.095 similarity score

Input song “I Hope You’re Happy Now” by Carly Pearce & Lee Brice

Number 1: I Hope by Gabby Barrett Featuring Charlie Puth with 0.451 similarity score
Number 2: HIGHEST IN THE ROOM by Travis Scott with 0.144 similarity score
Number 3: Say Something by Justin Timberlake Featuring Chris Stapleton with 0.141 similarity score
Number 4: Be Careful by Cardi B with 0.136 similarity score
Number 5: Him & I by G-Eazy & Halsey with 0.131 similarity score

Many of these songs do not really seem similar in terms of genre but the lyric features are conclusively the most similar of the last 3 years' top 100 songs. It is interesting to see a Maroon 5 song recommended for a song by Drake but when we take a closer look at the lyrics they are fairly similar.

For all three different inputs, we get songs from almost all types of genres. “God’s Plan” by Drake is considering R&B while “Wishing Well” is rap and “I Hope You’re Happy Now” is country. “I Hope You’re Happy Now” has a very strong score against “I Hope” by Gabby Barrett Featuring Charlie Puth. This is one of the highest similarity scores in the entire dataset.

TFIDF does typically remove a lot of high-frequency words because it may not be as valuable as a word not mentioned as often. If a song does not have a lot of words or uses the same words over and over (Which is something we see fairly often now) the song may not have a lot of features to actually compare and it would be found to be similar to almost any other song.

Another way we can rank our songs on similarity is by leveraging Levenshtein Distance. This distance metric tells us the number of characters that differ between 2 strings. For example cat and bat, the Levenshtein distance here is 1 because c and b are the only difference between the words.

We will use the same songs so we can compare results against lev distance.

song:Walk Me Home artist:P!nk score:1140

song: I Like Me Better artist:Lauv score:1148

song: You Say artist:Lauren Daigle score:1149

song: Sad! artist:XXXTENTACION score:1156

song: Shallow artist: Lady Gaga & Bradley Cooper score: 1157

Input “Wishing Well” by Juice WRLD

song: Old Town Road artist: Lil Nas X Featuring Billy Ray Cyrus score:1394

song: All To Myself artist: Dan + Shay score:1405

song: Rain On Me artist: Lady Gaga & Ariana Grande score:1409

song: One KissCalvin Harris & Dua Lipa score:1411

song: Feel It Still artist: Portugal. The Man score: 1415

As we can see the results are extremely different from using TDIDF and cosine similarity. With Lev distance, we are taking in every single character into account and with cosine similarity paired with TFIDF we get a population of features. The features that are deemed to be most important to the population of song lyrics.

Limitations

The issue with only using song lyrics is that we do not account for many other variables that can greatly differ a song from one another. The beat/tune of the song can be equally as important as the lyrics. If we were able to obtain that information we could potentially make much greater recommendations to the user.

Take-Aways

When beginning this experiment I really thought that a few of the songs from both tests would overlap with one another. Obviously, after running this experiment I am completely wrong in that assumption. I am sure using Euclidean Distance vs Cosine would not change results too much but possibly replacing Levenshtein Distance with Jaccard I would obtain very different results as well. Since Jaccard and Levenshtein use fairly similar methodologies I can assume they would give similar results.
I could potentially use the features from the TFIDF to see if Cosine Similarity and Levenshtein get more overlap. This would be another approach I could try to strengthen recommendations.