Props for crunching the numbers! As a musician and a researcher myself, I’d like to inject a few things to add a bit more depth into this discussion. As you pointed out at the start, you’re only looking at lyrics, so it’s really insufficient to make the claims you do towards the end (addressing “music” in general. I know I’m nitpicking here, but necessary):
- Song length — Technology was a key limiting factor in the dissemination of popular music. Pop songs in the 1960-70s were much shorter, in the range of 2min 30sec compared to their modern 3–4 min counterparts. They couldn’t fit enough long songs into EPs and LPs. When you stratified the data into 10-year regression lines, it became fairly obvious to me that longer songs with no sig. increase in words seems to hint towards more repetition (which is one of the common arguments for how pop music is ridic now)
- Also a bit of a pet peeve maybe because I play music: Different song/artist does not necessarily indicate a “changing musical landscape” namely because of the way culture industries function: High-risk-high-return environments that encourage hit-factories and formulaic song-writing. Although I grew up in the 90s, I could never tell the difference between Westlife, Nsync, and Backstreet Boys…
- You might want to check out these papers (they’ve gotten quite a lot of press in the past) — http://dx.doi.org/10.1038/srep00521 (they did a pretty good job of operationalizing various musical traits) and
- Model on complexity and commercial success http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0115255#pone.0115255.s001
- In addition, Adrian North and David Hargreaves wrote an INCREDIBLE book on the social psychology behind music preference.
It’s really quite hard to just restrict the discussion to just lyrics, because the method of analysis doesn’t do it justice. It’s like trying to determine the merit of a research paper based on the number of unique words. The inherent semantic aspects, along with instrumentation, tonality, timbre, loudness, are all lost!