Dear NFL Media Coverage, Please stop using the term analytics!
Seriously… you have no idea what you are talking about.
The NFL has entered the information age. There is no doubt. But the media coverage’s down right ridiculous use of the term analytics is misleading and potentially dangerous. Yes, there is more data than ever before. Yes, there is more analytics than ever before. But the vast majority of “analytics” reported by media coverage of various games is down right horrible.
Now you may think I am splitting hairs. Certainly not all analytics is good. So why should NFL coverage be any different? But calling their nonsense analytics is a bit like declaring anyone with a sharp knife a surgeon. You can do it — but you are going to scare a lot of people. And, as far as I know, no one is actually doing that.
Oh, come on! That surgery analogy implies bad analytics could hurt someone!
Yes… yes it can. You know what even good analytics can’t do? Protect someone. At least not if you tell everyone about it…
You see good analytics in the Risk and Behavioral space tell us that efforts to lower human risk lead to humans compensating by engaging riskier behavior. We all have a risk balance. Make our car safer and we will simply drive faster or with less care. Make our helmets better and we will stop trying so hard not to get our bell rung. So yes — bad analytics can hurt…
But all of this data must be a good thing!
Let me stop you there. “All this” data really isn’t … well, all that. An NFL season really doesn’t provide much data at all if you are analyzing decisions to help you win. Sure you can stack seasons together… if the rules didn’t keep changing.
Even if you are arguing that you are going to measure play results, 24K plays a season isn’t a big base either. Next you need to consider all the possible combinations, permutations, and externalities that occur. For the NFL media, that is things like down and distance, time on the clock, score of the game, and field conditions. All of these factors make even the most routine play anything but…
Which brings us back to AWS. How many times in the last two decades was this pass been attempted that you could calculate a pass probability to three decimal points? With what confidence interval? Sorry… you either made up the numbers OR you have a simulation engine that makes them up for you…
The latter is at least defensible, but that is not how anyone in the greater media is portraying any of this. No one is saying the analytics “predicts”. We are all lead to believe that each team now has a team of crack analysts that know everything. Everyone is so damn smart now… until they lose.
Me: “How do you evaluate whether you have hired good analytic talent?”
Billy Beane: “George, I can’t. I just know I can hire them cheaper than anyone else.”
Yes. The NFL analytic gurus are in fact the lowest bidders. That is — people so excited to work in the NFL, MLB, etc — that they take minimal salaries in a field that pays serious bank. That may not speak to their skill set… but it doesn’t speak for their skill set either. And by the way, much like Wall Street, if you are good at sports analytics — you can make some serious coin. That wasn’t always true, legally, but sports gambling is now above board.
Look sports data is awesome. Sports analytics — the real stuff — is exciting. Just don’t be fooled by all this media hype. Data is not analytics… it is just data. Analytics doesn’t “say” — it predicts — it might even suggest.
And as all students of game theory know — any time the data does become conclusive — the competition will price it into their analysis, too. Just like Wall Street and Money Ball — it will quickly lose its effectiveness. Any analytics you are hearing about… only worked in the time series. If it ever worked at all…
Honestly — you get the sense that a lot of folks just want to sound smart. Unfortunately, if the NFL lacks for sample size and often oversimplifies, it is a rather efficient feedback loop. Each week — all those “analytic” decisions are put to the test. Spend a lot of time spouting nonsense analytics and you won’t sound smart for very long. Just ask some of these guys:
