How scorekeeper biases affect shot-selection data
One of the most popular topics in college basketball analysis this year has been shot selection. Thanks largely to the database at Hoop-Math.com, which provides shot location and efficiency data (among other interesting info), the subject is a staple of Luke Winn’s weekly power rankings, was featured by Big Apple Buckets, and has been the subject of several articles on this site, among others I’m sure I’m missing.
It is undeniably useful to know where a team attempts its shots, both to describe its style of play and to identify potential inefficiencies. But like any metric, shot-selection data is only as good as its inputs — and in this case, the inputs are far from perfect.
Most publicly available shot selection data is based off of play-by-play logs, which are standardized across the NCAA. (Synergy Sports’ college basketball statistics, which I believe include shot locations, are based off of video scouting, but those are proprietary and mostly purchased by teams themselves.) According to NCAA scorekeeping practice, each shot is classified as a dunk, layup, jumper, or three-pointer. (For analytical purposes, dunks and layups are usually combined into “shots at the rim,” since there are so few dunks.)
There’s little ambiguity in classifying three-pointers or dunks, but the line between “layup” and “two-point jumper” is much less clear. Certainly, a 19-footer isn’t a layup, and an uncontested finger roll isn’t a jump shot. But what about a three-foot leaner from the post? This call must be made immediately, so the scorekeeper can be ready for the next rebound or shot — and it’s ultimately a subjective decision.
Where there is subjectivity, there is the possibility of bias. Not “bias” in the pejorative sense that indicates a nefarious motive, but “bias” in the statistical sense — scorekeepers at different arenas might have more (or less) inclusive definitions of the term “layup,” leading them to count more (or fewer) shots in that category. So if our statistics indicate that a given team takes lots of two-point jumpers, it could be because they actually do — or it could be because the scorekeeper in their arena classifies most in-between shots as “jumpers.”
How can we test if these biases matter? If scorekeepers vary significantly in how they record these shots, then teams who are credited with lots of layups are likely to have scorekeepers with an inclusive definition of “layup” — in which case, their opponents would take lots of layups as well. So, to test this theory, we can see if a team’s attempted layups and its opponents’ attempted layups are positively correlated.
(Ideally, we would only look at home games, since teams play for different scorekeepers on the road, but I only have data at a full-season level. It seems safe to assume that road games are played for an approximately random set of scorekeepers, in which case, there will be no aggregate scorekeeper bias in road games. Therefore, any consistent bias in home games should still show up in full-season data; it just won’t be quite as strong, since home games only comprise about half of the sample.)
In fact, when looking at the shot selection of all Division I teams in 2011–12 and 2012–13, there is a noticeable correlation (R2= .31) between the percentage of two-pointers a team takes at the rim and its opponents’ corresponding percentage. (All data from Hoop-Math.com)
Now, this correlation doesn’t prove that scorekeeper bias in shot selection exists. Maybe teams that truly take more shots close to the rim also truly allow more shots close to the rim. I don’t think that’s likely — if anything, I think the opposite should hold in theory. (To the extent that shot selection is a strategic choice, teams that emphasize taking shots at the rim more on offense should also scheme more to not allow those shots on defense, which would lead to a negative correlation.) But there are plausible theories — maybe teams that take more shots at the rim trade defense for offense, or maybe players with certain skill sets cluster in certain conferences.
Fortunately, we have a control group of sorts: three-pointers versus two-pointers. Unlike the layup/jumper distinction, there is no subjectivity involved when defining a three-point attempt — either the shooter is behind the line, or he’s not. So if the above correlation is due to scorekeeper bias, it should disappear when comparing each team’s share of three-point attempts to its opponents’. On the other hand, if teams that truly take more shots at the rim also allow more shots at the rim due to stylistic factors, we should probably see the same pattern when looking at three-pointers.
There is essentially no correlation (R2= .002) between the share of shots a team takes from three-point range and its opponent’s corresponding percentage. Maybe there’s a stylistic reason why teams that truly take lots of shots at the rim also allow lots of them, while those that take lots of threes allow only an average number; if you have a theory I haven’t considered, leave a comment, email me, or send a carrier pigeon. To me, however, it seems the most likely story is that different scorekeepers have different standards for “layups” and “jumpers.”
A complete proof of this theory would compare shot selection data from home and away games; if scorekeepers in different arenas influence shot selection, there should be teams who attempt and allow significantly more “jumpers” at home than on the road, and others who attempt and allow significantly fewer. I only have season-level data, but in examining a few extreme cases, that pattern seems to hold.
Take Missouri State, who took more jumpers as a share of its two-pointers than all but one D-I team in both 2011–12 and 2012–13. On the road in conference play last season, 65% of the Bears’ twos and 56% of their opponents’ were classified as jumpers. But against the same exact teams at home, those numbers were 91% and 93%, respectively, which is how Missouri State ends up in the bottom-left corner of the first chart above. (In the most extreme case, 2011–12 Purdue, every single two-pointer taken by the Boilermakers or their opponents at Mackey Arena was classified as a “jumper.” I don’t know if that was miscommunication, a software bug or something else — but that couldn’t possibly be representative of Purdue’s actual shot selection.)
I haven’t yet seen this particular theory posed anywhere else, but the line of thinking isn’t new. In baseball, Colin Wyers found the classification of “line drives” versus “fly balls” is inconsistent across stadiums, and in college football, I’ve found some scorekeepers call more tackles “solo” versus “assisted.” Frankly, I’d be much more surprised if it turned out every college basketball scorer somehow defined “layups” and “jumpers” the same way.
I don’t think this means shot-selection data based off of play-by-play logs is worthless; I know I’m still planning to use it occasionally. Given that the classification of many two-point shots seems more or less unambiguous, I think there is still some value in the information we have. But I encourage everyone to keep in mind that this data is not perfect — and that, especially for teams at the extremes of shot selection, it may say as much about the scorekeeper as it does about the players.
Originally published at beanpothoops.com on January 6, 2014.
