College Scorecard: For Analysis Not Action

Today the U.S. Department of Education released a mountain of data about colleges and universities with the following statement:

“Here, you can get the data behind the College Scorecard, as well as other data on federal financial aid and earnings information. These data provide insights into the performance of schools eligible to receive federal financial aid, and offer a look at the outcomes of students at those schools.”

Exciting indeed. Mainly because by dropping this info as soon as they could (on a Saturday morning no less) ED provided a strong rebuttal to critics who claim we don’t need government at all, and those who claim ED is hiding something. That’s all bunk and it’s been going on for far too long. So — go government! Good work.

But here’s the problem. Now there are people running all over Twitter proclaiming that the data shows that X college is better than Y college. Or that students can now decide what college to attend using this data.

And that is a disaster. Taking the graduation rate or the earnings of a college that admits 80% of applicants and enrolls loads of students from low-income families and comparing those outcomes to those of a college that cherrypicks the wealthy students it admits is irresponsible. It’s malfeasance and it goes on all the time. And the researchers on Twitter aren’t doing that (thank goodness) — but they are comparing institutions that may not be SO blatantly different but are nonetheless plausibly different and making comparisons in a way that sets back educational policy research decades. We know we can’t infer the impact of a college from descriptive statistics — these are outcomes not impacts — and we’ve been teaching our students that for years, trying to move the field forward. Why move backwards now just because we’re excited about data?

Moreover, as it’s become increasingly clear that college is an expensive decision and that students need to enroll in a place that’s a good fit for them, why would we ever suggest that they use descriptive statistics based on averages to choose where to attend? This is the exact opposite of the #ReachHigher initiative that supports high quality school counseling. You cannot conclude that UW-Madison is “better” for you because its graduates earn more than UW-Eau Claire’s graduates. They serve very different students. Please, practitioners, do not urge students and families to “make decisions” using this data.

Here’s what you can responsibly do with the College Scorecard data just released:

You can explore. That’s it.

If you are a researcher, look at patterns in the data in order to find out what deserves further investigation. But to understand what you find, you need individual-level data, not aggregate. So, good luck. If you want to figure out which schools really do better than others, you’ll need to use methodological techniques that facilitate causal inferences, and that is HARD. So you’ll have to work, not just tweet.

If you’re a student or parent, you can peruse the data to test your assumptions. Did you think College X admitted a lot of low-income students or had a high overall graduation rate? Maybe you’re wrong. Good to know. You can see who pays what and where. That’s helpful (though you shouldn’t conclude that a lower price at a private university means they “care” more — it’s because publics don’t control their prices for the most part, legislators are decimating their funding, and they lack the big endowments created by pandering to the wealthy). But take your “findings” to a knowledgable professional and discuss them. Do not rule out a college or decide on one using this information.

This data dump is about transparency — we should applaud the government for it. We should look askance at colleges and universities fighting its release. But we should also act responsibly, and not pretend that it opens up the great mystery of higher education: Which colleges and universities are having positive impacts on students? Which are not? That’s what we really need to know.