Measuring What Matters: Defining Outcomes in Higher Education

It should come as no surprise to anyone who reads this blog that outcomes in higher education are important to me. I think proving return on investment (ROI) is increasingly important to students and to the continued success of higher education, and to do that, you need to measure the outcomes of education.

But what do we mean when we talk about outcomes? In higher education, I think most of us agree that student outcomes are important and that we should focus on improving them. What we don’t agree on is what outcomes we want to improve. What’s our rubric for success when we’re talking about student outcomes? How do we measure it?

As Peter Drucker once said, “If you can’t measure it, you can’t manage it.” So let’s talk a little bit about what we mean when we use the term “outcomes.”

Graduation and Retention Outcomes

These are probably the strongest outcomes to measure because, for the most part, they are quantifiable and people care about them. How many students graduated? How many students finished a given term and started the next one? How many took a leave of absence, and of those, how many came back?

Most schools track these outcomes out of necessity; knowing how many students are enrolling, moving from term to term and graduating is vital to the basic function of a school. However, they only tell a small part of the story.

Career Outcomes

Career outcomes are, for the most part, about ROI in a traditional sense. If you pay $120,000 for your education, does that translate into a job and an income worth that investment?

While these outcomes are important to track and to prove the ROI of a degree, collecting these metrics can be notoriously difficult. It takes coordinated effort on the institution’s part to gather the information from students and alumni, and then a system needs to be developed to maintain this information. Eventually, universities want to be able to say how alumni are doing six months, a year, five years and 10 years down the road.

In addition to hard information such as job changes or salary increases, colleges might consider sending periodic qualitative surveys to their alumni. Questions could include if they use an alumni database for networking, how helpful they found their education on the job, and if they would recommend your institution to others. All of these data points will help prove the value of a degree to prospective students.

Financial Outcomes

There’s a corollary to figuring out whether students are getting jobs worth the tuition cost: tracking whether they’re able to pay off their student loan debt. In my last post, I spoke a little about the taxpayer burden of student loans. If we have a high rate of default or forgiveness on student loans, that money still has to be paid back from somewhere, and the taxpayer is the one who winds up footing the bill. For reference, the last reported cohort default rate was 11.3 percent nationwide. Given that we’ve got somewhere around 44 million student loan borrowers in this country, 11.3 percent represents a huge number of defaulting loans: almost 5 million. Income-driven repayment plans can exacerbate this further; the American Enterprise Institute (AEI) did a great piece on the hazards of income-driven repayment plans and how they can mask the high cost of higher education. These plans are a win for the school and the student, but put a disproportionate burden on the taxpayer, who does not control and may not even know what the terms of these loans are. When you take into account the fact that the Department of Education appears to have dramatically underestimated the taxpayer cost of student loan forgiveness, you can see why this kind of outcome is important to track.

Part of the responsibility for tracking these outcomes is on colleges and universities. If students have a high rate of default, then the degrees are either too expensive or do not provide enough value. It’s possible that degree prices should be correlated to projected earnings after graduation, with public services degrees, for example, being less expensive than MBAs.

While I am not one to ask for government intervention unnecessarily, tuition costs are one area where I feel like the government has an obligation to monitor financial outcomes. Because so many students are using federally subsidized student loans to pay for their education, it’s up to the government to ensure that a virtuous cycle is being created, one where students are paying what their degree is worth. Right now, there is no incentive for schools to create that cycle, because the government loans are serving as a safety net for inflated tuition costs. Ultimately, though, when students go into default on their loans because they are unable to find a high-enough-paying job, it’s the taxpayer who suffers. I would argue that the government should require outcomes tracking as part of its loan-granting process, forcing colleges and universities to put some skin in the outcomes game.

Learning Outcomes

But there’s more to an education than what happens after graduation. Colleges and universities dedicated to providing an exceptional learning environment also need to know if the courses they are offering are effective at transferring knowledge to students.

Simply put, learning outcomes are about tracking how effective our courses are at teaching what they’re supposed to teach. Passing a course and getting a certification or diploma is only part of the value that course needs to provide; it also needs to teach students the skills they need to excel in their field. The diploma helps them get the job; the effectiveness of the course helps them keep it and move up the ranks.

Course effectiveness is harder to track, but still trackable, and it’s something actively measure here at Learning House. Our quality rubrics measure student outcomes, and we refine courses based on those outcomes for continuous improvement. All of our learning objectives are based on Bloom’s taxonomy; we want to ensure that students aren’t just parroting information back, but rather are synthesizing the concepts for a deeper, richer learning experience.

Soft-Skills Outcomes

But even that doesn’t tell the whole story. There are many reasons people attend college and, while getting a better job is certainly one of them, it’s not the only reason. According to the most recent Online College Students report, while 20 percent of students enrolled in their degree program to qualify for a career in a new field, fully 15 percent simply wanted the satisfaction of completing their undergraduate or graduate education. It can be easy for us, as educators, to focus on the tangible benefits of earning a degree: a better career, the possibility of promotion or learning specific skills. But I want to be clear that there are other outcomes as well, ones that are just as important.

I’ve talked a lot about the kinds of soft skills that are critical in the modern workforce, and a large part of the responsibility for teaching these skills rests firmly on higher ed’s shoulders. Teaching the nuts and bolts of how to be an engineer isn’t enough; colleges need to take responsibility for teaching students in a more holistic way, ensuring that students leave their halls with the vital soft skills necessary to succeed in the workforce.

Critical thinking, for example, is an important outcome of a solid education. When I hire people, I want to work with those who don’t just accept everything I say, but who challenge me, who think broadly, and who can analyze the data and make good decisions.

Writing skills also are important, in every job. I know sometimes I’ll encounter people in technical fields who think that communication isn’t as critical for their role, but in my experience, no role lives in isolation. Every job function involves some level of communication, and those who can’t write clearly and speak cogently will struggle to succeed.

Then there are the intangibles, like grit and resilience and adaptability. Higher education offers a safe space to practice these skills, helping graduates not just succeed in their career, but in their life. While this is of benefit to students, it also benefits institutions — students who have these kind of “mindset” skills are far more likely to succeed in any career path, thus providing the kinds of strong outcomes institutions want.

The challenge, of course, is how to measure these kinds of skills and outcomes. I don’t actually know the answer — it’s something I still struggle with measuring myself! But I do think it’s a challenge we, as an industry, need to face sooner rather than later.

Getting Started

While definitions of outcomes can differ, one thing is true: They are hard to measure. It starts with defining success: What do you think good outcomes look like? Once you know that, you’ll know where to start measuring. Maybe that’s alumni surveys. Maybe it’s looking at entrance and exit exams. Whatever you decide, narrowing your parameters and understanding what data you are trying to gather are critical to the process.

I think that, the more schools start measuring these kinds of outcomes, the more confidence will be built in their ability to deliver value to students and to society on the whole. Transparency and accountability are powerful tools for building trust, and public trust is going to be necessary if higher ed institutions intend to grow and thrive.

As an industry, we need to agree on what outcomes mean. Right now, we are comparing apples to oranges, if one institution measures only graduation rate and another looks at employment after graduation and yet another looks at student-debt ratio … you get the point. I encourage everyone to start the conversation about what outcomes matter, and then we can all start to measure effectiveness.

What do you think “better outcomes” means? How would you track them?