Step 2: Has The Question Been Asked And Answered?

A case study on the analytical question, “Is College Worth It?”

Adam Ross Nelson
Feb 25 · 7 min read

TL;DR: This is a case study-based tutorial on how to see if a question has been asked and answered ~ fodder for the review of existing results. Sometimes also called a review of literature. Scroll below to the appendix showing selected findings on why college is ‘worth it.’

Introduction

The first step in the scientific process is often, ask a question. Or otherwise sometimes it is about defining a problem.

Usually, the second step in the scientific workflow is to look at whether anyone else has already asked and answered your question. This article is a case study on the question “Is college worth it?

First, spoiler alert: This question has been asked and answered. Second spoiler alert: There are positive returns on education, there are negative returns, and there are non-pecuniary returns. On balance, it seems the evidence establishes that college is worth it!

This article shows the process associated with looking at whether anyone else has already asked and answered an analytical question. In some contexts scientific professionals better describe this as looking to see if the problem already has an existing solution.

This is a case study-based tutorial. For this specific analytical question, this step in the scientific work-flow that should require a skilled professional less than 2–3 hours. Other specific example analytical questions could require more time (or less). And, by more time, depending on the question, it may be necessary to spend days or weeks on this step.

Getting An Initial Point Of Reference

The earliest phase of this task is to find initial points of reference. To find an initial point of reference, I will search Google’s Scholar database for keywords. I will use the keywords: “College Worth It.” By enclosing the keywords in strings, I limit the results to items with that specific string. I will further limit my search to articles that contain this string in the title.

Subject expertise can also be helpful at this stage. As a subject expert I also know that “college worth it” is a colloquial expression. I know from my experience that a more formal term of art is “returns on college.”

I will expand that list of keywords as I move through this project. But these two will work for starters. Armed with these keywords I look at Google Scholar results for “college worth it.”

There are 1,430 results. The first result is a book. The next is an article from The New York Times The third is from the National Bureau of Economic Research (NBER) (an authoritative source). The forth is a book, and the fifth is an article.

Given these results I focus on the third result, for three reasons. 1) This third result is from the well recognized NBER. 2) This third result is less than 10 years old. 3) By a large margin it is the most cited of the other top five search results. Just below each search result is the “Cited by” figure. The operative theory is that, the more anyone cites a source, the more authoritative it will be.

There it is. This article by Oreopoulos et al. is the initial reference point.

Going Deeper

There are two techniques for going deeper. The first will help find sources that came before this reference point. The second will find sources that followed.

To find sources that came before this reference point, browse the reference list. To find sources that came after this reference point, browse the “cited by” list.

Looking Forward

While browsing these lists, it will be important to note other key words and phases authors use to describe the question or problem.

Of these first ten “cited by” results, the tenth looks most promising. It is “Returns to education: The causal effects of education on earnings, health, and smoking” (Heckman, Humphries, and Veramendi, 2018). This title gives us two new key word phrases. “Returns to education” and “causal effects of education.”

Google permits the opportunity to search within search results. These new key word phrases will help focus the list.

Looking Back

A browse through the reference list from Oreopoulos et. al. (2013) points to additional sources that will be worth a look including another by the same author “Priceless: The Non-pecuniary benefits of schooling” (Oreopoulos and Salvanes, 2011). Another from the 1990s “Labor-market returns to two- and four-year colleges: Is a credit a credit and do degrees matter?” (Kane and Rouse, 1995).

Back To The Keywords

Returning to a keyword strategy based on information learned above, the next step will be to search article titles for these keywords. This time the search will be more sophisticated:

allintitle: (schooling OR college) AND (payoff OR return OR "worth it")

This search string returns 415 results. That is just over forty pages of results. A skim of the titles over those forty pages is a manageable and reasonable step in this process (and a step I took).

For this article, I focus on the second result for three reasons. 1) It is from a well-recognized authoritative publication. 2) It is almost twenty years old. 3) Over 1,200 others have cited this source.

I also recognize this article’s authors (Dale and Krueger) as well-recognized both for volume and quality of their work.

Earlier I showed where recent articles can be valuable. Newer articles let you look over the literature from our recent perspectives. The world changes. Reviewing the reference list of this 2002 article, will help us understand what was important for this problem nearly twenty years ago.

Repeat the above steps until you have your answer. An indication that you have completed your search of the available literature is that your efforts begin to return the same results over-and-over. In other words, once the above techniques disctontinue yielding new results you’re ready to move on.

Conclusion

This article gives a tutorial on how to see if a question has been asked and answered. Usually, the second step in the scientific workflow is to look at whether anyone else has already asked and answered your question. While, the first step in the scientific process is often, ask a question.

In less than 2–3 hours of work this article identified dozens of sources that tested if college is ‘worth it.’ Some phrased the question as estimating the economic or financial returns on education. At least one source tested the non-pecuniary benefits of education.

This article featured seven sources that I listed in the References section below. In the Appendix are selected findingsthat show how and why college is ‘worth it.’

Appendix ~ Selected Findings

Less than optimally Oreopoulos & Salvanes (2011) also reported “we do find a tendency for college graduates to report wanting to spend more time with friends and in leisure activities.” (p. 173).

“The evidence presented… suggests there are, in general, significant returns to a college degree. While these benefits are not constant across all college programs and occupations, college gradautes do enjoy an earnings premium across all major ocupation sectors.” (Oreopoulos & Petronijevic, 2013, p. 32).

“Schooling opens up valuable options for future schooling.” (Heckman, Humphries, & Gregory, 2016, p. 54). “Schooling has strong causal effects on market and non-market outcomes.” (Heckman, Humphries, & Gregory, 2016, p. 56).

Discussing change over time, Emmons, Kent, and Rickets (2019) reported that “increasing income and wealth premiums in aggregate data associated with families whose heads have a bachelor’s or higher over families whose heads have no postsecondary degree are misleading. Comparing bachelor’s degree and postgraduate families to nongraduate families of the same race and ethnicity born in the same decade, we confirmed that the income premium generally remains positive for all birth decades between the 1930s and the 1980s.” (p. 317).

References

Card, D. (2001). Estimating the return to schooling: Progress on some persistent econometric problems. Econometrica, 69(5), 1127–1160. (Online).

Dale, S. B., & Krueger, A. B. (2002). Estimating the payoff to attending a more selective college: An application of selection on observables and unobservables. The Quarterly Journal of Economics, 117(4), 1491–1527. (Online).

Emmons, W. R., Kent, A. H., & Ricketts, L. (2019). Is college still worth it? The new calculus of falling returns. The New Calculus of Falling Returns, 297–329. (Online).

Heckman, J. J., Humphries, J. E., & Veramendi, G. (2018). Returns to education: The causal effects of education on earnings, health, and smoking. Journal of Political Economy, 126(S1), S197-S246. (Online).

Kane, T. J., & Rouse, C. E. (1993). Labor market returns to two-and four-year colleges: is a credit a credit and do degrees matter? (No. w4268). National Bureau of Economic Research.

Oreopoulos, P., & Salvanes, K. G. (2011). Priceless: The nonpecuniary benefits of schooling. Journal of Economic perspectives, 25(1), 159–84. (Online).

Oreopoulos, P., & Petronijevic, U. (2013). Making college worth it: A review of research on the returns to higher education. National Bureau of Economic Research Working Paper Series, (w19053). (Online).

The Faculty

Stories about life in academia

Sign up for The Lounge

By The Faculty

A weekly digest of what is happening in Education, Science, and Academia around the globe. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Adam Ross Nelson

Written by

I mentor new (💫) and aspiring data scientists enter (🚪) and level up (📈) in the field. I help data pros find work they love (❤️) and that loves them back.

The Faculty

A community of academics and storytellers writing and sharing thoughts about teaching, learning, research, and life at the faculty.

Adam Ross Nelson

Written by

I mentor new (💫) and aspiring data scientists enter (🚪) and level up (📈) in the field. I help data pros find work they love (❤️) and that loves them back.

The Faculty

A community of academics and storytellers writing and sharing thoughts about teaching, learning, research, and life at the faculty.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store