A few sources on assessment metrics in adult and CBE programs

I’m in the process of writing a chapter for a forthcoming book on competency-based education and non-traditional learning environments and specifically looking at some new ideas on how to measure work in these settings. Here are a few articles and sources which showed up in my literature review, with some thoughts on relevance.

Brower, A., & Kallio, S., et al. (2017). Measuring what matters. The UW Flexible Option’s framework to measure success from the student vantage point. In K. Rasmussen, P. Northrup, & R. Colson (Eds.) Handbook of Research on Competency-Based Education in University Settings, 268–282. Hershey, PA.: IGI Global. doi:10.4018/978–1–5225–0932–5.ch014

This chapter, part of an academic book looking at competency-based, post-secondary education, demonstrates that competency-based education requires a different set of student success metrics, and introduces the metrics framework developed by and for the University of Wisconsin’s UW Flexible Option (UW Flex). The audience is policymakers, educators, and administrators exploring university-level competence curriculum, viewed through the lens of the UW Flex experiment underwritten in part by Lumina Foundation. The work explores the success metrics around non-traditional post-secondary education courses of study, specifically movements away from the credit hour dynamic. Aaron Brower is provost and vice chancellor at the University of Wisconsin-Extension and Sandra Kallio is a senior writer in the University of Wisconsin system. The work is directly related to questions about competence programs I am exploring, as well as the work being done by Lumina in this arena. It is highly useful contextual information as it explores the lessons learned so far in this new partnership and instructional model.

Bureau of Labor Statistics. (2017, October 26). Occupations typically requiring postsecondary education gained 5.3 million jobs, May 2007–16. The Economics Daily. Retrieved October 21, 2018, from https://www.bls.gov/opub/ted/2017/occupations-typically-requiring-postsecondary-education-gained-5-3-million-jobs-may-2007-16.htm

The source is data from the Bureau of Labor Statistics, the fact-finding agency and information clearinghouse for the United States Department of Labor. This item is a snapshot breakdown of data around postsecondary educational requirements by U.S. industry and business from 2007 to 2016. It is released for policymakers, journalists, educators, business analysts, and general public consumption as a U.S. government service. The report is not intended to answer a question, per se, as the government collects myriad data on nearly limitless subjects, but rather is packaged to show trends in educational attainment and hiring over a 10-year period in order to add context to the broader public discussion. The data is gathered via the Occupational Statistics Program, which “produces employment and wage estimates annually for over 800 occupations” based on information collected by the government through a variety of sources and collection methods. There is no single author credited with this report as the authority is derived via the U.S. government as a whole. The information is useful in that is speaks to educational requirements for various industries, especially as it pertains to postsecondary education and the need to further training for career advancement. It provides a contextual backdrop around adult learning needs in that it provides statistical heft to the need by industry sector of a well-trained and education employment pool.

Center for Community College Student Engagement. (2012). A matter of degrees: Promising practices for community college student success (A First Look). Austin, TX: The University of Texas at Austin, Community College Leadership Program.

This information comes via a website, though one with scholarly ties and is intended for educators, students, academics, researchers, and policymakers interested in the development and maintenance of community college systems and curriculum. This report is a synthesis of best practices and research findings with regard to the changing needs and demands for community colleges, in particular, because of the economic structural upheaval which occurred around the collapse of the U.S. housing market and the unique situation created for postsecondary educators serving this particular student community. The data and findings presented in this update were collected via surveys administered across academic terms to students, faculty, and administrators of 900 institutions nationally and utilizes qualitative and quantitative data sets. The Center for Community College Engagement is a project undertaken at the University of Texas at Austin charged with “survey research, focus group work, and related services for community and technical colleges interested in improving educational quality through strengthened student engagement and student success.” The Center assists in monitoring the nation’s community and technical colleges for qualitative and curriculum questions and performs a variety of frontline and support tasks around data accumulation with regard to these institutions, the student body as a whole, and educator support and information sharing. The information and findings here are relevant due to the importance of community college education to adult and non-traditional learners and in particular offer contextual insight into the need served by these schools in relation to external demands and according to the needs expressed by the student base which they serve. The data and survey results offer valuable insights into who is being served, why, and how and work to inform educational content theory with practical application potential.

Characteristics of postsecondary students. (2018, April). The condition of education. Retrieved October 3, 2018, from National Center for Education Statistics http://nces.ed.gov/programs/coe/indicator_csb.asp

This is a statistical abstract looking at the makeup of postsecondary degree students across 2- and 4-year institutions of various types, the intent of which is to codify who students seeking advanced degrees are in the current educational landscape. The data are presented as a contextual source of information for educators, academics, and policymakers, particularly those focused on the bridge for learners from high school attainment to more advanced academic pursuits. The information presented is an overview of student populations, broken down through multiple layers of age, socioeconomic, and educational background situations in order to show who is attending American colleges and universities and how, broadly speaking, they got to that level of learning and is part of a larger statistical look at secondary and postsecondary educational ecosystems. The data, culled from the Integrated Postsecondary Education Data System, is compiled as part of governmental research and is a useful snapshot of who the American student is as a whole, and, drilled down, where the system is defined with regard to adult and non-traditional learners, information which is of key interest to my research.

Hoxby, C. M. (2018, July). The productivity of U.S. postsecondary institutions (Working paper No. C13875). Retrieved September 27, 2018, from National Bureau of Economic Research website: http://www.nber.org/chapters/c13875.pdf

This NBER working paper is an analytical report intended to become a chapter of a book, Productivity in Higher Education, though this version was not the final iteration for that publication. It is information produced for academics, researchers, and policymakers with a focus on higher education. The paper is essentially a cost-benefit analysis of postsecondary education as a whole, but also a granular examination about whether the value proposition around a “selective” institution merits the expense of education compared to other budget-friendly alternatives and whether resource allocation and use between selective and non-selective institutions has an effect on productivity at the institution. The paper examines “productivity of a postsecondary institution is its causal effect a student’s lifetime outcomes (“value-added”) divided by the lifetime cost of producing this effect.” The author uses “administrative data on college assessment scores, score sending, postsecondary enrollment, and 2014 earnings from wages and salaries for people in the high school graduating classes of 1999 through 2003 who were aged 29 through 34 in 2014” and offers both quantitative and qualitative methods in order to develop and process data. Caroline M. Hoxby is a research-oriented economist and is the Scott and Donya Bommer Professor in economics at Stanford University and director of the Economics of Education Program for the National Bureau of Economic Research. Hoxby’s research here pertains to my question in that it addresses questions of qualitative equality with regard to the selection of and access to postsecondary institutions for adult and non-traditional learners and further adds context to the question of educational equality and access for underserved and marginalized communities. It adds value to my research in that it offers a better understanding of how institutional equivalency plays a role in preparing learners not just through obvious educational attainment, but also with regard to the preparation for earning potential through a structured, resourced program of learning.

Karmelita, C. E. (2018). Exploring the experiences of adult learners in a transition program. Journal of Adult and Continuing Education, 0(0) 1–24. 147797141879158. doi:10.1177/1477971418791587

This is a scholarly article looking at the informal support structures and relationships available to non-traditional, postsecondary adult learners. The information is geared toward adult learners themselves, education and industry professionals, and those who work with adult learners in support and counseling capacities. The paper looks at the perspectives and experiences of a small group of adult learners in transition programs, including how the relationships with each other formulated a supplemental support structure with effects on self-perception and use Schlossberg’s Transition Theory and Cross’s student barrier categorizations as a framework to examine student experiences. The data for this research study was: “observations, interviews with the instructors and participants, participant journals, class discussion posts, and class artifacts.” Narrative inquiry and hermeneutic analysis were used by the author to collect and analyze the data used. Dr. Courtney Karmelita holds her Ph.D. in adult education from Penn State University, where she works as the assistant director of orientation in the Division of Undergraduate Studies. She also has served as an academic adviser at the university and has a background in special education in public and private schools and holds a master’s degree in secondary education. This is a useful secondary source of information with regard to mindset and challenges adult learners and other non-traditional students may face with regard to emotional and personal support around what can be a difficult undertaking — especially where gaps may exist due to lack of foresight from existing mechanisms developed out of traditional educational models.

Lumina Foundation strategic plan for 2017 to 2020. (2017, April 10). Retrieved October 21, 2018, from https://www.luminafoundation.org/resources/lumina-foundation-strategic-plan-for-2017-to-2020

The Lumina Foundation is a private, non-profit organization focused on the evolution of postsecondary education and curricula strategies in the space. A large part of Lumina’s focus is directed toward its Goal 2025, an initiative to help elevate the postsecondary educational attainment so “that 60% of Americans hold degrees, certificates or other high-quality postsecondary credentials by 2025” in order to meet the needs of the business community by focusing in particular on adult and non-traditional learners. This report, a plan for Lumina’s goals and standards, is geared toward educators, institutions offering a focus on non-traditional, post-secondary students, and policymakers dealing with curriculum planning and financial assistance and education funding. Lumina’s work is relevant to my research because they are working to innovate how postsecondary education is seen and experienced in the United States as a necessary part of the post-industrial economy. The foundation’s work in various strata, including at the University of Wisconsin, offers data and results around postsecondary learning systems which can help drive conversation around adult learning and competency-based education programs.

Osborne, K. (2008). Chapter 2: Education and schooling: A relationship that can never be taken for granted. Yearbook of the National Society for the Study of Education, 107(1), 21–41. doi:10.1111/j.1744–7984.2008.00128.x

The source material here is a chapter from a scholarly annual publication and is intended for academics, educators, administrators, and policymakers concerned with learning environments. At the start of his chapter, the author quotes an aphorism attributed to several people through history about their education being interrupted by their schooling. He’s writing here about how the formalized nature of the classroom, with curriculum requirements and graded assignments, can, unintentionally, quash the quest for knowledge. This can be particularly true with regard to adult and other non-traditional learners who may be poorly served in their educational pursuits by the traditional trappings of desks and lectures and the other flotsam of educational best intents. The theory presented is intriguing because of the unique nature of a non-traditional learner base, often removed by years or even decades from the rote ritual of formalized schooling. The author, Ken Osborne, is a Canadian historian, former high school history teacher, and a professor emeritus of education at the University of Manitoba. Osborne’s writing is a useful reminder, with historical perspective, to remember that not all students are best served by educators from the same playbook.

Rivers, C. (2016, August 03). Competency-based education: The importance of metrics and data collection. Retrieved October 2, 2018, from https://evolllution.com/programming/program_planning/competency-based-education-the-importance-of-metrics-and-data-collection/

This article is written as an analytical piece for the online newspaper The EvoLLLution, a publication focused on postsecondary education policy, professionals, and students, and is written for those focused on accountability and metrics around educational attainment measurement in non-traditional learning environments. In particular, an effective framework for measuring effectiveness around competency-based programs for those involved in creating curriculum structures is the focus in this work — concentrating on questions on student demographics, program performance, and program efficiency as the parameters on importance. The main data used in the report is gathered in qualitative analysis from the University of Texas system in order to better define the basic demographic information and education attainment background for non-traditional postsecondary students in the system. Carlos Rivers is the Operations Research Analyst for the Institute for Competency-Based Education at Texas A&M University-Commerce and holds an MBA. The information presented is focused specifically on the UT system, but the questions of metrics and measurement around CBE programs and curricula is directly tied to broader questions being examined with regard to a national discussion on how to measure educational achievement by adult and non-traditional learners in higher education programs outside the traditional pipeline of service.

Soares, L. (2013, January). Post-traditional learners and the transformation of postsecondary education: A manifesto for college leaders. Retrieved October 2, 2018, from https://www.acenet.edu/news-room/Pages/Post-traditional-Learners-and-the-Transformation-of-Postsecondary-Ed.aspx

The source is a trade article produced as an analytical look for the American Council on Education as an analytical look at the state of non-traditional postsecondary education, intended for educators at the postsecondary level, policymakers, and higher education as an industry group from an advocacy perspective. The article is a “manifesto” by the author, pressing the needs for innovation in educational policy thinking to influence curriculum planning around the changing global economy, particularly with adult and non-traditional learners and postsecondary programs of study in mind. The author makes the argument that higher education is at a crossroads of opportunity to rethink and retool the postsecondary experience “not unlike the one that generated the emergence of community colleges and the English liberal college/German research university hybrid in the 19th century.” Louis Soares serves as a special policy advisor to the president of ACE, and is a policy and practice consultant providing executive insight for higher education leaders and a fellow at the Center for American Progress in Washington, D.C. Soares’ writing here is a useful summary of some of the challenges presenting for educators and institutions in the adult and postsecondary spaces, but also offers ideas and solutions to be considered within the contextual framework of the needs of employers and learners from practical and experiential perspectives. It’s useful as a guiding voice in researching the state of an industry tailored to non-traditional students, one which is experiencing a fair amount of introspective discussion and discovery around ideas of relevance and useful service to educational consumers.