7 Levels of Success in Teaching: Rethinking Evaluations

See also my Prezi slides: 7 Levels of Success in Teaching

As a teacher and trainer for over eight years, when it comes to evaluating my teaching I have been working on extending a framework first introduced by Donald Kirkpatrick in the 1960s and popularized by his 1994 book Evaluating Training Programs. Rather than just a simple end-of-course form, Kirkpatrick recommended four different levels of evaluation. I have added on three more levels which I will describe further below, but first here is a summary of Kirkpatrick’s four levels:

Kirkpatrick’s Four Levels of Evaluating Training.
  • Level 1: Reactions
    How did participants react to the teaching at the end of the course?
  • Level 2: Learning
    What did participants learn by taking the course?
  • Level 3: Behavior
    What are participants doing differently because they took the course?
  • Level 4: Results
    What specific results are participants able to achieve after taking the course? How are those results translating to the success of their organization?

The first level, Reactions, is the most commonly evaluated level, usually through an end-of-semester course evaluation form like this sample from Oregon State University.

A typical Level 1 evaluation method (source: OSU).

These evaluations are often called “smile sheets” since they’re so heavily determined by whether students enjoyed their experience of the course. However, because they’re cost effective and efficient to administer, most institutions rarely evaluate their teaching at any of the other levels.

The second level, Learning, usually takes the form of graded quizzes or tests. Or, in courses where grades are not involved, this level can be evaluated using surveys done before and after the course.

The third level, Behavior, is harder to evaluate since it involves longer term behavioral changes that often cannot be quantified or measured. However, it can still be assessed through qualitative interviews, for example by contacting participants several weeks before the course and then again several weeks after.

The fourth level, Results, is more complicated—it depends on how participants define results and what specifically they expect to achieve. Assessing success at this level requires developing a longer term relationship with participants, well beyond the context of a specific course, in order to best understand what success requires in the contexts most meaningful for the participants themselves.


How These Levels Shape Teaching

Thinking about success at any of these levels changes how a teacher approaches a course. For example, since most institutions often only implement Level 1 metrics, most teachers might find themselves paying disproportionate attention just to the experiences of students in a particular course. This leads to quite a myopic short-term approach to success; since this level focuses so much on the reactions of students, the teaching approach also tends to be reactive. And that in turn sets up students as the ultimate deciders of what counts as success for teaching, leading to such student-centered websites like www.ratemyprofessors.com with grossly distorted ways of evaluating teaching:

How students view Level 1 evaluations.

Shifting the focus deeper can lead to quite different courses—and quite different ways of teaching. For example, if a teacher aims for Level 2 success, i.e. to ensure that students achieve the best Learning objectives through the course, that teacher will find themselves devoting considerable effort to building in learning assessments and content-focused modules in their courses.

Most teachers, of course, already feel that their teaching is about Level 2 success, but they are evaluated almost exclusively with Level 1 metrics that are not well suited to measuring a teacher’s Level 2 quality. And, since most students expect a high Level 1 experience from their courses, this leads to a significant disconnect between teachers and students when it comes to what each group expects from a course.

Whereas, let’s say a teacher aims for Level 3 success, i.e. focusing on behaviors or actions that students can do differently. That teacher will find ways of reaching out to students before and after the semester in order to get a better understanding of students’ relevant behaviors in particular contexts where the teacher wants the course to have the most transformative impacts.

Meanwhile, a teacher oriented toward Level 4 success takes a longer term and relational view of their teaching oriented toward results for students. A single course isn’t the teacher’s only focus; rather that teacher is thinking and working in terms of multiple semesters of courses, with a cohesive teaching philosophy oriented toward helping students achieve better results. And such a teacher proactively builds relationships with students that last well beyond a course in order to have longer term conversations about what “results” mean for students.


Going to Seven (and Changing the World)

However, what about levels beyond “results”? What innovative teaching approaches might be available if we extended these levels further? Over the past several years I have developed three additional levels. These deeper levels shape my teaching in more significant ways than just the four Kirkpatrick levels; they show up in almost every element and approach I use in how and why I teach. My extended model aligns the entire process of evaluating teaching around a broader purpose that seeks nothing less than to change the world:

Seven levels of success in teaching.
  • Level 5: Influence
    How are participants influencing their organization or the field they are working in? How are they making their perspectives heard and felt? How are they contributing to shaping broader success for their organization or their field in partnership with others?
  • Level 6: Leadership
    Going beyond influence, in what ways are participants becoming leaders of their organization or of their field? How are they leading and trailblazing new directions?
  • Level 7: Global Change
    How are participants working toward changing the world through their leadership of their particular organization or field?

This approach puts Level 1 evaluations in much-needed perspective: rather than treating them as the single most important metric in a teacher’s portfolio, this approach treats Level 1 evaluations as a scratch on the surface of what should be a longer-term multi-layered process. And that process, in turn, becomes shaped by careful attention to achieving success at each of these levels.

Each of the levels involves a more complex and longer term approach to measuring and evaluating success than the one before. Level 1 might be the simplest to evaluate, but it also should carry the least weight when it comes to assessing an overall teaching program. Levels 5 and 6 might seem impossibly difficult to assess, since at present it may seem like only anecdotal perspectives might be available for these levels. However, with attention and planning we can design thoughtful criteria that can be tracked over the long term to gather key insights from participants throughout their careers. Those criteria can then be woven in different ways within individual courses and teaching practices. As a result, a course built within a framework aiming for Level 5 or Level 6 success will look quite different from a course built purely for Level 1 or Level 2 success.

Thus, a course built for traditional Level 1 or Level 2 success might involve a lot of knowledge assessment and participation metrics, such as how well students do on quizzes, how often students participated in key course activities, attendance measures, etc. But a course built for, say, Level 5 success will include elements designed to introduce students to models of influence, for example by organizing class visits by guest speakers who are experts in the field. If the course is part of a framework of evaluating Level 5 concretely, it can include ways of tracking how many students pick up opportunities to connect and network with such experts beyond the context of the course.

The key to all this, however, is Level 7. Achieving Level 7 success requires a cohesive and clear vision for what changing the world looks like. That vision in turn should motivate and shape how teaching happens at each of the previous six levels.


An Example of a Level 7 Vision: The Women in Public Service Project

The Women in Public Service Project has a clear, specific, and breathtaking Level 7 vision: they seek 50% representation of women in leadership around the world by 2050. Here is how they articulate this vision:

Through innovative research, dynamic learning institutes, and strategic peer-to-peer and two way mentoring, the WPSP is committed to a new global partnership aimed at reaching a minimum of 50 percent representation of women in public service by 2050 (“50 × 50”).
Source: http://womeninpublicservice.wilsoncenter.org/about-us/

What I love about this vision statement is the specificity and scope of how they envision achieving Level 7 success. Rather than just thinking in terms of individual courses or training programs, the WPSP is able to imagine a variety of methods and approaches: “innovative research, dynamic learning institutes, and strategic peer-to-peer and two way mentoring.” Similarly, rather than restricting their focus to a single institution or location, the WPSP commits to building “a new global partnership.” This means their Level 7 vision has the ability to include and cohere multiple players globally, in ways that allow for many different contributions to the broader vision’s success.

Among the partners involved in the WPSP are key teaching institutions like Smith College and the City College of New York. How would such a vision change the ways teaching might be evaluated at these institutions? For example, consider a single course taught at, say, Smith College. While the usual Level 1 approach might rely only on student evaluations of such a course, thinking of success more broadly oriented toward Level 7 will change key aspects of the course from the ground up.

First, in collaboration with faculty, the course itself will involve key elements of the WPSP vision in its design. For example, since the WPSP vision calls for “strategic peer-to-peer and two way mentoring”, the course might include specific activities designed to equip women taking the course with ways of making peer-to-peer mentoring more effective and relevant. Similarly, the course might invite key women leaders to volunteer as mentors to students in setting up mentoring relationships that extend beyond the duration of the course.

Second, course elements that relate to the WPSP vision can be tracked—initiatives such as peer-to-peer mentoring don’t have to be ‘merely’ anecdotal, rather the anecdotes themselves can be part of a portfolio of broader assessment of how that one course helps toward success at all seven levels. Without the WPSP vision to provide guidance, success for the course might revert to tracking such short-term Level 1 goals like student enrollment and participation, or even shorter-term Level 2 goals like knowledge retention. When the WPSP vision is integrated into the course, the course experience can become significantly enhanced and more meaningful, for students and teacher alike.


So What?

What this model has done for me is pose the really challenging question: what is MY level 7 vision for success in my teaching? And, I would say, this model poses a similar challenge for anyone interested in evaluating their teaching. I am still crafting what my vision is; I am inspired by the WPSP vision to develop something similarly specific and breathtaking. For now, here is what I have, even though it lacks specificity and scope:

To coach and mentor everyone I encounter as potential world changers, by teaching radical inclusion through a lifelong framework built on joy, love, faith, and hope. And the greatest of these is love.

What do you think? If you are a teacher, what does this model offer in terms of crafting what success means for you? If you are someone evaluating teachers, what does this model offer in terms of rethinking ways you might do so differently? And if you are a student, how do you feel this kind of framework might change your learning experiences?


Notes

  1. Other researchers in organizational development have proposed extensions to Kirkpatrick’s model:
    - In 1991, Kaufman, Keller, and Watkins developed a fifth level that focuses on “societal change” similar to my thinking around level 7.
    - In the 1970s, Jack Phillips also proposed a fifth level but focused on “value for money” or a return on investment approach.
    - In 2002, Tamkin, Yarnall, and Kerrin published a useful review of other extensions and models similar to Kirkpatrick’s.