Using Analytics to Meet the Needs of Students in the 21st Century
Below is excerpted from a keynote address that I delivered on November 8, 2016 at Texas A&M at Texarkana for its National Distance Education Week Mini-Conference
Right now in the US, nearly a quarter of all undergraduate students — 4.5 million — are both first generation and low income.
Of these students, only 11% earn a bachelors degree in under 6 years. That’s compared to the rest of the population, which sees students graduate at a national rate of 55%. What this means is that 89% of first generation, low income students stop out, perpetuating a widespread pattern of socio-economic inequality.
Since 1970, bachelors degree attainment among those in the top income quartile in the US has steadily increased from 40.2% to 82.4 in 2009. By contrast, those in the bottom two income quartiles have seen only slight improvements: under an 8 point increase for the bottom two quartiles combined. In the US, a bachelors degree means a difference in lifetime earnings of more than 66% compared to those with only high school.
This widening achievement gap is bad for students’ quality of life, and it is bad for liberal democracy. But it also has a significant economic impact for the US. In 1990, the U.S. ranked first in the world in four-year degree attainment among 25–34 year olds; today, the U.S. ranks 12th. This drop in rankings is not just a point of national pride. Over the next decade, employment requiring education beyond a high school diploma is expected to grow faster than jobs that do not. Acknowledging the importance of post-secondary education to meet 21st century workforce demands, President Obama set a goal for the country to regain its number one position by 2020.
The large population of first generation and low income students is among the least likely to persist through to graduation. Meeting the needs of these students is a major challenge for US colleges and universities, but it is also a significant opportunity as these same institutions face economic, governmental, and reputational pressure to increase their retention and graduation rates. To put things in perspective, if we can eliminate the achievement gap for these students, and bring their graduation rates up to the same level as the rest of the student population, it would mean increasing the national retention rate to 55%, which amounts to producing nearly 2 million more graduates, and (I think) making the US once again first in the world in terms of per capita bachelor degree production.
Student success is not a student problem. Students do not end up in the condition of being ‘at risk’ merely because of their demographics. Instead, the very institutions that make students students (remember that one becomes a student only as a result of being admitted to an institution) and want to see them succeed are also erecting barriers to that success. Colleges and universities are complex bureaucracies that are incredibly challenging to navigate even by those from the most privileged of backgrounds. For first generation and low income students, the situation is made worse: they tend to lack the ‘know how’ necessary to navigate all of the administrative requirements of a college education, at the same time as institutions themselves — often unknowingly — systematically disadvantage ‘non-traditional’ students who may have to commute long distances, work at least part time, and/or support a family.
For example, in 2011 Georgia State University looked to the data and found that as many as 1000 students who were on track for graduation were dropped because of financial shortfalls of as little as $300. In response, the university created the Panther Retention grant program which offered students identified through the use of predictive analytics an average grant amount of $900. Since 2012, GSU has awarded more than 5,000 grants and has found that 60% of seniors who received the grant were either retained or graduated within a year.
Of particular relevance to distance education week, Lewis & Clark Community College was able to mine its institutional data to discover that students enrolled in online courses who came in with a GPA of under 2.7 were very unlikely to succeed. Upon interviewing these students, they further discovered that this was in part a function of a lack of technical literacy. As a result, the college established 2.7 as a minimum GPA necessary for enrollment in online courses, and implemented mandatory technology training for all those taking these courses.
Other recent discoveries made by other colleges and universities as a result of mining their data include:
- Students are not able to enroll in the courses they need to graduate, either because there are not enough sections to meet demand, or sections are not offered at times that work for students
- Many students are not satisfactorily completing required gateway courses, are repeating them over and over again, with little success, and so are prevented from persisting in their degree program
- Changes in major during the first two years are resulting in a significant number of excess credits, and a lower likelihood to complete in 4 years (if at all)
- Assumptions that students are making about requirements for entering a particular program (nursing, for example) based upon published entrance criteria are much lower than is required for entry in reality, resulting in many students either dropping out or switching major with a minimal number of transfer credits.
Increased access to educational data is allowing institutions to face themselves in the mirror, to identify structural barriers to student success, and mitigate their effects. But not every obstacle is easy to remove. It is therefore vital that, at the same time as institutions work to remove barriers to student success, they also provide students with the guidance they need to navigate the hurdles that remain and persist.
Proactive advisement works. A randomized controlled study published by Eric Bettinger and Rachel Baker in 2011 found that data-informed intensive coaching saw an increase in persistence of 5.2 percent, and that coached students were more than 3% more likely to be retained even two years later. This promising research is currently being validated as part of a massive 11 institution validation study funded through the department of education.
Why does proactive advising work? Where traditional walk-in models of academic advisement fall down is that they rely on students to actively seek out help. But the students who seek out help are already among the most likely to succeed, as they demonstrate by their willingness to proactively seek out help in the first place. The students who are in the most need of academic advisement, first generation and low income students, are also among the least likely to seek out help. Proactive advisement works because, thanks to our increased access to student data, advisors can identify students at crucial points in their academic journey as they begin to exhibit signs of risk, and offer help where it is needed but would not otherwise be asked for.
We know that proactive advising works. Despite this, Ithaka S+R has estimated that 34% of public universities in the us require students to see an advisor, and only 2% of institutions advise proactively on the basis of alerts. At institutions like Georgia State University, Integrated Planning and Advising Services (IPAS) and early alert systems are enabling advisors to make a significant difference in the lives of students, but there continues to be two major problems use of analytics in support of academic advisement today:
- It is a heavy burden on schools to radically change processes and models, not only in order to facilitate a proactive approach to student advisement, but also in order to adapt to large complicated systems that put a huge burden on otherwise immature advising programs. Remember that only 2% of schools are using this technology today. The schools that we seeing the biggest impact from using these technologies have been working with them since their infancy, and have co-adapted, maturing together. In many cases, introducing these technologies as full solutions to schools with traditional advising models is like putting old wine into new wine skins. Schools need to start reaching students now, but the state of more well- established systems is such that they often take an incredibly long time to implement, and even longer to operationalize relative to a robust set of advising practices and procedures. But we need to start doing a better job of reaching students now. Today. Not 18 months from now.
- Most of these systems are trigger-based, relying exclusively on information stored in the student information system. These capabilities are very important. They have allowed us for the first time a way to systematically and consistently identify students as they fail or drop an important course, perform sub-optimally in a gateway course, enroll in a course that does not apply to their degree, etc. For the first time, we are able to adopt a systematize approach to getting students back on track. But the fact that these are trigger based means that we need to wait for students to go off track before we have the information necessary to do anything about it. What if we could predict at risk behavior and proactively intervene before student go off track in the first place?
Analytics are not value neutral. What we are engaged in is a fundamentally ethical enterprise, and we need to take seriously the fact that educational technologies that are designed to change human behavior also assume sets of values that affect how teachers, learners, and administrators think about themselves and each other. The work being done in product development today is very important, but it also carries with it a great burden of responsibility.
At Blackboard, the development of new technologies is an opportunity to respond to the needs of higher education here and now. Notice that I say needs, and not demands. At times, the demands of the education market do not align with the needs of students, faculty, staff and institutions. We know, for example, that there is a high demand for something called ‘predictive analytics.’ But we also know that analytics alone are useless. It’s not data that makes a difference in students’ lives. It’s people, through good judgement and effective interventions. With this in mind, at Blackboard we think very carefully about the kinds of information that student success professionals need, and the ways in which this information is presented. It is easy as a ‘data nerd’ to want to give people all kinds of data, dashboards, and statistics, but too much power too easily introduces complexity, increases cognitive load, and leads to inaction. It’s not hard to create technologies that ‘do it all,’ but a high powered analytics system produced based on what is possible rather than based on what people need can get in the way of action rather than facilitate it.
I am proud of Blackboard Predict. To me, it is a powerful tool because it is the culmination of a lot of thinking about (1) problems facing students, (2) issues created by a disconnect between the maturation cycles of technology and institutions, and (3) areas in which we should refuse to compromise if we are going to serve students in the long term.
At Blackboard, we are addressing the problem of graduation and retention by supporting a strategy that works: proactive advisement. Using predictive analytics that make use of learning management system data in addition to student information system data, we are able to identify students at risk within the first few weeks of class and intervene before they go off track. Early interventions mean lower DFW rates, fewer wasted credits, lower student debt, and higher rates of degree completion.
We are addressing the problem of complexity in the educational technology market. Acknowledging that the high powered systems that are being used successfully at the Georgia States of the world place huge demands on institutions to change culture and overhaul practices, and that the demands of technology actually represent a huge burden for the 98% of institutions that are not currently engaged in data-informed advising, we developed Blackboard Predict with a view to simplicity. We want to facilitate action today even as we support institutions through the maturation process.
Lastly, we are committed to an audacious set of values, which are apparent in several of our design decisions. The Product management team for Blackboard Analytics are all educators first, and data nerds second.
Putting People First. When I say it’s about putting people first, I mean this in two ways. On the one hand, I mean that what we do (both the products we design and the ways in which we use them) should be guided by a fundamentally humanistic set of values. As Marshall McLuhan famously observed, information technology both embodies and shapes our values. We must constantly attend to what our values are as educators and ensure that our investments and practices are aligned with those values. On the other hand, I mean that technology should not be sought as a means to replace human teachers and advisors. Whether we are looking at learning analytics or adaptive learning technologies, what we see time and time again is that the thing that makes the difference is not the technology itself, but the set of practices that these technologies allow. Too often we hear about the power of big data and predictive analytics to improve student success rates. But what the language and media attention surrounding analytics in education obscures is the fact that what makes the difference is not the data itself, but the ways in which it is used. In both of these senses, what I want to underline is the fact that what we are engaged in is a fundamentally ethical exercise that we — both vendors and practitioners — need to take this very seriously.
Transparency. Predictive analytics are not magic. There is no ‘secret sauce.’ The highest quality results are achieved when vendors apply well-established data mining techniques to clean and well-defined institutional data. There is no advantage to using a proprietary technique or predictive model. If anything, a vendor’s refusal to share the model they have produced using your data should be reason for suspicion. If the model can’t be shared, it can’t be independently validated. If you don’t know the model along with its underlying assumptions, you also can’t meaningfully interpret its results. When it comes to educational data, trust needs to be earned and constantly renewed, not expected as a matter of course.
Flexibility. It is important that analytics tools and products complement existing workflows, and allow institutions the flexibility to change their practices when doing so is in the best interests of their students. What this means is that these tools should not demand significant institutional change in order to work. True, significant cultural and structural changes are likely to be in the best interest of the institution, and those changes may also increase the usefulness of a particular technology. It’s not a bad thing when cultural and technological change take place in the service of a larger institutional vision. But when the cart of educational technology begins to lead the horse of institutional culture, values and priorities begin to be misplaced. Part of our commitment to flexibility is our advocacy of Interoperability. An institution doesn’t want to be limited in its adoption decisions by pre-existing investments (the biggest ones being its student information and learning management systems). It also shouldn’t feel locked into a sub-optimal learning management system because a critical analytics tool doesn’t work with anything else. That’s why we built Blackboard Predict to be LMS and SIS agnostic.
I want to be clear here that this is not a sales pitch. I don’t mean to describe Blackboard Predict as a panacea. I don’t want to say that Blackboard ‘got it right,’ or that it is the best thing for every institution. As far as the way we are thinking about educational problems and the values we are bringing to the space, I do think that Blackboard Analytics is unique. But this uniqueness is not a good thing. I wish that our values and approach were shared much more widely.
Education in the US is in need of change, but this change is not vendors’ to make. Instead, it is important for practitioners like yourselves to consider your educational technology investments seriously. It is you responsibility to make sure that the values they represent are in line with your own as you work to educate and intervene in the lives of students. It is also vitally important that the technologies that you adopt are embedded within the context of a set of reflective practices that not only make effective use of technologies, but also consider whether certain technologies should be used at all. Lastly, it is up to you to make sure that the tail doesn’t wag the dog when it comes to institutional educational technology investments. I strongly believe that institutions should be uncompromising in terms of the demands they make of educational technology vendors. They should put pressure on vendors to be transparent about their data models and encourage a climate of cooperation and interoperability. For me, it is unacceptable to sacrifice what is best for students for the sake of a business model. Business models that make sense only at the expense of students are irresponsible and unsustainable. But vendors don’t always know what is working, what isn’t, and why. Although many of us used to be educators, and still consider ourselves as such, we are not on the front line like you, and so need your feedback and active cooperation as we shape the future of higher education together.
Originally published at Timothy D. Harfield.