Using machine learning to focus advisor efforts each week
At our college, we are working to improve the retention of students. To accomplish this, our advisors are going to be more actively managing students. However, some advisors are assigned upwards of 800 students to advise each semester. To help advisors focus their efforts each week on the students who may be most at risk of not retaining, we are using a machine learning algorithm.
We called this project “Behavioral Alerts” based on what the algorithm focuses on. Instead of using historical information or demographics to in the algorithm determine which students are at risk, we focus on behaviors that we can help the student improve.
These features that we are watching for each student are mainly coming from our learning management system: logins, page views in the class, assignments missed, assignment grades, number of days prior to an assignment due date that they submit the assignment, etc. For each of these features, we check against four views of that metric:
1) The literal value
2) The change from last week
3) The individual’s average
4) The class section average
We look at the metric in these four ways because we want to find all kinds of “risky” behavior that could lead to retention issues such as:
- The student’s behavior has changed from last week to this week, possibly signaling a change in their life.
- The student has continually been performing poorly in this class and is at risk of not succeeding.
These metrics designed this way will also help eliminate issues where the instructor might not be using the learning management system consistently which could create false positive reports.
This algorithm runs for all students on a weekly basis and saves the data into a table in our data warehouse. We then have an application built in Oracle APEX that displays the results to our advising team. The application will show only the students who the algorithm decided need focus this week. We also add other information about the student to give the advisor an informative view of the student without needing to look up details elsewhere. This other information includes their entire class schedule and any notes or flags in our advising management tool.
Because we felt it is important to provide the advisors with information about why the student was being flagged (because you can’t just call a student and say that computer told you that they are struggling), we felt it was important that our algorithm is not a black box. To accomplish the transparency that we desired of showing what features caused a student to be flagged, we used Shapley values, which is essentially a process that runs through every iteration for every student. This is why our algorithm runs weekly instead of nightly because to run through 20,000 students takes about 8 hours of processing time in our current environment.
Quick Technical Details:
Algorithm is a neural net written in python
Executed via Wherescape scheduler (our ETL tool)
Saved into Oracle database
Displayed via Oracle APEX to users
