Predicting cancellations in visit-by-appointment systems with TensorFlow
Visit-by-appointment systems offer many advantages to both patients and doctors, where time is maximized and used by both parties. The endless waiting by the patients is over, whilst instead the doctor can prepare for personalized attention and make an efficient organization of work, directly and positively affecting their productivity.
But, are we getting the most out of the appointment system?
One of the problems that we have detected in health centers and private consultations that are attending by appointments, is the loss of that space (slot) or time; either due to the non-presentation of the patient, known as a no-show, or due to the prior cancellation of the appointment, which, even if it had been with a certain advance, does not reoccupy that space with another patient.
Anticipating and predicting possible cancellations can be beneficial for better efficiency in visit-by-appointment systems. Having a predictive model that can calculate when this is most likely to occur and can be of great help to the health-care professional. Knowing these patients with increased tendency of canceling, it is possible to take some actions in advance. For example: trying to turn the cancellation into a schedule change (reschedule), allowing overbooking, or prioritize patients with less cancellation probability.
In this article, we have decided to focus on the prediction of early cancellations by the patient and not on no-shows. The reason is given by a fundamental difference between these two, early cancellations are always registered in our system, it is the patient himself who does the action; while the registration of no-shows depends on the specialist notifying after the visit, which does not always happen.
That is why the no-shows are more difficult to predict since we do not have reliable data to do a proper analysis.
Method
For the development of this model, we have used a subset of the appointments booked online on the znanylekarz.pl for one year. Due to the large volume, it was decided to select only a subset. Appointments that were created more than 14 days in advance were also discarded, analyzing those cancellations that could have a greater negative impact. As we will see below, the age of the appointment plays a weight factor for the prediction analysis. Maintaining only two weeks allows us to focus on appointments in the near future and also reduce outliers.
Data description
Many of the models that currently exist for these predictions are based on a large number of variables for example:
- Demographic: gender, age, marital status, distance to the health center, city, country, etc.
- Appointment: day of the week, month, age of the appointment, reason for the visit, etc.
- Patient history: future visits, past visits, cancellation or no-show history, etc.
In our case study, many of these variables are unknown. Due to the origin of the data, the information available is limited.
Variables we had:
- Day of the week, month ,time
- Appointment duration, appointment age
- If an automatic appointment reminder was sent
- If the patient left comments for the doctor
- Price of the visit
- Doctor’s specialty
- Province where the consultation is located
- City where the consultation is located
- Whether the patient has insurance or not
- Number of previous cancellations, number of previous visits
- Cancel by the patient (Variable to be predicted)
Let’s do some data analysis to help us understand what is going on
Cancellations:
As can be seen in figure 1.0, patient cancellations are unbalanced, which represents a challenge for the classification algorithms. As the number of patients who attend appointments are considerably greater than those who do not, the accuracy measure can lead us to erroneous conclusions.
Previous visits and previous cancellations:
As we can see in figure 2.0, previous appointments are clustered from 0 to 80 and seem to be related to the number of cancellations. It also appears that as previous visits increase, cancellations decrease on average. In this case, we have eliminated some outliers so that they do not interfere with the prediction.
Month:
Figure 3.0 shows the number of appointments and cancellations by months. Does not look like there is a direct correlation that can help us to predict future cancellations. Also as we mentioned before we took only one year sample so it could take us to under-fit our model. Finally, we exclude the variable Month from the model.
Weekday and time:
However, in figure 4.0, shows the combination of time and day of the week does seem to follow some patterns. It is visible that days such Saturday and Sunday, cancellations are more likely or that cancellations are triggered in the early hours of the morning and late at night.
Appointment age:
Figure 5.0 shows how the number of cancellations is distributed taking into account the age of the appointment (the age of the appointment is the number of days in advance with which it was booked) It Is easy to see the direct relationship between the age of the appointment and the cancellations. It gives us to think that it could be a weight variable with a high correlation with cancellations.
Doctor’s specialty:
We noted that patient cancellations were related to the specialty of the doctor. Some specialties had a much lower cancellation percentage than others. In figure 6.0 we observe that the cancellations seem to follow a normal distribution.
Because of the large number of specialties and being a category type variable we reduce the dimension of the variable to 3.
Specialty Type 1: Where the historical cancellation percentage is less than 22%
Specialty Type 2: Where the historical cancellation percentage is greater than 22% and less than 27%
Specialty Type 3: Where the historical cancellation percentage greater than 27%
Other variables:
Some of the selected variables were not very useful and showed a low correlation with cancellations. An example was: If the patient left comments for the doctor, although in the first instance it could be thought that a patient who took his time to left a comment with a reason for the appointment, could be a symptom of interest and present a lower cancellation rate. In practice it proved not to be the case; Also, the number of comments was not significant for the sample. Other unused variables were province and city; these were not very useful due to the number of undefined values that were available. The price of the appointment also did not prove to be very helpful.
Variables used
Of all the variables we had, we decided to use:
- Day of the week
- Time
- Appointment age
- If an automatic appointment reminder was sent
- Doctor’s specialty
- Whether the patient has insurance or not
- Number of previous cancellations
- Number of previous visits
Implementing a binary classifier with TensorFlow
With the data filtered and cleaned, we can start with the definition of our model.
Let’s first split our datasets into training and testing. We will use 70% to train and 30% to test the model:
Second, we scaled the data in a range of [0.1]. This will allow us to converge our model faster than preserving the original values. For this, we will use the MinMaxScaler function which maintains the correct distribution of the data. Keep in mind that both training and test data must use the same scaler instance.
As we saw previously, the variable that we want to predict is not correctly balanced (Figure 1.0). To fix this issue we use an oversampling technique called SMOTE from Synthetic Minority Over-sampling Technique. The imblearn library has good implementations for smoothing techniques.
With the scaled training and test data we are almost ready, we then proceed to model the classifier. We have used Keras and Tensorflow 2 for the simplicity it provides to define a neural network.
As we can see in our model we have used sigmoid as an activation function of our last layer and binary_crossentropy as the function of loss. This measures the differences between two probability distributions, in our case (y, 1-y).
We have also added 3 Dropout instances in the hidden layers and the input layer. The Dropout layer randomly sets the output of a neuron to 0 in 20% (0.2) of the cases, in this way we reduce the over-fitting of our model.
All ready to train and predict.
To evaluate the accuracy of our classifications we use the confucion_matrix. The confusion matrix will summarize the results of testing the algorithm as follow:
- True positive (TP)
- True negative (TN)
- False positive (FP)
- False negative (FN)
Results
Below we can see the confusion_matrix and the values of some of the most well known measures.
Note: For us it is essential to reduce false positives since it influences a doctor’s decision when making decisions to mitigate cancellations.
Recommendations for future Research
As we have seen, the predictions offer acceptable results, although there is room for improvement. There are also more variables that we could try, for example the distance to the consultation, or the gender of the patient to name just a few. Due to the nature of our source we did not have those
Another issue to consider is that in this model we have used an absolute value of the total of previous cancellations, without considering the relationship between cancellations and total visits, as well as we did not take into account whether they were recent or not.
We also consider interesting to be able to compare the results obtained to the use of others binary classifications models.
