Transfer learning for tabular data
The summary of the metaMIMIC article regarding hyperparameter transferability between medical domain prediction tasks
What is it all about?
A heuristic approach to problem-solving is natural for humans. We instinctively look for analogies and adapt previously acquired experience and skills to create practical solutions to newly encountered problems. This is one of the aspects of human thinking that make up the level of intelligence of our species that is unparalleled anywhere else.
Transfer learning, the idea of using the knowledge gathered while solving past machine learning problems in order to model faster and more effectively in the future, is one example of this, as it directly refers to the phenomenon described in the previous paragraph.
Transfer learning is widely used in deep learning, especially for computer vision or natural language processing tasks. For example, weights of a neural network trained to recognise cats in photographs can provide a much better starting point for solving a task of identifying dogs than random initialisation. However, there is little research on transfer learning concerning tabular data. Our research aims to fill this niche by exploring hyperparameter transfer capabilities.
Hyperparameter optimization and transferability
At the forefront of algorithms that achieve the best results for tasks based on tabular data are the so-called boosting algorithms, of which the XGBoost algorithm is perhaps the best-known example. However, their effectiveness is very sensitive to the values of a number of internal settings called hyperparameters, the tuning of which is computationally complex.
We seek to answer whether the resemblance of optimal hyperparameters is related to the structural similarity of the problems under consideration through the presented research. If this is indeed the case, it is natural to follow with how one can use it to solve the aforementioned optimisation tasks more efficiently, which we also attempt to do.
The philosophy of our research
The issues presented in the previous section are explored through extensive computational experiments. We create a library of prediction tasks from the medical domain with varying degrees of structural similarity. Then, we test the effectiveness of the XGBoost algorithm for each of them, each time considering a predefined grid of a thousand sets of hyperparameters. Finally, we analyse the results obtained regarding the similarity of the optimal configurations and the problems’ structure.
Furthermore, we test a hyperparameter optimisation method using rankings of configurations based on previously calculated performance metrics as a simple proof-of-concept. In our opinion, the results are promising.
A brief introduction to the MIMIC-IV database
The prediction tasks library mentioned above was based on the fourth version of the Medical Information Mart for Intensive Care — MIMIC-IV for short. It is a large, freely available database comprising de-identified health-related data from patients who were admitted to the critical care units of the Beth Israel Deaconess Medical Center. In MIMIC-IV, you can find information such as values of continuously monitored vital signs, results of laboratory tests, diagnoses made by doctors, or even medicines administered during the patient’s stay. More information about the database can be found here: https://physionet.org/content/mimiciv/1.0/.
The correlation between task similarity and hyperparameter transferability
In line with intuition, the results obtained suggest that there exists a positive relationship between the similarity of tasks and the transferability of their hyperparameters.
We simulated degrees of structural similarity by analysing real-life prediction tasks built on:
- exactly the same data
- different sets of observations but with the same predictors
- different sets of observations with varying number of predictors
To assess the power of hyperparameter transferability, we checked how many best hyperparameter sets are shared between pairs of tasks.
We compared those results to the ones obtained from the OpenML repository. What our analysis shows is that the number of common hyperparameter sets between tasks from the MIMIC-IV database is significantly higher, which suggests the existence of a correlation between task similarity and hyperparameter transferability.
Can we benefit from it?
With access to historical tuning data of models built on similar data to ours, we can speed up the search of the hyperparameter space by ranking the hyperparameter configurations and searching only the best ones. Just selecting the first best set for our task in most cases leads to improved performance to XGBoost defaults.
It is just a really simple approach to the use of hyperparameter transferability but it already brings good results. What we hope to see in the future is the development of better hyperparameter tuning methods based on transfer learning (e.g. Warm Starting Bayesian Optimization).
Conclusions
Overall, the similarity between prediction tasks results in better transferability of hyperparameters. The problem with utilising this relationship is that for most datasets, the hyperparameter tuning history is not available. We, therefore, call for the sharing of tuning results, which may lead to further developments in the area of hyperparameter transfer.
If you are interested and want to read more, we invite you to take a look at the metaMIMIC article, available here: https://github.com/ModelOriented/metaMIMIC/blob/main/preprint.pdf
The code allowing to reproduce the experiments can also be found in the repository linked above.
If you are interested in other posts about explainable, fair, and responsible ML, follow #ResponsibleML on Medium.
