The Learning Curves of Federated Learning with a Trusted Aggregator
Abstract
In this essay, we will explore the learning curves of federated learning with a trusted aggregator. Why? Because they give us a good understanding of what federated learning is. These learning curves were generated when training 4 datasets of 4 different hospitals. The datasets contain features and 2 diagnoses of 2 urinary diseases: Inflammation of urinary bladder and nephritis of renal pelvis origin. This medical dataset truly needs privacy! Because we cannot divulge the sexually-transmitted diseases of patients. So, what we learned about PySyft and OpenMined is applied in this project. Federated learning will protect the privacy of datasets in each hospital and at the same time, a more robust machine learning model will benefit all hospitals.
All the ideas in this essay are based on my final project for the Secure & Private AI Challenge (SPAIC), which you can study and download from this website:
Diagnosing Acute Inflammations of Bladder
https://github.com/aksht94/UdacityOpenSource/tree/master/Juan%20Carlos%20Kuri%20Pinto
Relevance & Potential
Machine learning projects are great to improve our world, to solve problems, and to take informed decisions. This project could help doctors to diagnose diseases of the urinary tract correctly. And then doctors could take the appropriate actions to cure such diseases. In fact, this machine learning system is 100% accurate; whereas human doctors can commit mistakes when diagnosing these 2 diseases. However, forgotten aspects of machine learning are security and privacy. This machine learning is not only very useful and accurate; but it also protects the privacy of datasets in each hospital by using federated learning. Moreover, some patients could be ashamed of their urinary diseases and can consult the diagnosis of this machine learning system before visiting a human doctor.
This machine learning system also has a lot of potential for the future. The code of this ML system is an useful pattern that can be copied and extrapolated to more complex kinds of diagnoses for other diseases. For example, we can change the logistic regression algorithm for a convolutional neural network capable of dealing with datasets of medical images. And the logic to protect the privacy of datasets in each hospital will be the same.
DATASET
Acute Inflammations Data Set
https://archive.ics.uci.edu/ml/datasets/Acute+Inflammations
Data Set Characteristics: Multivariate
Attribute Characteristics: Categorical, Integer
Associated Tasks: Classification
Number of Instances: 120
Number of Attributes: 6
Number of Diagnoses: 2
Missing Values? No
Area: Life
Date Donated: 2009–02–11
Source:
Jacek Czerniak, Ph.D., Assistant Professor
Systems Research Institute
Polish Academy of Sciences
Laboratory of Intelligent Systems
ul. Newelska 6, Room 218
01–447 Warszawa, Poland
e-mail: jacek.czerniak@ibspan.waw.pl or jczerniak@ukw.edu.pl
Data Set Information:
The main idea of this data set is to prepare the algorithm of the expert system, which will perform the presumptive diagnosis of two diseases of urinary system. It will be the example of diagnosing of the acute inflammations of urinary bladder and acute nephritises. For better understanding of the problem, let us consider definitions of both diseases given by medics.
Acute inflammation of urinary bladder is characterised by sudden occurrence of pains in the abdomen region and the urination in form of constant urine pushing, micturition pains and sometimes lack of urine keeping. Temperature of the body is rising, however most often not above 38C. The excreted urine is turbid and sometimes bloody. At proper treatment, symptoms decay usually within several days. However, there is inclination to returns. At persons with acute inflammation of urinary bladder, we should expect that the illness will turn into protracted form.
Acute nephritis of renal pelvis origin occurs considerably more often at women than at men. It begins with sudden fever, which reaches, and sometimes exceeds 40C. The fever is accompanied by shivers and one- or both-side lumbar pains, which are sometimes very strong. Symptoms of acute inflammation of urinary bladder appear very often. Quite not infrequently there are nausea and vomiting and spread pains of whole abdomen.
The data was created by a medical expert as a data set to test the expert system, which will perform the presumptive diagnosis of two diseases of urinary system. Each instance (each line of the text file) represents an potential patient. The data is in an ASCII file. Attributes are separated by TAB. Each line of the data file starts with a digit which tells the temperature of patient.
Attribute Example:
For example, ‘35,9 no no yes yes yes yes no’
Where there are 6 features and 2 diagnoses (8 columns):
‘35,9’ Temperature of patient
‘no’ Occurrence of nausea
‘no’ Lumbar pain
‘yes’ Urine pushing (continuous need for urination)
‘yes’ Micturition pains
‘yes’ Burning of urethra, itch, swelling of urethra outlet
‘yes’ decision: Inflammation of urinary bladder
‘no’ decision: Nephritis of renal pelvis origin
Attribute Information:
There are 6 features and 2 diagnoses (8 columns):
a1 Temperature of patient { 35C-42C }
a2 Occurrence of nausea { yes, no }
a3 Lumbar pain { yes, no }
a4 Urine pushing (continuous need for urination) { yes, no }
a5 Micturition pains { yes, no }
a6 Burning of urethra, itch, swelling of urethra outlet { yes, no }
d1 decision: Inflammation of urinary bladder { yes, no }
d2 decision: Nephritis of renal pelvis origin { yes, no }
Relevant Paper:
J.Czerniak, H.Zarzycki, Application of rough sets in the presumptive diagnosis of urinary system diseases, Artifical Inteligence and Security in Computing Systems, ACS’2002 9th International Conference Proceedings, Kluwer Academic Publishers, 2003, pp. 41–51
Federated Learning with a Trusted Aggregator
In this demo, there are 4 hospitals. The 4 hospitals cannot share the cases of their patients because they are competitors and it is necessary to protect the privacy of patients. Hence, the ML model will be learned in a federated way.
How? Federated learning is iterated 1000 times. At each iteration, a copy of the shared model is sent to all the 4 hospitals. Each hospital trains its own local model with its own local dataset, in 5 local iterations. Each local model improves a little bit in its own direction. Then we compute the local losses and local accuracies to keep track of them and to make graphs of them. We send the local models to the trusted aggregator that will average all the model updates. This averaged model is the shared model that is sent to all the 4 hospitals at the begining of each iteration.
In this way, only the ML model will be shared. Whereas the local cases of each hospital will be kept private and they will be used to train model updates in a local way. Federated learning will protect the privacy of datasets in each hospital and at the same time, we will generate a more robust machine learning model, which will benefit all hospitals. This shared ML model preserves the privacy of individual patients and at the same time, reveals important statistics of stereotypical cases.
The whole process is done in a trusted aggregator, in 1000 iterations. (We can vary the number of iterations.) At each iteration, a copy of the shared model is sent to all the 4 hospitals. Each hospital trains its own local model with its own local dataset, in 5 local iterations. (We can vary the number of local iterations.) Each local model improves a little bit in its own direction. Then we compute the local losses and local accuracies to keep track of them. So, we will able to create graphs of the learning curves: Training Losses versus Iterations and Training Accuracies versus Iterations. We send the local models to the trusted aggregator that will average all the model updates. This averaged model is the shared model that is sent to all the 4 hospitals at the begining of each iteration.
We train the machine learning model to diagnose the Inflammation of Urinary Bladder, in a federated way. As you can see in the graphs, the training losses of each hospital drop quickly to almost zero and the training accuracies of each hospital reach the 100%. The testing accuracy is also 100%. Notice that this machine learning system diagnoses this disease in a perfect way; whereas human doctors can commit mistakes.
We train the machine learning model to diagnose the Nephritis of Renal Pelvis Origin, in a federated way. As you can see in the graphs, the training losses of each hospital drop quickly to almost zero and the training accuracies of each hospital reach the 100%. The testing accuracy is also 100%. Notice that this machine learning system diagnoses this disease in a perfect way; whereas human doctors can commit mistakes.
The learning curves Training Losses versus Iterations and Training Accuracies versus Iterations have 4 colors for all 4 hospitals. Each graph has 4 curves of different colors: Blue, orange, green, and red. The curves are not lines; they are rather regions. Why? Because each iteration of federated learning is complex: First, 5 local iterations in each virtual worker (each hospital) to train each local model. Each local model improves a little bit in its own direction. Then, the 4 different models are sent to the trusted aggregator that averages them. Finally, the averaged model is sent back to the 4 hospitals. Such averaged model can have lower performance in comparison to the local models, which are more locally adapted to the local datasets. That’s why the progress in the learning curves goes back and forth. Moreover, the graph has 1000 iterations. That’s why the curves becomes regions. Because the curves go back and forth too often and are quite dense.
Thanks for your time!
I hope you have enjoyed the explanations of this machine learning system with federated learning.
