What is Federated Learning?
Google Gboard App’s next-word prediction accuracy increased by 24% with the use of Federated Learning!
What is Federated Learning?
Lots and lots of data is being generated today on mobile phones and IoT devices (also termed as “data on edge devices”). This data provides benefits such as low-latency, privacy, and offline availability for many actions. However,
when it comes to Machine Learning(ML) and Analytics on data distributed across those devices, the usual practice for training ML models is to collect that data across different cloud or local storages at one central machine or repository. But:
- How to regulate data sharing policies while gathering the data from heterogeneous locations?
- How to ensure privacy when the data to be shared is sensitive such as healthcare data?
- How to arrange an infrastructure that has enormous storage space for all the data and security to avoid any breaches?
- How to build a robust predictive model that is trained on a variety of data samples (both in quantity and quality)?
The answer is “Federated Learning” — Machine Learning on decentralized data! (Also known as “Collaborative Learning”)
Instead of the conventional approach of gathering all local datasets in one place, Federated Learning aims at training the ML model across multiple Edge devices (On-Device Training) such as mobile phones on local data samples!
(Without exchanging the data samples & keeping the training data on the device itself)
How does it work?
A) The devices download the shared prediction model which is trained on a proxy-data at first
B) The devices improves the downloaded model by learning from the local data available on the edge devices (mobile phones)
C) Summarizes the changes as a small update and only this update is sent to the cloud where it is averaged instantly with the updates from other users as well and that improves the shared prediction model
(This iterative process is done until a high-quality model is developed.)
D) So, all the training data stays on the device and no individual updates are stored in the cloud which maintains privacy too!
Use-Cases:
Recommendation engines, Fraud Detection Models, and Healthcare Models are the majority use-cases of Federated Learning.
Google’s Gboard (Google Keyboard) also uses Federated Learning.
When we see a recommendation on Gboard, our phone locally stores our actions (whether we clicked on the recommendation query or not) and that on-device history of data suggests improvements to the shared prediction model.
As mentioned previously, this technology has to be deployed to millions of phones which are heterogeneous and running Gboard, which is possible by on-device training that uses a miniature version of TensorFlow.
Well, the training only happens when the device is idle, plugged in on a wireless connection to avoid any impact on the phone’s performance!
Fact: Gboard learns new words and phrases without sending the text we speak or type to Google, only what is learned by Gboard learns might be sent to Google services! Well, that’s the level of data privacy by Google!
Benefits:
- Privacy — The data remains on the user’s device itself
- Low latency — As the model trained is available locally and can be used to generate predictions
- Accurate predictions — The model is trained on local data on top of the proxy data, so the results come out to be more accurate and personalized.
- Low power-consumption — The local model training helps in power consumption optimization
- Minimal hardware — Reduces the amount of hardware infrastructure needed and runs on low-end hardware devices such as mobile phones.
Challenges:
- Communication: Federated networks are generally comprised of many devices and communication in that network can be slower than local computation so effective and inexpensive communication needs to be done!
- Data heterogeneity — Different local datasets may add bias and there will be some difference in the dataset sizes too!
- Temporal heterogeneity — Each dataset’s distribution may vary with time!
- Loss of updates — Partial or Total loss of model updates due to device failure may affect the shared prediction model.
References: Nvidia, Google AI Blog, Unit.AI, Wikipedia, Google I/O
