Federated Learning for dummies.

Published in

Analytics Vidhya

4 min readMar 22, 2021

Introduction

In my opinion it’s useful to know why certain technology or algorithm has been developed. So whenever there is new technology or algorithm that I want to study I often ask my self these questions: What limitations did prior algorithm or technique face that led to development of such algorithm or tech? How does it work? and Is there a problem it didn’t solve or did new problem arise?

We will take the same approach and quickly learn about why Federated Learning were developed.

What limitations did prior algorithm or technique face that led to development of such algorithm or tech?

Standard machine learning collects all necessary data and model in one place for training. Here two main problems faced by standard machine learning practice:

Private information from individual devices are sent to the server on where model is trained.
Transmission of raw data to and from server can hurt user experience due to: bad internet connection, network latency, and more.

To mitigate above two issues Google developed Federated Learning which is based on core idea of decentralized learning.

How it works

Main concepts of Federated learning can be explained by following six steps:

1.Each mobile device downloads shared model from the server

2. Improve model by using data in your own device

3. Summarize changes(weights, bias, and other parameters of the model) that happened to the model.

4. Send summarized change(encrypted) back to the server where we downloaded our initial model

5. Average (technique called Federated Averaging) with other users’ summarized changed to update shared model.

6. Repeat steps 1–5 to continue improving shared model.

Before you go and start using Federated Learning there are two important things you must consider:

Model is NOT distributed to all users

Must check if user’s device is: Idle(Not active), plugged in, on wifi. If these three conditions are met, the user’s device is available.
But not all available device have sufficient size of data that is suitable for training, meaning not enough data to update the model.
users’ device that are available and are suitable are referred to as suitable available.
Model is deployed/distributed/downloaded to only suitable available users

2. Secure aggregation protocol

For additional security google makes the server aggregate summarized changes if more than 100 or 1000 device has given their summarized changes(i.e. more than 100, 1000 squares).
This ensures no individual devices’ summarized updated is inspected (This is still on development for production phase according to GoogleAI blog in Apr.6/2017)

What are some challenges

Slow upload speed

Upload speeds are typically slower than download speeds. However Google has come up with way to compress summarized model changes using random rotation and quantization.

2. Communication cost

Federated network comprises of large number of devices which are slower than local computation.
Must reduce number of communication rounds and size of transmitted messages. I.e. sending small size model updates less often.

3. Systems Heterogeneity

Devices in Federated network are different in many ways (CPU, memory, network connectivity, etc…) which leads to only small fraction of device being suitable available and staying that way for the whole training process instead of dropping out.
Therefore federated learning must anticipate low amount of participation, tolerate different hardware, and be robust to devices dropping out.

4. Computation in own device

Federated Learning requires device to train model, therefore mobile devices lacking compute power may not be possible.

Conclusion

In conclusion it was just a simple introduction to high level concepts of Federated learning which is gaining lots of spotlight and is area of active research. As it is relatively new field of machine learning there are lots of chances to participate in the development. Hope this was a good stepping stone to Federated Learning, there are many good resources out there for those who want more in-depth intuition. Anyways thanks for reading my blog and please comment if there are any misinformation or other resources you want to share! :)

References:

Federated Learning for dummies.

Written by Haneul Kim