Introduction to Federated Learning

What is Federated Learning?

In machine learning, we typically congregate all available data into a single location to be trained off of, which allows for a streamlined and efficient training process. However, given the need for digital privacy, sharing data to a central location is not always feasible.

In Federated Learning, a model is trained across multiple decentralized devices - each one with it’s own respective data samples that are not exchanged with any other device.

How Exactly Does it Work?

All edge devices, or nodes, are tasked with training a model on it’s own data. The parameters from each model are then merged together to build a global model that is representative of the data from all the nodes.

There are two primary forms of federated learning systems:

• Centralized Federated Learning: a central server is tasked with coordinating the training of all other nodes. This central node receives model parameters from other nodes and is responsible for compiling the global model, as well as deciding the global model architecture.
• Decentralized Federated Learning: nodes in the network communicate amongst themselves to build the global model, removing the need for a central server altogether.

Since only model parameters are being exchanged between nodes, the data for each node remains private. This is essential for building models in fields where digital privacy is the first consideration, such as cyber security and healthcare.

The Advantages

In addition to data privacy, federated learning offers many features not easily found in traditional machine learning.

• Multi-Task Learning: once the global model is compiled, it can be distributed to nodes in the network and paired with transfer learning to customize the model to that specific node’s needs.
  This new model has the advantage of having been pre-trained on data that is not available to that node. Often times, any single node does not have enough data by itself to train an effective model.
• Clustered Node Learning: multiple global models can be trained specifically for different clusters of nodes that share characteristics. If two nodes build local models that have similar parameters, it is likely that the data and needs of those two nodes are similar as well.
  Sharing global models between similar nodes can yield higher accuracy and customization than using a single global model.
• Homomorphic Encryption: this special scheme allows us to perform computations directly on encrypted data. With this, model parameters being sent between machines never have to be decrypted during the learning process. This builds another heavy layer of protection and privacy for the nodes involved.

The Drawbacks

Federated learning gives us our privacy, but at a cost. There are inherent disadvantages that come with using a federated scheme:

• High / Consistent bandwidth is required between nodes, as model parameters can be very large in size, and are in need of being shared with other nodes frequently.
• Local computing power is needed at every single node, especially in decentralized schemes.
• Each node needs to be independently prepared with framework and API interfaces to participate.
• Non-IID Data: in traditional machine learning, we can normalize our aggregate data set and correct for unbalanced distributions. In federated learning, each node’s data has it’s own unique sample distribution. Non-IID data, or Non Independently and Identically Distributed data is a major obstacle in training a global model, and can heavily affect training performance. The simplest example of this is one node having a larger data set than another node.
• Data privacy has the potential to render models susceptible to backdoor attacks. A malicious node can contribute intentionally harmful parameters to the global model, slowly teaching it a specific response to certain inputs. Each node needs to be vetted before it can be accepted into the node network.

Federated Learning is Here to Stay

Federated learning has quickly become the next step in computational intelligence, digital fairness, and privacy assurance. It addresses many of the problems associated with centralized machine learning, and puts the interests of the users first.

In a world that is increasingly focused on both data privacy and machine learning, it is essential to anticipate, and promote, the intersection of these two concepts.