FEDERATED LEARNING: THE FUTURE OF DISTRIBUTED MACHINE LEARNING

• By Lakshmi sathyan.

Google introduced Federated Learning (FL) in the year 2017. It is a specific category of machine learning wherein its models are trained using decentralized data available on devices like mobile phones, self-driving cars, etc. It allows us to do machine learning while keeping the data on-device. It is resilient and very much secure.

Federated Learning allows for smarter models, lower latency, and less power consumption, everything while ensuring privacy. And this approach has an important benefit, i.e. along with providing an update to the shared model, the improved model on our phone can also be used immediately, powering experiences personalized by the way we use our phone.

How does federated learning work?

The significant insight is to realize that the nodes, which are the sources of training data, are not only data storage devices, but also computers capable of training a model themselves. The federated solution takes advantage of this by training a model on each node.

The server first sends each node an instruction to train a model of a particular type, such as a linear model, a support vector machine (SVM), or, in the case of deep learning, a particular network architecture.

After receiving these instructions, each node trains the model on its subset of the training data. Many iterations of an algorithm would be required for the complete training of a model, such as gradient descent, but in federated learning, the nodes train their models for only a few iterations, which means that each node’s model is partially trained after following the server’s instruction.

The nodes then send their partially trained models back to the server. And, they do not send their training data back.

The server combines the partially trained models to form a federated model. The average of each coefficient is taken to combine these models, weighting by the amount of training data available on the corresponding node. This is known as federated averaging.

The combined or the averaged federated model is then transmitted back to the nodes, where it then replaces their local models and is used as the starting point for the next round of training. After many rounds, the federated model converges to a good global model. In each new, the nodes can acquire new training data. Some nodes may even drop out, and new ones may join.

How is traditional machine learning different from federated machine learning?

1)PRIVACY ISSUES:

Traditional machine learning is based on a centralized approach where the training data is aggregated on a single machine or a data-centre. The well-known big companies like Google, Facebook, Amazon, etc have been doing so for many years. They collect a large amount of data and store it in their data centers where the machine learning models are then trained. But unlike the traditional machine learning method federated learning uses a decentralized approach, which is not privacy-intrusive like the conventional way. In a centralized approach, one has to trade their privacy by sending their personal information stored in their mobile phones to the clouds which can be accessed by the companies owning them.

On the other hand, in federated learning, the user doesn’t have to share any kind of private data to the cloud, and it allows mobile phones located at various geographical locations to concertedly learn a machine learning model keeping all personal data on the device itself. Hence it is much safer and secure in comparison to the conventional method.

2) EXPENSIVE COMMUNICATION:

Federated networks are basically comprised of a large number of devices (e.g., smartphones, smart vehicles, etc), and communication in such networks can be very slower than local computation. It is also much more expensive than the classical data center environments.

3) SYSTEM HETEROGENEITY:

Storage and communicational capabilities of devices in federated networks may differ due to variations in network connection, power signals, hardware, etc. The network size and systems-related constraints on every device result in a small fraction of the devices being active at once.

What are the challenges in Federated learning?

Federated learning involves learning a single, global statistical model from data stored on potentially millions of remote devices, which is beyond the thinking capacity of humans. To be exact, the goal is typically to minimize the following objective function:

m is the total number of devices, Fk is the local objective function for the kth device, and pk specifies the relative impact of each device with pk≥0 and ∑mk=1pk=1.

The local objective function Fk is often defined as the empirical risk over local data. The relative impact of each device pk is user-defined, with two natural settings being pk=1/mor pk=nk/m, where n is the total number of samples over all devices. Even though this is a common federated learning objective, there do exist some other alternatives such as simultaneously learning distinct but related local models through multi-task learning where each and every device corresponds to a particular task.

The multi-task and meta-learning perspective enable personalized or device-specific modeling, which can be a natural approach to handle the statistical heterogeneity of the data.

What will enable the growth of Federated learning?

In the coming years, model building along with computation on the edge, based on Federated Learning and secured with Homomorphic Encryption will definitely raise the peak of federated learning’s growth. As a large number of mobile phones equipped with AI chips and tremendous computing power will be available in the market in the coming years ahead, many ML models will be able to run simultaneously and locally on these devices. Distributing the heavy-duty analytics and computations over smartphones “on the edge”, as opposed to central computing facilities, will exponentially reduce time to develop data products such as hyper-personalized recommendation engines, e-commerce pricing engines, etc. Enterprises will go with a distributed machine learning model building framework for taking advantage of faster model deployment and which provides a quicker response to fast-changing consumer behavior, besides at a highly reduced cost.

For machine learning programmers, this shift provides a thrilling opportunity to customize AI. It also opens up new ways for adopting new tools and paving their way to deal with large-scale ML problems.

Though model development, training, and evaluation with no direct access to or labeling of raw data seems challenging at first, but in emerging markets such as our country (India), where hyper-personalization and highly contextual recommendation engine will be the key for app adoption and e-commerce advertisement, will play a huge role in the tech market, which indeed comes under federated learning.

Conclusion

Federated learning will create a future in which we work collectively to apply machine learning to some of the toughest problems that humanity faces, with each retaining control over our own data. It has the capability to solve the problems that the most regulated, competitive, and profitable industries face.

REFERENCES:

https://ai.googleblog.com/2017/04/federated-learning-collaborative.html

https://federated.withgoogle.com/

https://blog.ml.cmu.edu/2019/11/12/federated-learning-challenges-methods-and-future-directions/