Flower, FATE, PySyft & Co. — Federated Learning Frameworks in Python

A Comprehensive Comparison of Federated Learning Frameworks for 2023

Addition 2024: This is a snapshot of the state of FL frameworks as of January 2023. Many frameworks like Flower or FedML have grown ever since, adding features and addressing weak points. So please read this article keeping in mind some of its information (especially details) might be outdated.

Often, the most valuable data is also private and protected. For example, personal information must be protected to comply with laws, and internal company records are often classified. In situations where machine learning models are trained on multiple datasets with specific access restrictions, Federated Learning (FL) can be applied. FL is a distributed learning technique to train models across distributed datasets without the requirement to exchange or centralize the data. Modern FL frameworks orchestrate access to different data sites while ensuring the data privacy and protection requirements of the data owners.

This article is based on work for SUMEX AG, a leading provider of ERP and claims management software for the Swiss healthcare system, a subsidiary of ELCA. SUMEX is utilizing machine learning among other methods to detect errors and anomalies in medical invoices. Its next-generation analytics solution aims to train ML models on the data of multiple, potentially competing, insurers. We are implementing federated learning to protect patient records and business-critical information. In the process of developing a concept for federated learning during my internship at SUMEX, our team gained insights into many FL frameworks.

Whether you want to experiment with FL algorithms or create a product powered by FL, there are several frameworks available to choose from. In this blog post, we will outline the criteria that should be considered when selecting a FL framework and give an overview of the options that are currently available.

The currently most common FL frameworks

Introduction to Federated Learning

Federated learning is a distributed machine learning approach that allows for collaboration on machine learning projects without the need to share sensitive data, such as patient records or classified secrets. In this approach, data owners (clients) train models locally on their own data and share only statistics like model parameters or gradients with a centralized server, which are then aggregated to create a global model. After that, the global model is redistributed to the clients. This process repeats over several communication rounds to achieve convergence.

The Federated Learning workflow explained step by step. (source: Federated learning process central case by Jeromemetronome)

There are two main types of federated learning: horizontal and vertical. In horizontal federated learning, the data at different clients is in the same feature space but the samples are different, for example, standardized tumor images from different hospitals. In contrast, in vertical federated learning, the clients hold different features that belong to the same sample, such as the health insurance record and hospital data of the same patient.

Criteria for FL Frameworks

Before diving into the concrete analysis and comparison of FL frameworks, let’s point out some criteria that should be considered when choosing a FL framework. The core functionalities of a FL framework are client-side training, server-side aggregation, and most importantly communication. Additionally, a local simulation mode should be available. However, in other features the frameworks can differ heavily

• Model Support: FL frameworks can be limited to types of ML models: Almost every framework works with neural networks, while only a few support tree-based models like gradient-boosted decision trees or random forests. Additionally, some frameworks provide clustering and statistics about the global data set.
• ML Framework Support: On a practical side, it is important to consider which ML libraries are supported by the FL framework. While most FL frameworks support models made with TensorFlow and PyTorch, there are also framework-agnostic FL frameworks that work with any ML library.
• Expandability: Most FL frameworks have implemented standard FL models and aggregation algorithms, which can serve as a reference for adapting custom ML models to the federated setting. The complexity of adding your own model or aggregation algorithm will depend on the type of the model and the FL framework being used.
• Aggregation Algorithms: Federated averaging (FedAvg), the most common aggregation algorithm, is widely supported in FL frameworks. However, the specific algorithms offered by each framework may vary.
• Privacy Methods: FL inherently provides some level of security by transmitting statistics like gradients or parameters instead of the data itself. Nevertheless, information about the data can be retrieved from those statistics. In case more security is needed, FL frameworks offer additional privacy methods like differential privacy, homomorphic encryption, and secure multi-party computation / secure aggregation.
• Readiness for commercial usage: Not all FL frameworks are ready for real-world applications or have been used to create a business product before. If you are aiming for production, it may be helpful to select a framework that offers additional tools such as provisioning, deployment, orchestration, monitoring and serving. Notably, there is a tradeoff between those extra functionalities and expandability & usability.
• Device / OS support: FL client devices can vary from computers to phones or even IoT devices. While not every framework runs on every type of device, concerning computer OS, most frameworks either work with all Linux/Unix, macOS and Windows or provide a containerized application.

As FL is a relatively new concept, most of the frameworks are recently founded and under constant development. Therefore, some general criteria for new software apply:

• Project Maturity: Active development, solid funding, and long-term support are important.
• Open source: Most FL frameworks, and all discussed here, are open source.
• Good Documentation: The quality of the documentation varies. Working examples are very useful as a starting point for creating custom models.

FL Framework Overview

There are many federated learning (FL) frameworks available, each with its own focus and capabilities. Some are more specialized for research purposes, while others are geared towards creating business-ready products. This section will give a general overview in the list below. Note that all of these frameworks are open-source if not mentioned differently.

WARNING: This survey is as of January 2023. The frameworks develop and change quickly.

NVFlare

NVFlare is a business-ready FL framework by Nvidia. It serves as a backbone for NVidia Clara, a bundle of AI products for healthcare applications, and has been made open source in 2021. More details next section.

• Company: NVidia
• Pros: Suited for commercial use, framework-agnostic, offers tree-based models, tools for monitoring and provisioning
• Cons: Needs some time to get started
• Status 01/2023: Ready-to-use, 298 ⭐ GitHub

NVFlare (chart by author)

FATE

FATE was first introduced to the market in February 2019. Developed by WeBank, a subsidiary of Tencent in Shenzhen, FATE is one of the earliest open-source frameworks for FL that is ready for commercial use. More details next section.

• Company: WeBank
• Pros: Suited for commercial use, plenty of FL algorithms, including decision trees, various additional features, orchestration with Kubernetes, compatible with Spark clusters
• Cons: Difficult to extend, usability
• Status 01/2023: Ready-to-use, 4.8k ⭐ GitHub

FATE (chart by author)

Flower

Flower originated from a research project at the University of Oxford, and is now managed by adap gmbh, a German distributed AI company. Due to its background, Flower makes it relatively simple to customize, extend or override components. More details next section.

• Company: adap gmbh
• Pros: Easy to Use, Flexible, Framework-Agnostic
• Cons: Limited Extra Features
• Status 01/2023: Ready-to-use, 1.8k ⭐ GitHub

Flower (chart by author)

PySyft

PySyft is an open-source FL framework by the distributed computing initiative OpenMined. In a bigger context, PySyft is not only a FL framework, but also a remote data science platform. More details next section.

• Company: OpenMined
• Pros: Remote data science, differential privacy, large community
• Cons: Limited ML model support, documentation not up to date
• Status 01/2023: under construction, tutorials do not work with the current version, 8.5k ⭐ GitHub

PySyft (chart by author)

IBM FL Community Edition

This framework is a research-oriented community edition of IBM’s FL service. It supports various models, including federated clustering and decision trees, and even implements algorithms from recent research papers.

• Company: IBM
• Pros: Flexible, framework-agnostic, large variety of recent FL models and algorithms
• Cons: Not suited for commercial use
• Status 01/2023: Ready-to-use, 374 ⭐ GitHub

IBM FL Community Edition (chart by author)

OpenFL

OpenFL started out as a collaborative research project between the University of Pennsylvania and Intel to create the Federated Tumor Segmentation platform FeTS. It is actively maintained and enhanced ever since, making it ready for real-world applications.

• Company: Intel
• Pros: Suited for commercial use, framework-agnostic
• Cons: No tree-based models, no differential privacy, sparse documentation
• Status 01/2023: Ready-to-use, 402 ⭐ GitHub

OpenFL (chart by author)

TensorFlow Federated TFF

TFF was developed to expand existing TensorFlow/Keras models to the federated setting. It is supports Tensorboard, and allows for customized aggregation algorithms.

• Company: Google
• Pros: Well-suited for TensorFlow/Keras
• Cons: Limited to TensorFlow/Keras
• Status 01/2023: Ready-to-use, 2k ⭐ GitHub

TensorFlow Federated (chart by author)

FedML

FedML is an AI start up originated from the University of Southern California. Since a successful 1-st round of fundraising in March 2022, FedML is under active development working on the first official release, and has the potential to become a top-tier FL framework.

• Company: FedML
• Pros: Award-winning white paper, a working, but buggy demo, partial documentation
• Cons: No official release yet
• Status 01/2023: under construction, demo version running,
  2.3k ⭐ GitHub

FedML (chart by author)

There are more FL frameworks, some of them specialized on specific sectors, ML frameworks or models like tree-based models:

• Substra: Framework launched by biotech company Owkin for healthcare applications
• FedJAX: Federated Version of Google’s JAX ML framework, research-oriented
• FLUTE: Microsoft’s Federated Learning Utilities and Tools for Experimentation for large-scale FL simulations on top of Azure
• PaddleFL: FL Framework by Chinese search engine Baidu based on their FL library Paddle Paddle
• Specialized Frameworks on medical applications
  - Fed-BioMed
  - LEAF
• Not open source
  - IBM FL in IBM Cloud Pak
  - Sherpa.AI FL
• Specialized Frameworks on tree-based methods:
  - Federated xgboost as part of the xgboost package
  - FedTree
  - Secure xgboost
• Many research projects & papers (including combinations of FL and blockchain)

NVFlare, FATE, Flower, and PySyft in Detail

There are four frameworks that we tested and would like to present in detail. Two business-ready solutions, NVFlare & FATE, the easily extendible lightweight framework, Flower, and the remote data science platform PySyft.

NVFlare — FL on Industry Level

NVFlare is a business-ready FL framework by Nvidia. It serves as a backbone for NVidia Clara, a bundle of AI products for healthcare applications, and has been made open source in 2021. It supports a variety of models, including neural networks, tree-based models, and other statistics. NVFlare is framework-agnostic, meaning that nearly every ML model can be easily migrated to the federated setting. The framework is designed to develop business products and therefore provides a bunch of additional features. It is possible to create an FL project that connects server, client sites, and users with different roles in a network. Depending on their role, users upload FL jobs, perform training or evaluation and monitor through the command line interface or the NVFlare Dashboard. Jobs can be scheduled or even run in parallel, orchestration is only possible with docker-compose. Security-wise NVFlare provides server-client authentication and a filtering layer that can apply privacy methods like differential privacy or homomorphic encryption to the payload.
One of the main goals of NVFlare is to facilitate the transition from a centralized ML model to an FL model. Therefore, NVFlare provides working examples for various models as a blueprint. In case a different model needs to be implemented, the API can be extended easily and configured to be used in workflows. In that regard, NVFlare is highly flexible, although there is a learning curve to understanding the architecture. Lighter frameworks like Flower make the transition from centralized to federated even easier.
All in all, NVFlare is a business-ready, ML framework agnostic FL framework. It is highly flexible, but a little effort might be needed to get the hang of it.

FATE — FL on Industry Level

The Federated AI Technology Enabler Framework (FATE) is offering similar functionalities as NVFLare. With its first release in February 2019, it was one of the first business-ready FL frameworks. FATE is an open-source project by the Chinese private bank WeBank, a subsidiary firm of Tencent, based in Shenzhen. The framework comes with a large number of modules including preprocessing, different machine learning algorithms such as gradient-boosted decision trees, statistics, privacy methods (no differential privacy at the moment), and more. Modules can be stacked together in pipelines. FATE is not framework-agnostic but has a backend for the Deep Learning libraries PyTorch and TensorFlow which is enough for many applications. Custom modules can be created but FATE is less flexible by design than its competitors.
One of FATE’s advantages is the functionality around it. FATE supports live visualization with FATEBoard and has its own model serving with FATEServing. FATE can also be run on top of Spark clusters, and KubeFATE even allows the deployment of FATE via Docker Compose and Kubernetes.
The usage of FATE is not always straightforward with its many different modules and options. FATE comes with extensive documentation, that can be hard to navigate and, in its English version, incomplete at times.
Overall, FATE is a business-ready FL framework, with great pre-build modules from which pipelines can be composed and a lot of extra functionality, including orchestration. However, it is not easy to build and use custom modules and usability could be improved.

Privacy Protecting Machine Learning, © author, generated by DALL.E

Flower — Flexible for research problems

Flower is a flexible, easy-to-use, and easily understood open-source FL framework. It is framework-agnostic, meaning that nearly every ML model can be easily migrated to the federated setting using Flower. It offers good documentation and helpful toy examples for different ML frameworks.
Flower originated from a research project at the University of Oxford, and is now distributed by adap gmbh, a German distributed AI company. Due to its background, Flower makes it relatively simple to customize, extend or override components, e.g. to apply new aggregation algorithms, which makes it well-suited for research and study projects. Other advantages are that Flower is interoperable with different operating systems, runs on a wide range of devices, and is scalable to millions of clients. In general, horizontal FL is more straightforward to implement than vertical FL. Recently, tree-based models have been added to Flower.
Flower has some limitations as well. It does not provide privacy, monitoring, deployment, or orchestration functionality at its core. Version 1.1 implements a wrapper for Differential Privacy that is currently at an experimental stage. At the moment, no other privacy methods or client authentication are supported. Some of the limitations are accepted to keep usability, e.g. tree-based models, whilst others are planned to be included at some point in the future, e.g. more privacy methods. Meanwhile, there are some third-party projects that extend Flower. For example, the ML company katulu combined KubeFlow and Flower to orchestrate the FL training.
All in all, Flower is an easy-to-use, extendable, and research-oriented FL framework, with fewer extra features and privacy methods than other frameworks.

PySyft — Remote Data Science powered by Differential Privacy

PySyft is an open-source FL framework by the OpenMined project, a distributed computing initiative funded by big tech and supporters. In a bigger context, PySyft is not only an FL framework but a remote data science platform. Experiments on protected data can be performed remotely whilst privacy is ensured through differential privacy and secure multi-party computation. When performing FL experiments PySyft is not framework-agnostic, but supports only the deep learning libraries PyTorch and TensorFlow, and will provide a TensorFlow Federated API starting from version 0.7.
PySyft must be used with PyGrid, a network that connects data owners, the FL clients, with data scientists, the FL server. PyGrid is a Linux application but can be run on any OS using containers, and is best used with HAGrid, “the swiss army knife of OpenMined’s PySyft and PyGrid”.
PySyft used to be well documented, including video tutorials, a couple of years ago. However, it appears that the tutorials may not always match the current version of the framework, which makes it challenging for newcomers to use PySyft in the best way possible. It seems that PySyft is currently undergoing a lot of development, which may be contributing to this issue. However, according to the active community on Github, these problems may be addressed in the next version (0.7).
In summary, PySyft goes beyond normal FL frameworks by creating a platform where not only FL but also federated data science is protected by Differential Privacy. PySyft is still very much under development. Currently, it does not have up-to-date documentation. Nevertheless, it has a lot of potential.

Conclusion

There are several FL frameworks available, each with its own strengths and limitations. It is important to consider factors such as usability and flexibility, as well as any extra features like privacy methods, deployment, and orchestration, that may be useful for your application. Flower, IBM FL, OpenFL, and NVFlare are frameworks that have different specializations but they are all extensible and framework-agnostic. FATE is more of a black box application that stands out from the group. It can be a powerful tool with unique orchestration features if one can handle the complex documentation. PySyft is not limited to FL (the training of ML models) but enables general data science on remote data. However, it requires further development to live up to its potential.
FL with tree-based models is supported by NVFlare and FATE. And with the next release of xgboost, federated xgboost is going to also become part of the standard package.
For our project at SUMEX, we decided to use NVFlare in our prototype. It combines features to create a deployable product with the flexibility needed for our experiments. It was a welcome bonus that the framework also supports tree-based models and it is backed by a strong industry player.

Sources:

• Introduction into remote data science, Openmined Course, https://courses.openmined.org/courses/introduction-to-remote-data-science , 1.10.2022
• Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directions, https://arxiv.org/abs/2208.03392 , 11.01.2023