coMind, Collaborative Machine Learning
We are in the midst of a huge transformation of the means of production. In the transition to an information society, data is becoming a strategic asset. The most valuable organizations no longer transform raw materials, they transform data into meaning. The main difference with data as an asset is that it’s already distributed among people and its value is extracted with software.
Machine Learning is going to take over many tasks, releasing huge productivity gains, which will lead to the destruction of many jobs. Many new ones will be created, but a whole generation is being put at risk. The conflict will be obvious when people realize that ML models don’t learn, they are taught by the very same people who are being replaced.
For over a year we have been working on the answers to several fundamental questions:
- How can we distribute the value of data?
- How can we value individual contributions?
- How can we preserve the privacy of sensitive data?
The first two questions lay in the economics of data, and since value is fundamentally a matter of subjective preferences, there are no obvious answers. More on this later, in this post we are addressing the last one.
Privacy is key not just as a personal choice and a right, it avoids the depletion of data, protecting its value and ownership. After extensive research on several distributed machine learning algorithms like, federated averaging or federated multi-task learning. Today we are excited to release code and three tutorials for people to start experimenting with coMind: a set of tools to empower people to build machine learning models together.
Federated Averaging
Federated averaging has a set of features that makes it perfect to train models in a collaborative way while preserving the privacy of sensitive data.
TensorFlow has already implemented traditional federated learning (distributed learning from now on) and it may look like the best starting point. So why didn’t we set for distributed learning? What advantages does federated averaging offer over it? To understand this point we first need to understand how both systems work.
A cluster for both distributed learning and federated averaging has the following components. Some devices act as parameter servers, this means they won’t perform any operation, they just store the parameters the other devices need to do their job. The rest of devices act as workers and one of them is designated chief (aggregator).
The distributed learning works like this:
- Each workers trains a batch locally, and then calculates the gradient that is applied to its weights to minimize the cost function.
- Then it sends the gradient to the parameter server.
- All the workers or a subset of them that we define, send their gradients to the parameter server.
- Once all the workers have sent their gradients to the parameter server, the chief reads all the gradients, averages them and updates the model stored in the parameter server.
- The workers read it and update their weights.
On the other hand, in Federated Averaging:
- Each worker will be updating its weights locally, as if it were the only one training.
- Every set number of steps it will send its weights (not the gradients, but the weights themselves) to the parameter server.
- The chief (aggregator) will read the weights from the parameter server, average them and update the global model.
- The workers read it and update their weights.
In a setting like the one described in the introduction, communication efficiency is of the utmost importance and minimizing the number of rounds of communication is the principal goal. As we have explained, distributed learning needs communication between devices after each batch, while it has been demonstrated that federated averaging can process several batches of data between two synchronization events.
As we already mentioned, TensorFlow has great tools for distributed training. However, this is not the case for Federated Averaging, so we are releasing several tutorials and a custom TensorFlow optimizer.
Tutorials
Visit Github to find the Custom TensorFlow optimizer and find more advance examples.
Join us in our journey! We encourage everyone interested in Collaborative ML to start experimenting with our code. We would love to hear from you. Let us know if you have any questions or suggestion regarding our project. Don’t hesitate to ask for help and please, let us know if you build something cool!
If you are interested in what comes next… Join the conversation on Slack or Telegram.
— The coMind Team
Originally published at medium.com on September 7, 2018.
