Ray Tune & Weights and Biases: Simple Developer Tools for Scaling Machine Learning
By Richard Liaw, Kai Fricke
Ray Tune is one of the first scalable hyperparameter optimization tools and has grown rapidly in a space that historically required users to build their own toolkits. After just a few years, we are now seeing Tune training algorithmic trading strategies at JP Morgan and being part of some of the most successful ML platforms, including Microsoft’s Azure RL and Amazon’s SageMaker.
Users love Ray Tune because it allows them to easily optimize parameters with state-of-the-art algorithms. They get access to tools like Bayesian Optimization and Population-based Training without having to implement the core functionality themselves. Additionally, Ray Tune gives them the ability to fault-tolerantly scale from their laptop to clusters consisting of 100s of nodes and GPUs just by changing a couple of lines of code.
Lastly, Ray Tune is part of the larger Ray ecosystem giving users access to reinforcement learning, model serving, and scalable distributed systems capabilities. These features and ecosystem allow machine learning developers to focus on building their ML applications as opposed to managing infrastructure.
While Ray Tune (and Ray in general) provide excellent building blocks and baseline implementations to accelerate research, Ray Tune has always had a gap: experiment management and model versioning.
The Gap: Experiment Management & Model Versioning
Since the inception of Ray Tune, we’ve packaged Ray Tune to come with TensorBoard support right out of the box. While TensorBoard provides the basics, we’ve noticed a lot of gaps and pain points over the years.
Users would tell us that a lot of their time was spent copying and organizing TensorBoard files from different machines. Further, as the parameter sweeps grew in scale, TensorBoard would break down.
As we spoke to the community, the request to integrate with amazing experiment management tools kept coming up. We began to see a large opportunity to help our users: What if Ray Tune joined forces with best-of-breed machine learning developer tools to make running and tracking experiments effortless?
Ray Tune ❤️ Weights & Biases
After surveying the community, Weights and Biases emerged as a perfect partner to help save all the metadata generated by Ray Tune during training. After working with the team, we’ve integrated Weights & Biases into Ray Tune to make it trivial for users to automatically save their experiments and track their performance.
Why this integration is ✨magical✨
This integration is magical for several reasons. Firstly, it combines two excellent tools for scaling machine learning experimentation and model development.
Ray Tune does this by simplifying several aspects of scaling and running experiments:
- Simple Distributed execution: Ray Tune makes it easy to scale from a single node, to multiple GPUs, and further multiple nodes
- Large number of algorithms: Ray Tune has a huge number of algorithms including Population Based Training, ASHA, and HyperBand
- Framework agnostic: Ray Tune works across frameworks including PyTorch, Keras, Tensorflow, XGBoost, and PyTorchLightning.
- Fault-tolerance: Ray Tune is built on top of Ray, providing fault tolerance out of the box.
Weights and Biases greatly simplifies the tracking and production of experiments. Weights and Biases does this by:
- Fast integration: Weights and Biases is fast to integrate into your current python scripts to start recording metrics immediately.
- Centralized reporting & collaboration: Weights and Biases provides a simple, centralized dashboard for teams and individuals to track results and share progress using reports from any number of projects.
- Artifact logging: Save artifacts like datasets and models on W&B to track pipelines, tune models, or version data over time.
- Model reproducibility: Enables you to save everything you need to reproduce models in the future, either on W&B or in your own storage.
How do I use it?
Weights and Biases is a first class integration in Ray starting in version 0.8.7.
It is currently available on the nightly version of Ray — follow the instructions here to pip install the nightly version of Ray and take a look at the documentation to get started.
We also published a W&B report that shows you an example of using the application.
Simple Example of running W&B and Ray Tune together.
See the documentation for runnable examples.
Looking forward with Ray and Weights & Biases
The Ray and Weights and Biases team are hard at work collaborating to make developing machine learning applications simple and we’ve got a number of things coming up to help the community learn more!
Ray Tune Developers on W&B AMA: Join the developers of Ray Tune for an AMA on scaling hyperparameter tuning.
When: Friday, August 14 at 9am PT
Where: W&B Slack Community in the #ama-ml-questions channel
We invite you to start posting all your distributed hyperparameter optimization questions in #ama-ml-questions now. The Ray/Tune team will answer them from 9am — 10am on Friday.
Ray Summit: Ray Summit is a FREE Virtual Summit for all things Ray related! Join us to see talks by leading computer scientists, the founders of Anyscale and Weights & Biases.
Beyond these two upcoming events, we’re excited to bring you the best of distributed computing infrastructure and developer tools for machine learning to make it simple to go from model development to production in the fewest steps possible. We can’t wait to see what you’ll build.
