If you are serious about machine learning, you’ve probably spent a lot of time adjusting the hyperparameters of your models and staring blankly at your computer’s screen for hours only to see your performance slightly improve. Bender attempts to greatly accelerate this painful process!
- It finds the best set of hyperparameters for a model.
- It chooses the best model over a given problem.
- It allows you to visualize and explore your search space.
Experiment, Algo and Trial: Concepts to formalize the workflow
In the Bender ecosystem, we use different words to describe some key elements of your solution.
An Experiment is actually the problem you are facing. For example, classifying handwritten digits or predicting stock values.
An Algo is a way to solve this problem, it’s your code. For example, you might want to try using Random Forest and then Neural networks. You will be able to compare the performances of different Algos .
A Suggestion is a set of hyperparameters to try on a specific algo that Bender gives to you.
A Trial is what you give back to Bender in answer to a Suggestion so it can improve. It’ s the measure of the performance of an Algo with the suggested parameters.
Here, for example, we can see two Algos getting hyperparameters from Bender, processing the corresponding performance and giving back the results to it. This way, Bender can explore the search space defined by these hyperparameters, looking for the most promising areas. It is a cycle of both Suggestions and Trials: loop over this until you feel that you’ve reached a stable optimum in your algo’s results.
How does it work?
Bender’s workflow is centered on its API, available in open access at https://bender.dreem.com And you have two types of clients interacting with it.
First you have the “in-code clients” that are used to generate trials. They take suggestions from bender and send back results to it. As you can see below, the suggestions are not coming from the API itself but are instead generated by the benderopt library that is using the past trials listed in the database to suggest new ones.
Pseudo-code example:
bender = initialize_bender()
while not_satisfied
suggestion = bender.get_suggestion()
results = run_my_calculations(suggestion) # slow step
bender.send_trial(suggestion, results)Of course this code is parallelizable and will just work fine with multiple clients doing the same operation at the same time.
Then you have the web client, the purpose of which is to visualize trials in a graphical way and show a ranking of the different Algos within an experiment.
The web client is currently available at https://bender.dreem.com, also in open access, and uses our online API.
How to use it?
Bender is available in open-access, online on our servers. First of all, you just need to create an account on bender.dreem.com. You will then have access to the online visualization dashboard. Then you can find the python client on pypi and the R client on the CRAN.
A packaged version exists if you want to deploy the Bender API or the web client yourself.
Finally, find the complete documentation on ReadTheDocs.
Why did we create Bender?
It started as an internal project at Dreem that we then decided to give to the community.
The first reason is about scaling and flexibility. The existing open solutions didn’t provide an efficient way to work asynchronously on optimisation problems. So we began to work on Bender as a better alternative for us.
With Bender, we just want to create a cool open source project on which people can share ideas and improve the hyperparameters optimization’ state of the art. Feedback, help and ideas are always welcome! You can contribute to the project on it’t Github!
