HPJS: Hyperparameter Optimization for Javascript

Google’s TensorflowJS is exciting because of multiple advantages it has over the traditional Tensorflow Python (being able to run models in the browser, for example). But due to it being so new, there are far fewer accompanying libraries for Javascript than there are for Python. Because of this, my dad and I are releasing HyperparametersJS (hpjs), a Javascript library for hyperparameter optimization. (We also made a site for the project here, feel free to check it out)

This article will cover the advantages of using Javascript for machine learning, discuss what hyperparameters are and how they can be “optimized,” and go through how you can do this with hpjs.

Why Use Javascript For Machine Learning

There are multiple advantages to using Javascript for machine learning, such as:

  • Being able to train AND run machine learning models in the browser (or even on a node server)
  • For developers, there is no need to install libraries or drivers
  • For creators, no need to pay for expensive gpu servers to run models. Instead, models can run client side
  • Automatically using user’s gpus if available
  • The layers API of tfjs is mostly compatible with keras

In a way, TensorflowJS can be viewed as a step towards “democratizing” machine learning. With client-side machine learning, you can make products that would otherwise incur high server fees, since the users will now be using their own processing power as opposed to yours. This is awesome because the barrier to entry for many machine-learning ideas has shrunk.

As an example: if you were making a site where people customize their own machine learning models and then train/run them, you wouldn’t have to run their models on your servers (with fees proportional to the number of users you have). Instead, the models will train and run on the users’ browser without them having to do any setup.

What is Hyperparameter Optimization

In machine learning, there are certain parameters that need to be set before the learning part begins (Larger companies often hire data scientists who use their years of experience to set values). These parameters are called hyperparameters. Examples of hyperparameters include learning rate, which is how sharply the model changes its weights during training, and number of layers.

Tweaking the values of these hyperparameters even a bit can change the performance of your model immensely, which brings the question: how do you give your hyperparameters their best values?

In Python, there are multiple hyperparameter optimization libraries (many of which you can find here), but as my dad and I were learning TensorflowJS, we didn’t find any for Javascript. For that reason, we made HyperparametersJS.

So How You Go About Optimizing Hyperparameters?

To determine what values for the hyperparameters are best, you could obviously give different values to each hyperparameter one by one and test the performance. However, it can be incredibly time consuming and involves menial labor, which nobody likes. Thankfully, there are 3 main approaches to hyperparameter optimization that don’t involve menial labor.

  1. Grid Search

Put simply, grid search searches through every possible combination of hyperparameter values. Clearly, an advantage of grid search is its exhaustiveness, while an obvious disadvantage is that it’s time consuming.

2. Random Search

In random search hyperparameter values are, well, selected randomly. Compared to grid search, it is less intensive on time and computing power.

3. Bayesian Optimization

Compared to grid and random search, Bayesian optimization is a bit more involved, but basically involves creating a function to approximate the whole process. (For a more in depth look at Bayesian optimization, this article is pretty good.)

We’ve already completed random search and are currently working on grid search, while Bayesian optimization will follow.

Search Spaces and Parameter Expressions

For these three approaches, defining a “search space” is necessary. In the search space, we define the bounds for each of the hyperparameters. Let’s go through an example of defining a search space with hpjs.

Let’s say our hyperparameters are optimizer and learning rate. For the optimizer, we’re considering between adam, adamax, and rmsprop, and for learning rate, we’re trying to find an optimal learning rate between 0.0001 and 0.1. Using hpjs parameter expressions, we can define our search space like so:

const space = {
learningRate: hpjs.uniform(0.0001, 0.1),
optimizer: hpjs.choice(['adam', 'adamax', 'rmsprop']),
};

hpjs.uniform and hpjs.choice are the parameter expressions that define the boundaries for our hyperparameters. hpjs.choice randomly returns one of the options in the list each time it is called, while hpjs.uniform randomly returns a value from a uniform distribution between the low and high bounds.

In all, there are 10 parameter expressions. Each have their own uses and if you’re already using hyperopt, they should be very familiar. I’ll list them out below, but it’s honestly much easier to get an idea of how they work at our site for this project: https://hyperjs.herokuapp.com/explore. Here, you can see visuals of how these expressions work and also mess around with their values and see their outputs.

hpjs.choice(options)

  • Randomly returns one of the options

hpjs.randint(upper)

  • Return a random integer in the range [0, upper)

hpjs.uniform(low, high)

  • Returns a single value uniformly between low and high i.e. any value between low and high has an equal probability of being selected

hpjs.quniform(low, high, q)

  • Returns a version of of hpjs.uniform with step size “q” 
    Mathematically represented as (uniform(low, high) / q) * q

hpjs.loguniform(low, high)

  • Returns a logarithmic version of hpjs.uniform i.e. exp(uniform(low,high))

hpjs.qloguniform(low, high, q)

  • Returns a version of hpjs.loguniform with step size “q.” 
    Mathematically represented as (loguniform(low, high) / q) * q

hpjs.normal(mu, sigma)

  • Returns a real number that’s normally-distributed with mean mu and standard deviation sigma

hpjs.qnormal(mu, sigma, q)

  • Returns a version of hpjs.normal with step size “q” 
    Mathematically represented as (normal(mu, sigma) / q) * q

hpjs.lognormal(mu, sigma)

  • Returns a logarithmic version of hpjs.normal i.e. exp(normal(mu, sigma))

hpjs.qlognormal(mu, sigma, q)

  • Returns a version of hpjs.lognormal with “step size” q. 
    Mathematically represented as (exp(normal(mu, sigma)) / q) * q