Machine Learning Intern Journal — Keras Tuner

As the title indicates, this is the journal of a Machine Learning (ML) intern at the impactIA Foundation. I’ll be attempting to keep a weekly journal of my activities in the Foundation to keep track of my progress and leave a roadmap for the interns who come after me.

We spent most of this week experimenting with our quality control system. It has been the most coding-heavy week so far, and it felt so good to be back in my comfort zone: debugging for hours on end.

As mentioned in the previous blog, I’ve been using Keras Tuner to do hyperparameter optimisation. I’m going to take you through the code I created for this search.

To use Keras Tuner, one must create a model building function, which will be used by the ‘tuner’ instance to iterate over different model configurations in search of the best performing one. Above is the model building function I created. As you can see, it takes one parametre, ‘hp’, which is the HyperParameter instance from Keras Tuner. On line 3, we can see a call to ‘augment_images()’ — ignore it for now, we’ll come back to it in a minute. On line 5 we create our base model (A ResNet, I’m expecting all of you to be experts in ResNets after last week’s explanation), providing as input the augmented images. The cool part starts on line 9, where we create a tunable for-loop for a custom number of layers, and each layer is also tunable for the number of neurons. Furthermore, the dropout after each custom layer is also tuned. Wow, a lot of tuning — but I think it’s quite straight forward to understand? Once we start the search, the tuner will be able to create different network configurations, tuning for the optimal number of layers, neurons per layer and dropout rate. On line 19 we define our model, à la Keras. On line 25 we define a tunable learning rate, and finally compile the model. As you can see, the HyperParameter can take various forms: int, float and choice are the ones I use, but boolean and fixed are also available.

Now let’s look at the augment_images() method:

I think this part is really cool. We’re using experimental pre-processing layers (RandomZoom, RandomContrast, RandomRotation) from Keras, and tuning the use of them. The booleans determine if the layer will be used or not, and then we also tune the factors for each.

Then you simply create the tuner (Note: trials is the number of different parameter configurations to be tested):

The search_space_summary() method is a useful way to see all the tunable parameters in one place before beginning the search. And finally, the search:

I think calling the results_summary() method is the best way to visualise your results (it displays the top 10 network configuration with the best score attained).

So that was Keras Tuner! I hope I didn’t rush it too much, I believe the code is quite self-explanatory, but for more information don’t hesitate to check out their documenation. As Valentin mentioned in his blog, he started documenting all the experiments he ran. Not only did this help him better structure his ideas and thoughts, it also made it easy for us to keep track of who was doing what and to provide rapid feedback to keep moving forwards. Therefore, I started doing the same and it adds a lot of structure and method to the madness, which is very necessary when you’re running 10+ experiments a day. He also discusses our Foundation’s progressive approach to the work-life balance, which I will discuss in next week’s blog!