What separates us from AI, Part 1: The Periodic Table of Machine Learning
Dev Nag

We can clearly understand what the building blocks of F2 are going to be. In Deep Learning we know this to be layers, objective functions and optimizers. However, in F3 the space of components is a big unknown. What you have as examples for F3 are just mechanisms for juggling the entire model. So basically what you do in F3 in hyper-parameter optimization is that you juggle around F2 components to find out the best performing one. However, what is obviously missing is the building blocks that make up layers, objective functions and optimizers. What I mean is, what can we juggle around to create new kinds of layers, new kinds of objective functions and new kinds of optimizers. The later is known as ‘learning to learn’. But what about the other kinds?