Start Small, Then Build Up
We have this saying in Texas that goes like:
Everything is bigger in Texas.
And being born and raised in Houston and occasionally venturing out to other cities and states, I can definitely say that is true. We have bigger cars, people, houses, and plates of food. And as a society, we sometimes associate things that are bigger, with better.
We sometimes want both quantity and quality — and maybe even think they are intertwined.
Nowadays, I believe that this has also been the case for artificial intelligence networks.
Many of the out of the box artificial neural networks have been pre-built with millions and millions of parameters. Take the GPT-2 that has taken the AI community by storm in its amazing results in natural language processing — 774 million parameters!
That’s a ridiculous amount of neurons to tune.
Further, if you take an academic paper that is applying deep learning to a certain problem, you will be (most of the time) hard-pressed to find the number of parameters the authors used in the model.
As an AI community, there may be an inherent assumption that throwing more parameters at a well-designed network will make it better.
“Go deeper” as they say.
This tendency to scale up, with the hopes of achieving bigger and better things may be one of the strategies that are holding us back.
One good strategy to use when building, maintaining, and adopting interesting deep learning architectures for your problem is to try and start small. Shoot for hundreds of thousands of parameters instead of millions.
This could:
- Force your model to learn few, but important representations of your data since it cannot have as many redundant representations/filters
- Reduce overfitting
- Allow your model and associated data to fit in memory
- Improve training efficiency
- And maybe, just maybe, improve performance
Give it a shot. Start small and then scale up.
Thanks for reading.
