Power Cut After 5 Days Running Deep Learning code
Hi folks, in this post I want share with you some issues I faced and truly believe many other ML/DL practitioners face when working out with large datasets or running deep neural network architectures on poor machines. In this post I will mainly focus on deep learning but the solutions I propose here may also be applied in the cases when you are not using deep learning but simply applying machine learning on large data.
First Things First!
What is Deep Learning?
Deep learning is a class of machine learning algorithms inspired by the structure and function of the brain called artificial neural networks.
- Neural networks, a beautiful biologically-inspired programming paradigm which enables a computer to learn from observational data
Neural networks and deep learning currently provide the best solutions to many problems in image recognition, speech recognition, and natural language processing.
What I was trying to do?
I joined the Kaggle competition hosted by The Nature Conservancy, the problem setting in this competition was to predict 8 classes/categories/species of fishes.
- Competition: The Nature Conservancy Fisheries Monitoring
- Problem setting: Can you detect and classify species of fish?
So there were given a certain amount of images labeled as belonging to these classes:
And one reasonable thing to do is to try out the widely used algorithms for Computer vision known as Convolutional Neural Network (one of the State of The Art algorithms available in the industry).
Convolutional Neural Networks (ConvNets or CNNs) are a category of Neural Networks that have proven very effective in areas such as image recognition and classification. ConvNets have been successful in identifying faces, objects and traffic signs apart from powering vision in robots and self driving cars.
Bellow is an example of CNNs in action!
Playing in the competition
I quickly made a simple script with a simple neural network architecture and I was able to run the code in my laptop with in approximately 15min. But wait, as i said, That was a really simple neural network architecture, I only succeeded to run that because I reduced the images to 32 by 32 pixels (width and height), but this approach limits a lot what the computer can see!
The Nightmare Begins
I tried to improve my deep learning model and thought Kaggle Kernels(think of this as an R or python script) and forums I got new ideas, oh that should be great, but the more I dive in more into the competition my model was becoming too complex and took a lot of time to run, I really wanted to learn from this competition and see what results would I get from my current models, so I made a stupid decision:
“It doesn’t matter How much time it takes, I’m just going to leave my python script running until it finishes, until them I won’t shut down my laptop and will limit the amount of processes I do while the script is running.”
yeah, That was me being fool!
Something Terrible Happens, a power cut after my code being running for 5 days (from my predictions the 5th day would be the last)
I left school excited about the idea that my script would have finished running (being fool again). When I arrived home I was told about the power cut and I couldn’t believe that(actually this was the first power cut of the year), There were all lost, 5 days for nothing, but fortunately I learnt a lot from this and I found some useful solutions to run my script. My naive approach to leave my script all those days reminded me this quote:
“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richad Feynman
Deep Learning requires better hardware
here is the list of how you can get that and I’m not going to make in depth description about these solutions or even compare them as that would be a subject for a whole other post.
Solution 0: Buy a new Hardware
This may be the obvious solution, this one wasn’t applied because i couldn’t afford to buy new laptop or PC.
Solution 1: Google Cloud Platform
Actually Google Cloud was the first to go solution as I got $300 when signed up, With this I was able to create some Virtual Machines(VM), install all dependencies and prepare them to run my code.
Solution 2: Amazon Web Services (AWS)
AWS is an incredible amount of great cloud services, defenitily worth trying but one thing I don’t like in AWS Free Tier compared to other cloud based solutions like Google Cloud and Azure is that they give you a set of resources while to me is more suitable to have free credits and chose the services you want to try.
Solution 3: Microsoft Azure
If you create your free Azure account you’ll get $200 free credit to explore Azure services, one thing I like about this is that they have VM’s that comes with GPU’s while on Google Cloud or AWS you’ll have to ask them to add GPU.
Solution 4: Floyd hub
As they say “Eliminate engineering bottlenecks in Deep Learning” this couldn’t be more true, I love these guys, Floydhub offers 100 hours of free GPU (Deep Learning is better on GPU).
Conclusion
Deep Learning requires better hardware and generally artificial neural networks run better on GPU rather CPU, so if you are planing to play with deep learning you should think whether you have the hardware required for your tasks. If you paid attention most of the solutions there are cloud based, Cloud is being a huge player in the Artificial Intelligence Industry and maybe it fits to your needs.
Useful Links
- https://www.kaggle.com/ — Platform to learn and compete on all sort of Data Science Problems.
- https://cloud.google.com/
- https://aws.amazon.com/pt/
- https://www.floydhub.com/
- The Deep Learning Revolution (video)
- Why Is Deep Learning Hot Right Now? (video)
- https://developer.nvidia.com/deep-learning
Call To Action!
If you’re interested or curious about Machine Learning, Deep Learning, Data Science or even about the Cloud let me know your opinion in the comments.
If you enjoyed the writings then please use the ❤ heart below to recommend this article so that others can see it.
Happy Learning.
