9-months in the “hobby” of deep learning

Deep learning, AI, machine learning, and all of those others buzzwords are spouting out everywhere. No domain is safe from marketers trying to use these terms to sell a product, and no startup would be caught dead without them (or blockchain). So to enact my due diligence I wanted to jump on the deep learning bandwagon. The problem was my plate has been very full. These past 9-months had:

• Daddy duties
• Husband duties
• Work duties
• A month of Executive courses
• Preparing for DEFCON
• Fighting off a hurricane Harvey

So can a professional just take up deep learning as a hobby? Sure can!

I had tried going through Andrew Ng’s Coursera course, but I quickly got sidetracked. Fortunately, I discovered Fast.AI (Jeremy Howard and Rachel Thomas) and launched myself through the first two modules. I even got accepted into their follow-up part 1v2 as an international fellow. Despite all the time factors, there is something addictive about getting my hands dirty running and altering the scripts.

The Rig and the joy of GPUs

Some people love their cars or their guitars, but I am passionate about my computers. Lesson 1 form Jeremy is setting up an AWS instance. While the AWS instance worked, I quickly decided I need to take advantage of a GTX 1080 that sat idle most days.

There are several links (here, here, and here) describing the best way to make a deep learning rig. Fortunately, it is mainly just a gaming machine putting on adult clothing, and it only took a little bit of tweaking to get scripts running. The most significant change I needed was buying an SSD to hold my training data on and install a new version of Ubuntu.

Setting up SSH to allow me to log in remotely has also been vital. Every morning I can spend about an hour drinking my coffee and getting ready for the day to start. Having a remote connection to my rig allows me to quickly pick up exactly where I left off and not need to carry around the machine with me. Indeed, my deep learning computer does not even have a monitor because I merely login remotely, even at home.

How have I not used GitHub?

I have known about GitHub for quite some time, but I have not routinely used it. These last few months I have gone from one project with 9 lines of code to about nine notebooks. I feel like I am barely scratching the surface. I pull, push and clone but there is so much more I have not got to yet. It does allow me to quickly update and share my updates which I find valuable in case my rig went up in flames.

The Jupyter notebook

It was also shocking how much I have grown to love using the Jupyter notebooks. All the documentation, and saved outputs readily repeatable. Troubleshooting is amazingly more comfortable for me, and a large part of the data is just making sure I accurately see the formats for it. Jupyter gives that to me in an easy to understand way. I wish I used it back with Kali for pentesting documentation so that everything is both rapidly reproducible and documented.

The best features when dealing with larger training sets were the timing features for individual blocks. Having the ability to see how long an iteration takes and having a verbal warning when something completes is very valuable. If data is taking 30 minutes to load, finding an alternative loading mechanism makes much more sense until you need it.

Are you solving problems?

If you are expecting to solve world hunger, we might be a ways off. However, an excellent standby for testing what you have learned is with Kaggle competitions. The fast.ai course has plenty of real-world problems with real-world data. Seeing what other groups are solving has been helping me think about what I can apply to work immediately. Not with a billion dollar budget, but with what I have right now. Here are my three favorites.

Cats and Dogs — Kaggle Competition

Everyone needs to figure out how to better identify cats and dogs. This contest goes out of its way to keep the fight alive. Using several pre-computed models users can predict if an image is a cat or dog. In my 2000 images, results in only 13 incorrect answers. Here are some random examples of the correct pictures.

So that is pretty good, and these predictions all makes sense. However, when we look at the incorrect cats, we see the following.

We can see why a computer might get these wrong. These are bad pictures. My two year old wouldn’t get these either. The beauty of the project draws from its simplicity and ease of understanding. A great first project.

Statefarm- Kaggle Competition

I find this one much more interesting since it classifies human behavior. The images are cut up into multiple categories trying to show that humans are doing silly things while driving. While there are several defined activities, it is fun to catch people being a jerk in many different ways. Most of the distracted driving is simple.

Is the driver distracted and if so how? Are they texting, yelling at someone on their phone, drinking a soda, or something else? These classifications are extremely easy for a person to describe. However, it has been only recently that you can start thinking about how to get a machine to learn them.

As an aside, you can really mess with the dataset for this one. If you decide to have drivers in the training and validation set, you can add bias to the models. For whatever reason, my first interactions incorrectly labeled drivers with glasses as distracted. Every. Time. Upon review, I discovered my all my unsafe class predictions for a particular category had glasses! Imagine an employee sending a dataset to production that made the same mistake with hair length or skin color. It is terrifying.

On the plus side, I feel that this could have the most profound impact on behavior. Imagine people calling out when they are slouching in chairs. Alternatively, if a child is climbing on something that is too dangerous. Alerts, warning, corrections can help keep people safe and drive changes in behavior to make us better and safer.

Web Traffic Time Series Forecasting- Kaggle Competition

I go into this competition more in-depth here. However, since posting the contest has ended and I placed in the top 30%, this is pretty high up there considering it was the first competition I went into alone and unafraid.

The misunderstanding I had here (cat vs. cont variables) I was able to work on in my 2nd competition. So I am still thrilled about placement. Additionally, it started off my deep love for non-picture type problems. Plus there are two other similar examples right now which I will likely compete in.

My next steps

• I just finished Porto Seguro’s Safe Driver Prediction and should be getting another article out shortly regarding lessons learned.
• I am starting on Mercari Price Suggestion Challenge predicting products selling based on language.
• I am exploring an opportunity for testing a small dataset at work

In case you want to look at some unfinished code you can check out my work here on GitHub. However, I would recommend that instead you go and start taking a look Jeremy’s course. Seriously, take a day off work and try this. I know I learned mountains from it and think the way he approaches the teaching of others fantastic.

Its not THAT hard.