I just passed the TensorFlow certification… here are some tips for you

After finishing my Computer Science studies, I went from systems administration… to disaster recovery… to virtualization… to public cloud. That journey took 15 years of my life, following the trends of the market.

Specializing in IoT was my choice. It brought together one of my passions (tinkering with electronics) and my job (public cloud). But any IoT project is going to end up being a big data / analytics project, and hopefully, a ML project if you want to do something with that data… something ‘smart’…

So I found myself out of my comfort zone… trying to remember algebra from my first year at uni… extending my limited Python scripting skills to find trends on the data that hundreds of thousands of sensors were uploading every day, to do predictive maintenance, or visual inspection, or… you name it… my customers surprise me every day with (very interesting!) new use cases of ML for IoT data.

That’s when I started learning TensorFlow. I would definitely recommend ‘TensorFlow in Practice’, by Laurence Moroney and Andrew Ng. The courses take you from a basic model to image and text classification to time series of data in just a few weeks. There are plenty of tutorials (or notebooks ready to use) for any type of data and problem you can imagine… But there’s a moment when you want to demonstrate the world (and yourself) that you can do it!

So, when I heard in the TF Developer Summit that they were going to launch the TensorFlow Developer Certificate, I decided I was going to take it. It took a few weeks until I had the time to sit the exam, but I passed! And I’d like to share with you how the exam was, a few tips, and hopefully help some of you who are thinking about doing it!

The first stop, of course, is the landing page www.tensorflow.org/certificate and more specifically, the candidate handbook. Take a good read of it. I have to admit, I most often just skim-read through these documents, but it contains a skills checklist that is going to help you assess if you are ready to take the exam, or if you should polish up on areas where you might be rusty. I went through that list and searched for Colabs on the areas I needed to review. Don’t fall into the ‘Shift+Enter’ temptation. Force yourself to open a blank Colab and type it again yourself. That’s what really works for me.

You will sit the exam at home, using your own computer. The candidate handbook explains how to set up the environment. It’s easy. You’ll need to install PyCharm (2019.3 at the time of writing this post), Python 3.7, and some packages like (obviously) tensorflow, numpy, etc… The whole process is detailed in the ‘setting up your environment’ document.

To test my environment, I trained some models… and I found myself with an issue… I LOVE my Pixelbook Go. It’s light, battery lasts forever, I even have Crostini to test scripts for my IoT devices… but it struggles to train models with a big dataset. I never had this problem before, because I use Colab to prototype and Jupiter Notebooks on GCP’s AI Platform for my customers… But since PyCharm is a requirement, I spun up a linux GCE machine with 4 vCPUs and 1 GPU (the price is roughly $0.55 per hour) and installed remote desktop to connect to it. New tests, and my model training flies… There’s no GPU requirement for the exam, but for me, waiting for a model to train when you have a time limit is nerve-wrecking… And a VM with a GPU is much cheaper than buying one! (just remember to shut down the machine when you are not using it).

ID verification and fees payment ($100) is straight forward. You don’t have to start the exam straight away. You will be taken to a page with the final instructions before launching the exam. It will ask you 2 or 3 times to read a very important document that explains how to install the Google Developers Certificate plugin in PyCharm… and has a troubleshooting section. Read all carefully. I run into issues at the start of the exam on PyCharm that were covered on that section. Keep that document open to refer to it if you need to during the exam.

And when you are ready… on PyCharm, you’ll open the exam plugin and press ‘start exam’. It will create a Python virtual environment and install some packages. And from that moment, you have 5 hours to submit the exam.

This was my very first coding certification. As I mentioned, my background is in systems administration, so the certifications I own, like GCP Cloud Architect or GCP Data Engineer are based on multiple choice questions. So I have to admit, I was quite nervous.

You will be presented with five questions. Five models that you will have to code, increasing in difficulty. You don’t start from scratch. There’s the question (as comments) on top of the Python file, and prompts where needed (for example, to load the data, or to build the model itself). It is clearly marked where you have to add some magic by ‘#Add your code here’ comments.

I started with the first (very easy!) model, and I immediately hit a problem. PyCharm was not doing auto completion. I look at the top of the file, and it’s complaining that the packages were not present. First cold sweat. But I remembered that is something that I read about before on the troubleshooting guide, followed the steps, and in a few seconds I was up and running again.

Once you coded your model, you have to run the code. Doing so will train the model and run a function (provided in the script skeleton) that will export the TF model to a file.

On the right hand-side of the screen, the exam plugin has a button to ‘submit and test’. Clicking it will upload your model and run some tests. You will quickly receive a grade from 0/5 to 5/5. I don’t know exactly what that means. I could not find anywhere in the manuals and documents anything about it. My guess is that the system has a test dataset that is run against your model to get predictions, and if all the predictions are good, then you get the 5/5. You can submit a model as many times as you need, with no penalization.

As I said, the first question was very easy, but even with that, I did not get the 5/5 the first time. Quick check and I realized that my model was not very accurate. I was impatient. The solution? Add more epochs to the training and let it converge more. The result, a 5/5 score! On to the next question.

Question 2 was not too complicated, based on a well-known dataset. I got 5/5 on the first try.

Question 3 was a bit more tricky for me. The dataset was BIG. I coded my model, started to train it, and I quickly realized that it was going to take more than 30 minutes to finish. Five hours of exam sounded like a lot when I started, now I was not that sure. And I panicked a bit. I didn’t know how my model was performing at all… So I modified my code to split the dataset and have some validation data. I started to train again, and yes, it looked like epoch after epoch it was converging, so decided to let it finish. What can I say? It’s not fun to sit watching the batches and epochs slowly count… So I started to take a look at the other two remaining questions. I coded the models on both of them while Q3 finished training.

I submit Q3 and… have 3/5. What was going on? My validation accuracy was around 0.91, which I thought was good… But… was I over-fitting? Any changes in the model would need re-training. Waiting another 30 minutes? What if whatever I did to the model wouldn’t improve it and I had wasted another precious half hour?

That’s the time to sit back, look at the bigger picture, and remind myself that TensorFlow has tools to help models train faster and great documentation. Keep in mind that the candidate handbook says, “You may use whatever learning resources you would normally use during your ML development work”, so this is not cheating!

And that’s my next tip for you: don’t get obsessed on writing a model from head to toe… The exam does not limit what TF tools you can use… take advantage of as many of them as you need. The next few rounds were much faster training. However, I still wasn’t getting the accuracy I was aiming for…

What to do? Obviously, hyper-parameter tuning. Looking at the questions of the exam you might think that they are spoon-feeding you the data and force you to build a model just like those on the books (or the notebooks)… Try to apply your experience using TF with your customers, or Kaggle… follow your gut and play around with the hyper-parameters to improve your model. But I was not there yet… I could not get my validation dataset to go above a 0.91

That’s when I remembered about the improvements on the latest versions of tensorflow and tensorflow-estimator. I did a quick check on PyCharm, and I was surprised because the versions that the exam environment had were not the latest. Was that what was blocking my progress? I am 100% sure that the TF team has tested the exam and you can achieve 5/5 with those package versions… but they are much more smarter than me, for sure, and I could not find any other improvements in my model… So I took a chance, and upgraded both, tensorflow and tensorflow-estimator to the latest version. PyCharm lets you choose which version you want installed, so I thought I could always downgrade them if that broke the exam… But it did not! And I have to say that my accuracy went up an incredible 5% only by doing the upgrade.

That should be the one! I was at a 0.97 in my validation dataset… So I was expecting my 5/5… but no! I got 4/5. I was puzzled. So just out of ideas, without training the model, I re-sent the last one… And there it was!!! My 5/5!! I hold my breath for a second… What happened? Why the same model would give me a 4/5 and then 5/5? My guess is that when the platform tests your model, it does so with random data, not always the same. So trying a couple of times to send the model is my last tip for you.

I submitted the exam (remember, you must have run and ‘submit and test’ each question at least once)… and immediately got an email with the news. There is no score on it (the candidate handbook says you need a 90% to pass, and not all the questions have the same weight).

I hope this helps to anyone that is thinking about taking the certification exam… If you have any questions or comments, don’t hesitate to leave them here or on my Twitter: @TheCloudBarber

Good luck!