How to use half precision float16 when training on RTX cards with Tensorflow / Keras

Noel Kennedy
Jan 24, 2019 · 2 min read

If you are doing machine learning on NVidia’s new RTX cards, you will want to try out half precision floats (float16). The new NVidia RTX 2070 cards have less physical memory than the old GTX 1080ti cards but the RTX’s newer architecture supports float16. This effectively doubles the size of the models you can train using RTX cards because half precision floats take up half the memory space of the default float32 floats that are typically used in machine learning.

Image for post
Image for post

Now, as a sweet summer child, you will be thinking that it’s really simple to configure your setup so you can do FP16 training with your shiny new RTX cards using Tensorflow and Keras, right?

You may think that you just physically install your cards, find some drivers from somewhere and tell TF / Keras to use float16.

Image for post
Image for post

If you have come here, you probably know that this isn’t true…

The main problem

RTX cards require CUDA 10 and as of January 2019, Tensorflow only supports CUDA 9.

The solution

You need to install CUDA 10, then build your own version of Tensorflow against CUDA 10.

It turns even this isn’t enough, and you need 2 lines of code to get FP16 working after this:

Firstly the instruction to use float16. Secondly to adjust the ‘epsilon’ to a larger value because the default value is too small for FP16 calculations. If you don’t change the epsilon, you will often get NaN during training.

How to build Tensorflow from source

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store