Command line flags: Python and Tensorflow

I have recently came by many threads of confusion about flags in Tensorflow.

In order to quickly explain flags in Tensorflow, we have to first understand flags in Python. Here is a short tutorial.

For many Linux beginners, command line flags simply make no sense. They’re the weird things after ‘-’, such as

‘myapp — config’

However, more experienced developers will know that this can be very useful because flags allow you to change the initial conditions of your code straight from command line.

Model.py

from absl import flags
from absl import logging
from absl import app
FLAGS = flags.FLAGS

flags.DEFINE_string(‘model’, None, ‘model to run’) # name ,default, help

def main(argv):
   print(‘hello, world’)
   # logging.info(‘selected model’,FLAGS.model)
   print(‘selected model’, FLAGS.model)

If __name__ == ‘__main__’:
   app.run(main)

I created an example program show how to use command line flags. My favourite library for flags is absl, but it can also very similarly be done with

import argparse

( documentation here: https://docs.python.org/3/library/argparse.html)

Now, in the command line, we can run the standard

Python model.py

But this results in an error, expecting the flags to be defined. If we do not know what flags are defined, try typing

python model.py -helpshort
python model.py -helpful
flags.DEFINE_string(‘model’, None, ‘model to run’) # name ,default, help

Name of the flag is how the flag is called, in this case -model, the second argument is the default value, and the third the description of the flag when help is called.

In general, flags are a super easy way to use command line to define some switches in the code.

Now onto machine learning.

Machine learning models are ugly (or maybe I am yet to see some nice models? Who knows…) and usually require a lot of parameters to be set at the beginning of the model. At the same time, it might be beneficial for these parameters to be easily adjustable. This is for example the case of number of neurons in neural nets, learning rate or even optimizer, because a lot of these will be subject to optimization.

Luckily, there is a version of flags specifically for Tensorflow.

Source: https://github.com/egroup-develop/TensorFlow

flags = tf.app.flags
FLAGS = flags.FLAGS
flags.DEFINE_float(‘learning_rate’, 0.01, ‘Initial learning rate.’)
flags.DEFINE_integer(‘max_steps’, 2000, ‘Number of steps to run trainer.’)
flags.DEFINE_integer(‘hidden1’, 128, ‘Number of units in hidden layer 1.’)
flags.DEFINE_integer(‘hidden2’, 32, ‘Number of units in hidden layer 2.’)
flags.DEFINE_integer(‘batch_size’, 100, ‘Batch size. ‘
‘Must divide evenly into the dataset sizes.’)
flags.DEFINE_string(‘train_dir’, ‘data’, ‘Directory to put the training data.’)
flags.DEFINE_boolean(‘fake_data’, False, ‘If true, uses fake data ‘
‘for unit testing.’)
This has a very similar effect to the python flags.

Run

Python Tutorial.py -h

Given that there is no documentation on the Tensorflow website regarding flags, I would assume that this is Google’s internal coding requirement.

However, flags are really useful and should become every developper’s tool to manage global vars in a model.

So the final question is what is the difference between using python flags and tensorflow flags. At first, it seems that they both do exactly the same thing, and since tensorflow code interacts with python code, there is no reason to use the Tensorflow flags.

However, they are small differences in functionality that might be important in deciding which flags to use. (I’m partial to Tensorflow flags, so here’s a list of reasons to choose them!)

1. Tensorflow flags can occasionally be tensorflow-specific. For example, they enable us to set which GPU our code runs on. This is for example useful when you have multiple GPUs but you need your code to run on only some of them. Check the code below and see how it specifies the GPUs. Pretty cool.

if FLAGS.gpus is None:
if ‘TV_USE_GPUS’ in os.environ:
if os.environ[‘TV_USE_GPUS’] == ‘force’:
logging.error(‘Please specify a GPU.’)
logging.error(‘Usage tv-train — gpus <ids>’)
exit(1)
else:
gpus = os.environ[‘TV_USE_GPUS’]
logging.info(“GPUs are set to: %s”, gpus)
os.environ[‘CUDA_VISIBLE_DEVICES’] = gpus

Source: https://github.com/MarvinTeichmann

2. Tensorflow flags can be set anywhere in the code and are visible from everywhere.

3. You can have the flags predefined in a json and just import them into your model.

4. The most obvious reason: You are writing Tensorflow code, so write proper Tensorflow!

Happy coding!