TensorFlow — low and high level API
In this article, I want to provide a tutorial on implementing of a simple neural network using lower and higher levels API. My short tutorial is intended for people, who know Python or other programming language and have basic understanding of deep learning.
Low-level API
I prepared auxiliary functions to create weights for the model. I used truncated normal initializer with mean 0 and standard deviation 0.1 for weight and constant zero for bias to initialize values.
def weight_variable(shape):
shape = tf.TensorShape(shape)
initial_values = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial_values)def bias_variable(shape):
initial_values = tf.zeros(tf.TensorShape(shape))
return tf.Variable(initial_values)
I called this function to define variables and I also created placeholders for input and output data. Input has 4 dimensions, hidden layer has 2 units and output has 3 classes.
# Define placeholders
X = tf.placeholder(tf.float32, shape=[None, 4])
y = tf.placeholder(tf.int32, shape=[None, 3])# Define variables
w1 = weight_variable([X_input.shape[1], 2])
b1 = bias_variable([2])w2 = weight_variable([2, 3])
b2 = bias_variable([3])
I built network using straightforward code. Network has one hidden layer. Relu activation function for hidden layer and softmax for output layer.
# Define network
# Hidden layer
z1 = tf.add(tf.matmul(X, w1), b1)
a1 = tf.nn.relu(z1)# Output layer
z2 = tf.add(tf.matmul(a1, w2), b2)
y_pred = tf.nn.softmax(z2)y_one_hot = tf.one_hot(y, 3)# Define loss function
loss = tf.losses.softmax_cross_entropy(y, y_pred, reduction=tf.losses.Reduction.MEAN)# Define optimizer
optimizer = tf.train.AdamOptimizer(0.01).minimize(loss)# Metric
accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(y, axis=1), tf.argmax(y_pred, axis=1)), tf.float32))
You need to use explicitly session to run a graph for training model.
for _ in range(n_epochs):
sess.run(optimizer, feed_dict={X: X_train, y: y_train})If you want to know more about TensorFlow’s low-level, you can read this tutorial: https://www.tensorflow.org/get_started/mnist/mechanics.
High-level API (using Estimators)
I added all details in comments. This is the same network as above.
def model_fn(features, labels, mode):
X = features['x']
# Define weights initializer
kernel_initializer = tf.initializers.truncated_normal(stddev=0.1)
# Define network
hidden = tf.layers.dense(
X, 2, activation=tf.nn.relu,
kernel_initializer=kernel_initializer)
logits = tf.layers.dense(
hidden, 3,
kernel_initializer=kernel_initializer) # Define dictionary with predicted class
# and probabilities for each class
predictions = {
'class': tf.argmax(logits, axis=1),
'probabilities': tf.nn.softmax(logits)
} # If we are on prediction mode,
# we return estimator specification
# with predictions
if mode == tf.estimator.ModeKeys.PREDICT:
return tf.estimator.EstimatorSpec(
mode, predictions=predictions)
onehot_labels = tf.one_hot(labels, depth=3)
loss = tf.losses.softmax_cross_entropy(onehot_labels, logits) # If we are on training mode,
# we need to do few things.
if mode == tf.estimator.ModeKeys.TRAIN:
# Create optimizer
optimizer = tf.train.AdamOptimizer(learning_rate=0.01)
# Define training operation (minimize loss function)
train_op = optimizer.minimize(
loss, global_step=tf.train.get_global_step()) # Return estimator specification
# with loss and training operations
return tf.estimator.EstimatorSpec(
mode, loss=loss, train_op=train_op)
# If we are on evaluation mode,
# we need to define metrics operations
eval_metric_ops = {
'accuracy': tf.metrics.accuracy(
labels, predictions['class'])
} # Return estimator specification
# with loss and metrics operations
return tf.estimator.EstimatorSpec(
mode, loss=loss, eval_metric_ops=eval_metric_ops)
I defined classifier object using model_fn.
classifier = tf.estimator.Estimator(model_fn)Next step is creating input function. I used numpy_input_fn function.
input_fn = tf.estimator.inputs.numpy_input_fn(
{'x': X_train}
batch_size=200,
num_epochs=100,
shuffle=True)I fitted data to estimator using input_fn. For evaluation, you need two creating another input function with appropriate data set.
classifier.train(input_fn)Bonus (using predefined models)
TensorFlow offers a variety of predefined models. For this example, you can use Deep Neural Network Classifier.
feature_columns = [
tf.feature_column.numeric_column("x", shape=[4])]classifier = tf.estimator.DNNClassifier(
feature_columns=feature_columns,
hidden_units=[2],
n_classes=3,
model_dir="/tmp/iris_model")classifier.train(input_fn=input_fn)
If you want to know more about TensorFlow’s high-level, you can read this great tutorial: https://www.tensorflow.org/get_started/estimator.
Feel free to comment or give some feedback.