Understanding TensorFlow: Part 2
Here is today’s outline:
- Tensor creation
- Create Tensors by Python basic types and Numpy array
- Create Tensors by tf.random
- Create Tensors by special value function of TensorFlow
2. Tensor merge and split
3. Tensor comparison and sort
4. Mathematical operations
1. Tensor creation
The data type in TensorFlow is Tensor. A scalar is a tensor, a vector is a tensor, a matrix is a tensor, and the matrix of a matrix is a tensor, too. Tensor is a compound type. The elements in Tensor are python basic data types that we are quite familiar with, including bool, int, float, double, string, array.
TensorFlow 2.x is also quite friendly to Numpy. It perfectly supports Data type switch between Tensor and Numpy.
Let’s use examples to create tensors and make it more clear.
1.1 Create Tensors by Python basic types and Numpy array
import tensorflow as tf
import numpy as np
# int
print(tf.constant(5))
# float
print(tf.constant(5.0))
# double
print(tf.constant(5.0, dtype=tf.float64))
# bool
print(tf.constant(True))
# string
print(tf.constant(‘hello, world’))
# python list
print(tf.constant([1., 2., 3.]))
# numpy list
print(tf.constant(np.array([1., 2., 3.])))
# Returns (out) =>
tf.Tensor(5, shape=(), dtype=int32)
tf.Tensor(5.0, shape=(), dtype=float32)
tf.Tensor(5.0, shape=(), dtype=float64)
tf.Tensor(True, shape=(), dtype=bool)
tf.Tensor(b’tensorflow’, shape=(), dtype=string)
tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32)
tf.Tensor([1. 2. 3.], shape=(3,), dtype=float64)
Note that the dtype of Tensors created by python list and numpy list are different: float64 and float32. We should keep in mind that the default basic data type that switches from Numpy to Tensor is double. GPU processes float32 is much faster than processing double(float64). If you plan to run your code on GPU, you’d better explicitly point out the dtype should be float32 here by this:
print(tf.constant(np.array([1., 2., 3.]), dtype=tf.float32))
# Returns (out) =>
tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32)
tf.convert_to_tensor can also convert other data types to Tensor. One example is given below, and you can try other data types by yourself. Definitely you would get the same result in both ways. tf.covert_to tensor is often used after we vectorize the x_data to turn its data type into Tensor.
print(tf.constant([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]))
print(tf.convert_to_tensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]))
# Returns (out) =>
tf.Tensor(
[[0.1 0.2 0.3]
[0.4 0.5 0.6]], shape=(2, 3), dtype=float32)
tf.Tensor(
[[0.1 0.2 0.3]
[0.4 0.5 0.6]], shape=(2, 3), dtype=float32)
1.2 Create Tensors by tf.random
tf.random.normal: Outputs random values from a normal distribution. The default mean is 0.0. the default standard deviation is 1.0. The shape parameter need to be explicitly assigned.
import tensorflow as tf
# tf.random.normal.
print(tf.random.normal([2, 2], mean=0, stddev=1))
# Returns (out) =>
tf.Tensor(
[[ 0.07503338 0.690836 ]
[-0.54832345 -0.8049861 ]], shape=(2, 2), dtype=float32)
tf.random.truncated_normal: The generated values follow a normal distribution with specified mean and standard deviation, except that values whose magnitude is more than 2 standard deviations from the mean are dropped and re-picked.
# tf.random.truncated_normal. Outputs random values from a truncated normal
# distribution.
print(tf.random.truncated_normal([2,2], mean=0, stddev=1, dtype=tf.double))
# Returns (out) =>
tf.Tensor(
[[ 0.77151253 -0.10534472]
[ 0.33723833 -0.29132416]], shape=(2, 2), dtype=float64)
tf.random.uniform: The generated values follow a uniform distribution in the range [minval, maxval). The lower bound minval is included in the range, while the upper bound maxval is excluded.For floats, the default range is [0, 1). For ints, at least maxval must be specified explicitly.
# tf.random.uniform. Outputs random values from a uniform distribution.
print(tf.random.uniform([2, 2], minval=0, maxval=1))
# Returns (out) =>
tf.Tensor(
[[0.37484133 0.54192734]
[0.8302536 0.4088725 ]], shape=(2, 2), dtype=float32)
1.3 Create Tensors by special value function of TensorFlow
tf.ones: Create a tensor with all elements set to 1. This operation returns a dtype tensor with a shape of all elements are set to 1.
# tf.ones
print(tf.ones(shape=(1,3),dtype=tf.float32))
# Returns (out) =>
tf.Tensor([[1. 1. 1.]], shape=(1, 3), dtype=float32)
tf.zeros: Create a tensor with all elements set to 0. This operation returns a dtype tensor with a shape of all elements are set to 0.
# tf.zeors
print(tf.zeros(shape=(1,3),dtype=tf.float32))
# Returns (out) =>
tf.Tensor([[0. 0. 0.]], shape=(1, 3), dtype=float32)
tf.eye: Construct an identity matrix or a batch of matrices.
# tf.eye
print(tf.eye(4, dtype=tf.float32))
# Returns (out) =>
tf.Tensor(
[[1. 0. 0. 0.]
[0. 1. 0. 0.]
[0. 0. 1. 0.]
[0. 0. 0. 1.]], shape=(4, 4), dtype=float32)
tf.ones_like: Create a tensor with all elements set to 1. Given a tensor, this operation returns a tensor of the same type and shape as the tensor with all elements set to 1. Optionally, it can specify the tensor with a new type (dtype).
# tf.ones_like
print(tf.ones_like(tf.linspace(0.0, 1.0, 4)))
# Returns (out) =>
tf.Tensor([1. 1. 1. 1.], shape=(4,), dtype=float32)
tf. zeros_like: Create a tensor with all elements set to 0. Given a tensor, this operation returns a tensor of the same type and shape as the tensor with all elements set to 0. Optionally, it can specify the tensor with a new type (dtype).
# tf.zeros_like
print(tf.zeros_like(tf.linspace(0.0, 1.0, 4)))
# Returns (out) =>
tf.Tensor([0. 0. 0. 0.], shape=(4,), dtype=float32)
tf. linspace: An calculation instruction that evenly divides the given ‘start’ and ‘stop’ numbers by ‘num’ and generates values in the interval.
# tf.linspace
print(tf.linspace(start=0.0,stop=1.0, num=3))
# Returns (out) =>
tf.Tensor([0. 0.5 1. ], shape=(3,), dtype=float32)
tf.fill: Creates a tensor filled with a scalar value. This operation creates a tensor of shape dims and fills it with value.
# tf.fill
print(tf.fill(dims=(1,5), value=1.0))
# Returns (out) =>
tf.Tensor([[1. 1. 1. 1. 1.]], shape=(1, 5), dtype=float32)
2 Tensor merge and split
Merging refers to merging multiple tensors into one tensor in a certain dimension. There are two kinds of Merging-concatenation and stacking. Concatenation does not create new dimensions, but stacking creates new dimensions.
The concatenation can be performed on any dimension. The only constraint is that the length of the non-concatenated dimensions must be the same.
a = tf.random.normal([1,2,3])
b = tf.random.normal([2,2,3])
c = tf.concat([a,b], axis=0)
print(c)
# Returns (out) =>
tf.Tensor(
[[[-0.27534676 -0.16034059 0.52429354]
[-0.5917587 -0.23074977 -1.3515841 ]]
[[-0.4800986 -1.7884752 -0.75610226]
[-1.7305112 -0.7396962 1.0428202 ]]
[[-0.33405465 0.3331637 -0.26588857]
[-0.19221431 -1.5180566 -0.67606276]]], shape=(3, 2, 3), dtype=float32)
tf.concat directly merges data on existing dimensions and does not create a new dimension. If you want to create a new dimension when merging data, you need to use the tf.stack. The parameter axis specifies where to insert the new dimension. When an axis is a positive number, it will insert a new dimension in front of axis. When the axis is negative, it will insert a new dimension in the next position.
tf.stack also needs to meet the merging conditions of tensor stacking. It requires all merged tensors to have the same shape before they are merged.
a = tf.random.normal([2,3])
b = tf.random.normal([2,3])
c = tf.stack([a,b], axis = 0)
print(c)
# Returns (out) =>
tf.Tensor(
[[[ 0.7345577 0.6821109 1.2609174 ]
[ 1.6845615 0.7451898 0.04291777]]
[[-0.33740672 0.71333706 1.2531255 ]
[ 0.66583854 -0.8219059 0.278419 ]]], shape=(2, 2, 3), dtype=float32)
d = tf.stack([a,b], axis = -1)
print(d)
# Returns (out) =>
tf.Tensor(
[[[ 0.53225416 1.041896 ]
[-0.66618955 0.03342717]
[-1.6268841 0.18227476]]
[[ 0.11992308 0.33611476]
[ 0.14574063 -0.13129552]
[-0.59104127 1.382147 ]]], shape=(2, 3, 2), dtype=float32)
The inverse process of the merge operation is splitting. splitting is to split a tensor into multiple tensors. There are two kinds of split-split and unstack. Pay attention to the output shape.
#split
x = tf.random.normal([3,2,3])
print(x)
result = tf.split(x, axis=0, num_or_size_splits =3)
print(result)
# Returns (out) =>
tf.Tensor(
[[[ 0.3446128 1.8321378 -0.3682034 ]
[-0.59962803 -1.3437859 -0.20841458]]
[[-0.28618914 -0.37867725 -2.6242352 ]
[-1.4301056 -2.365182 -0.52069366]]
[[-0.5681175 2.4599526 -0.03526611]
[ 0.7553041 1.8161643 -0.7825681 ]]], shape=(3, 2, 3), dtype=float32)
[<tf.Tensor: shape=(1, 2, 3), dtype=float32, numpy=
array([[[ 0.3446128 , 1.8321378 , -0.3682034 ],
[-0.59962803, -1.3437859 , -0.20841458]]], dtype=float32)>,
<tf.Tensor: shape=(1, 2, 3), dtype=float32, numpy=
array([[[-0.28618914, -0.37867725, -2.6242352 ],
[-1.4301056 , -2.365182 , -0.52069366]]], dtype=float32)>,
<tf.Tensor: shape=(1, 2, 3), dtype=float32, numpy=
array([[[-0.5681175 , 2.4599526 , -0.03526611],
[ 0.7553041 , 1.8161643 , -0.7825681 ]]], dtype=float32)>]
#unstack
x = tf.random.normal([3,2,3])
print(x)
result = tf.unstack(x,axis = 0)
print(result)
# Returns (out) =>
tf.Tensor(
[[[-0.45672733 -0.6107957 -1.5969335 ]
[-0.05063461 0.90097845 -0.51631683]]
[[ 1.440197 -0.28877714 -0.52491015]
[-2.1945505 0.01744763 0.35254684]]
[[-0.66387564 0.63262576 -0.14352717]
[-0.08475702 -1.186847 0.7461847 ]]], shape=(3, 2, 3), dtype=float32)
[<tf.Tensor: shape=(2, 3), dtype=float32, numpy=
array([[-0.45672733, -0.6107957 , -1.5969335 ],
[-0.05063461, 0.90097845, -0.51631683]], dtype=float32)>,
<tf.Tensor: shape=(2, 3), dtype=float32, numpy=
array([[ 1.440197 , -0.28877714, -0.52491015],
[-2.1945505 , 0.01744763, 0.35254684]], dtype=float32)>,
<tf.Tensor: shape=(2, 3), dtype=float32, numpy=
array([[-0.66387564, 0.63262576, -0.14352717],
[-0.08475702, -1.186847 , 0.7461847 ]], dtype=float32)>]
3. Tensor comparison and sort
In actual work scenarios, we may only be interested in data that meets certain conditions, or needs to count the accuracy of a task and other indicators. Generally, we need to compare the predicted result with the true label. We need to calculate accuracy rate though the results that are the same as the labels or greater than a certain threshold. TensorFlow provides tensor comparison operations, including equal or greater or not.
y = tf.random.uniform([10],dtype=tf.int64,maxval=10)
print(y)
y_hat = tf.random.uniform([10],dtype=tf.int64,maxval=10)
print(y_hat)
result = tf.equal(y_hat,y)
print(result)
print(y_hat > 5)
# Returns (out) =>
tf.Tensor([4 8 8 3 5 8 2 1 6 7], shape=(10,), dtype=int64)
tf.Tensor([0 2 2 2 3 9 8 2 4 8], shape=(10,), dtype=int64)
tf.Tensor([False False False False False False False False False False], shape=(10,), dtype=bool)
tf.Tensor([False False False False False True True False False True], shape=(10,), dtype=bool)
In some cases, we need data that is ranked higher. TensorFlow 2.x provides a variety of sorting functions. In addition to the ordinary ascending and descending order, there are more convenient operations, like top data and the index of the largest data.
tensor = tf.random.uniform([10], maxval=10)
print(tensor)
# sort
print(tf.sort(tensor, direction=’DESCENDING’))
# indice of sort result
print(tf.argsort(tensor, direction=’DESCENDING’))
# top k
print(tf.math.top_k(a, 3))
# set all elements greater than 4
print(tf.maximum(a, 5))
# set all elements smaller than 4
print(tf.minimum(a, 5))
# set all elements greater than 3 smaller than 7
print(tf.clip_by_value(a, 3, 7))
# Returns (out) =>
tf.Tensor(
[8.151581 2.8347158 8.287186 1.431247 5.478798 4.6553183 5.820148
4.525962 7.1494665 6.572027 ], shape=(10,), dtype=float32)
tf.Tensor(
[8.287186 8.151581 7.1494665 6.572027 5.820148 5.478798 4.6553183
4.525962 2.8347158 1.431247 ], shape=(10,), dtype=float32)
tf.Tensor([2 0 8 9 6 4 5 7 1 3], shape=(10,), dtype=int32)
TopKV2(values=<tf.Tensor: shape=(3,), dtype=float32, numpy=array([8.287186 , 8.151581 , 7.1494665], dtype=float32)>, indices=<tf.Tensor: shape=(3,), dtype=int32, numpy=array([2, 0, 8], dtype=int32)>)
tf.Tensor(
[8.151581 5. 8.287186 5. 5.478798 5. 5.820148
5. 7.1494665 6.572027 ], shape=(10,), dtype=float32)
tf.Tensor(
[5. 2.8347158 5. 1.431247 5. 4.6553183 5.
4.525962 5. 5. ], shape=(10,), dtype=float32)
tf.Tensor(
[7. 3. 7. 3. 5.478798 4.6553183 5.820148
4.525962 7. 6.572027 ], shape=(10,), dtype=float32)
In big data processing, what we often care about is not each element’s value, but the average, maximum, minimum, and corresponding index in a certain dimension (default global dimension).
tensor = tf.constant([[1,2],[3,4]], dtype=tf.float32)
# max in dimension 1
print(tf.reduce_max(tensor, axis=1))
# min in dimension 0
print(tf.reduce_min(tensor, axis=0))
# average, default global dimension
print(tf.reduce_mean(tensor))
# index of max value in dimension 0
print(tf.argmax(tensor, axis=0))
# index of min value in dimension 1
print(tf.argmin(tensor, axis=1))
# Returns (out) =>
tf.Tensor([2. 4.], shape=(2,), dtype=float32)
tf.Tensor([1. 2.], shape=(2,), dtype=float32)
tf.Tensor(2.5, shape=(), dtype=float32)
tf.Tensor([1 1], shape=(2,), dtype=int64)
tf.Tensor([0 0], shape=(2,), dtype=int64)
One more example, after we use the loss function getting each sample’s loss, the average loss of the samples needs to be calculated.
y_hat = tf.random.normal([10,5])
y = tf.constant([1, 2, 3, 4, 5, 1, 2, 3, 4, 5])
y = tf.one_hot(y,depth=5)
loss = tf.keras.losses.mse(y,y_hat)
loss_mean = tf.reduce_mean(loss)
print(loss_mean)
# Returns (out) =>
tf.Tensor(1.4730119, shape=(), dtype=float32)
Of course, you can also get the maximum and minimum loss, even their indices which indicates the sample with maximum loss or minimum loss.
loss_max = tf.reduce_max(loss)
loss_min = tf.reduce_min(loss)
idx_max = tf.argmax(loss)
idx_min = tf.argmin(loss)
print(loss)
print(loss_max)
print(loss_min)
print(idx_max)
print(idx_min)
# Returns (out) =>
tf.Tensor(
[0.665666 0.50770426 0.37840793 0.550116 0.674733 0.5545435
2.1785192 2.1507242 0.500903 1.2179064 ], shape=(10,), dtype=float32)
tf.Tensor(2.1785192, shape=(), dtype=float32)
tf.Tensor(0.37840793, shape=(), dtype=float32)
tf.Tensor(6, shape=(), dtype=int64)
tf.Tensor(2, shape=(), dtype=int64)
4. Mathematical operations
TensorFlow 2.x support mathematical operations much more friendly that TensorFlow 1.x. The commonly used operations are listed below. It is clear and absolutely meets python programming habit.
a = tf.constant([[1,2],[3,4]], dtype=tf.float32)
b = tf.constant([[5,6],[7,8]], dtype=tf.float32)
# element-wise add
print(a+b)
# Returns (out) =>
f.Tensor(
[[ 6. 8.]
[10. 12.]], shape=(2, 2), dtype=float32)
# element-wise minus
print(a-b)
# Returns (out) =>
tf.Tensor(
[[-4. -4.]
[-4. -4.]], shape=(2, 2), dtype=float32)
# element-wise multipy
print(a*b)
# Returns (out) =>
tf.Tensor(
[[ 5. 12.]
[21. 32.]], shape=(2, 2), dtype=float32)
# element-wise divide
print(‘a/b=’, a/b)
# Returns (out) =>
tf.Tensor(
[[0.2 0.33333334]
[0.42857143 0.5 ]], shape=(2, 2), dtype=float32)
# element-wise floor division
print(‘a//b’, a//b)
# Returns (out) =>
tf.Tensor(
[[0. 0.]
[0. 0.]], shape=(2, 2), dtype=float32)
# element-wise remainder
print(‘b%a=’, b%a)
# Returns (out) =>
tf.Tensor(
[[0. 0.]
[1. 0.]], shape=(2, 2), dtype=float32)
# square
print(a**2)
# square
print(tf.pow(a, 2.0))
# square
print(tf.square(a))
# Returns (out) =>
tf.Tensor(
[[ 1. 4.]
[ 9. 16.]], shape=(2, 2), dtype=float32)
tf.Tensor(
[[ 1. 4.]
[ 9. 16.]], shape=(2, 2), dtype=float32)
tf.Tensor(
[[ 1. 4.]
[ 9. 16.]], shape=(2, 2), dtype=float32)
# square root
print(tf.sqrt(a))
# Returns (out) =>
tf.Tensor(
[[1. 1.4142135]
[1.7320508 2. ]], shape=(2, 2), dtype=float32)
# exp
print(tf.exp(a))
# Returns (out) =>
tf.Tensor(
[[ 2.7182817 7.389056 ]
[20.085537 54.59815 ]], shape=(2, 2), dtype=float32)
# log
print(tf.math.log(a))
# Returns (out) =>
tf.Tensor(
[[0. 0.6931472]
[1.0986123 1.3862944]], shape=(2, 2), dtype=float32)
# maxtrix multipy
print(tf.matmul(a, b))
# Returns (out) =>
tf.Tensor(
[[19. 22.]
[43. 50.]], shape=(2, 2), dtype=float32)
Other series of Understanding TensorFlow:
Serie1: https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562
Serie2: https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08
Serie3–1: https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc
Serie3–2: https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368
Serie4: https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9
