A quick journey through Conv1D functions from TensorFlow to PyTorch, passing via SciPy. Part 3

Causal Convolution

In this story we will talk about other two types of convolution:

• Causal convolution
• Full convolution

First of all let’s recall the definition of a causal system:

A system is called causal if : y(t0) = f(x(t)) with t ≤ t0, in words:

the output y of a system S at time t=t0 is a function of the input x(t) for values of t ≤t0.

In other words this means that a causal system cannot depend from future inputs but only from the present and the past values of the input. This property plays an important role in digital signal processing algorithms and in time series analysis.

As we did for part 2 of this series, we start our examples taking x[n] as input signal of length N = 4 and the filter (or kernel) w[n] of length M = 3

N = 4, M = 3
x[n] = [1, 2, 3, 4]
w[n] = [1, 1, 1]

TensorFlow

From the Tensorflow documentation we read that causal mode results in causal (dilated) convolutions, e.g. output[t] does not depend on input[t+1:]

In a very similar way respect to other modes, TF gives the possibility to compute the causal convolution simply setting padding="causal" in the Conv1D

x = np.array([1, 2, 3, 4]).astype(np.float32)
x = x.reshape([1, 4, 1])
w = np.array([1, 1, 1]).astype(np.float32)
conv1d = tf.keras.layers.Conv1D(1,3,
            kernel_initializer=tf.keras.initializers.constant(w),
            padding="causal",
            use_bias=False) 
y = conv1d(x)
print(y)
tf.Tensor([[[1.][3.][6.][9.]]], shape=(1, 4, 1), dtype=float32)

The output of the causal convolution is y[n] = [1, 3, 6, 9] of length O = 4

The causal convolution creates an output signal of the same length of the input. We have already seen that, in order to satisfy this requirement, we have to pad the input signal x with P = M — 1 zeros, (M — 1) / 2 zeros at left,
(M — 1) / 2 zeros at the right of the sequence x. In the causal mode, in order to preserve the causality we pad the sequence x with M — 1 zeros all at the left of the signal x:

x[n] = [1, 2, 3, 4]
x_pad[n] = [0, 0, 1, 2, 3, 4]

PyTorch

PyTorch does not directly give the possibility to select the causal modality as we did in TensorFlow. But we can set the padding we want. In our case, we set the padding = M — 1 = 2 but reading the PyTorch code documentation you find that padding is added to both sides of the input.

This means that setting padding = 2 , x will padded as:

x[n] = [1, 2, 3, 4]
x_pad[n] = [0, 0, 1, 2, 3, 4, 0, 0]

We will use this padding later when we will talk about the full convolution mode, but, for the moment, we don’t need the two extra zeros padded to the right.

x = np.array([1, 2, 3, 4]).astype(np.float32)
x = x.reshape([1, 1, 4])
w = np.array([1, 1, 1]).astype(np.float32)
x_torch = torch.from_numpy(x)
wt = w.reshape([3, 1])
conv1d = nn.Conv1d(1, 1, 3, bias=False, padding=2) 
conv1d.weight.data.fill_(1)
y_full = conv1d(x_torch)
print(y_full)
tensor([[[1., 3., 6., 9., 7., 4.]]]

As we can see the output is y[n] = [1, 3, 6, 9, 7, 4 ] (and this represents the output of a full convolution). In order to have a causal convolution we must be restrict the output only for the first N values or, in other words, we must discard the last M-1 samples.

y_causal = y_full[:, :, :-2]
print(y_causal)
tensor([[[1., 3., 6., 9.]]]

As we see, the result is the same produced by TF.

Full Convolution

A full convolution is done by applying the filter w[n] to every sample of the input signal. To do this, the input signal x must be padded with M-1 zeros at the left and M-1 zeros at the right, as we obtained with the previous PyTorch method. The output signal y[n] will have O = N + M — 1 samples. Doing calculations by hand i will show how it works.

x[n] = [1, 2, 3, 4]  # Take the input signal
x_pad[n] = [0, 0, 1, 2, 3, 4, 0, 0] 
#Pad x with O = M - 1 zeros at left, M - 1 zeros at right

y[0] = x_pad[0] * w[0] + x_pad[1] * w[1] + x_pad[2] * w[2] = 1 
y[1] = x_pad[1] * w[0] + x_pad[2] * w[1] + x_pad[3] * w[2] = 3
y[2] = x_pad[2] * w[0] + x_pad[3] * w[1] + x_pad[4] * w[2] = 6
y[3] = x_pad[3] * w[0] + x_pad[4] * w[1] + x_pad[5] * w[2] = 9
y[4] = x_pad[4] * w[0] + x_pad[5] * w[1] + x_pad[6] * w[2] = 7
y[5] = x_pad[5] * w[0] + x_pad[6] * w[1] + x_pad[7] * w[2] = 4

PyTorch

Setting conv1d = nn.Conv1d(1, 1, 3, bias=False, padding=2) automatically produces a full convolution.

TensorFlow

TensorFlow does not have a full mode convolution although it is always possible to pad manually the signal x and then using the conv1d with valid mode. But in the next story we will see how to do this in a smarter way using conv1dTranspose layer and we will see the similarities between this and conv1d. So, see you next time!

Happy reading,
Stefano