# All the Backpropagation derivatives

So you’ve completed Andrew Ng’s Deep Learning course on Coursera,

You know that ForwardProp looks like this:

And you know that Backprop looks like this:

But do you know how to derive these formulas?

# TL;DR

Full derivations of all Backpropagation derivatives used in Coursera Deep Learning, using both chain rule and direct computation.

If you’ve been through backpropagation and not understood how results such as

and

are derived, if you want to understand the direct computation as well as simply using chain rule, then read on…

# Our Neural Network

This is the simple Neural Net we will be working with, where x,W and b are our inputs, the “z’s” are the linear function of our inputs, the “a’s” are the (sigmoid) activation functions and the final

is our Cross Entropy or Negative Log Likelihood cost function.

So here’s the plan, we will work backwards from our cost function

and compute directly, the derivative of

with respect to (w.r.t) each of the preceding elements in our Neural Network:

As well as computing these values directly, we will also show the chain rule derivation as well.

# Note: we don’t differentiate our input ‘X’ because these are fixed values that we are given and therefore don’t optimize over.

#  Derivative w.r.t activation function

So to start we will take the derivative of our cost function

w.r.t the activation function

So we are taking the derivative of the Negative log likelihood function (Cross Entropy) , which when expanded looks like this:

First lets move the minus sign on the left of the brackets and distribute it inside the brackets, so we get:

Next we differentiate the left hand side:

The right hand side is more complex as the derivative of ln(1-a) is not simply 1/(1-a), we must use chain rule to multiply the derivative of the inner function by the outer.

The derivative of (1-a) = -1, this gives the final result:

And the proof of the derivative of a log being the inverse is as follows:

#  Derivative of sigmoid

It is useful at this stage to compute the derivative of the sigmoid activation function, as we will need it later on.

our logistic function (sigmoid) is given as:

First is is convenient to rearrange this function to the following form, as it allows us to use the chain rule to differentiate:

Now using chain rule: multiplying the outer derivative by the inner, gives

which rearranged gives

Here’s the clever part. We can then separate this into the product of two fractions and with a bit of algebraic magic, we add a ‘1’ to the second numerator and immediately take it away again:

The RHS then simplifies to

Which is nothing more than

Which gives a final result of

Or alternatively:

#  Derivative w.r.t linear function derivative of our linear function (z = wX + b)

To get this result we can use chain rule by multiplying the two results we’ve already calculated  and 

So if we can get a common denominator in the left hand of the equation, then we can simplify the equation, so lets add ‘(1-a)’ to the first fraction and ‘a’ to the second fraction

with a common denominator we can simplify to

now we multiply LHS by RHS, the a(1-a) terms cancel out and we are left with just the numerator from the LHS!

which if we expand out gives:

note that ‘ya’ is the same as ‘ay’, so they cancel to give

which rearranges to give our final result of the derivative

our final result is

#  Derivative w.r.t weights

This derivative is trivial to compute, as z is simply

and the derivative simply evaluates to

#  Derivative w.r.t weights (2)

This derivative can be computed two different ways! We can use chain rule or compute directly. We will do both as it provides a great intuition behind backprop calculation.

To use chain rule to get derivative  we note that we have already computed the following

Noting that the product of the first two equations gives us

if we then continue using the chain rule and multiply this result by

then we get

which is nothing more than

or written out long hand

So that’s the ‘chain rule way’. Now lets compute ‘dw’ directly:

To compute directly, we first take our cost function

We can notice that the first log term ‘ln(a)’ can be expanded to

Which simplifies to:

And if we take the second log function ‘ln(1-a)’ which can be shown as

taking the log of the numerator ( we will leave the denominator) we get

This result comes from the rule of logs, which states: log(p/q) = log(p) — log(q).

Plugging these formula back into our original cost function we get

Expanding the term in the square brackets we get

The first and last terms ‘yln(1+e^-z)’ cancel out leaving:

Which we can rearrange by pulling the ‘yz’ term to the outside to give

Here’s where it gets interesting, by adding an exp term to the ‘z’ inside the square brackets and then immediately taking its log

next we can take advantage of the rule of sum of logs: ln(a) + ln(b) = ln(a.b) combined with rule of exp products:e^a * e^b = e^(a+b) to get

followed by

Pulling the ‘yz’ term inside the brackets we get :

Finally we note that z = Wx+b therefore taking the derivative w.r.t W:

The first term ‘yz ’becomes ‘yx ’and the second term becomes :

Note that the 2nd term is nothing but

Which gives a final result of

We can rearrange by pulling ‘x’ out to give

which gives

#  derivative w.r.t bias

Again we could use chain rule which would be

This is easy to solve as we already computed ‘dz’ and the second term is simply the derivative of ‘z’ which is ‘wX +b’ w.r.t ‘b’ which is simply 1!

so the derivative w.r.t b is simply

which we already calculated earlier as

For completeness we will also show how to calculate ‘db’ directly. To calculate this we will take a step from the above calculation for ‘dw’, (from just before we did the differentiation)

remembering that z = wX +b and we are trying to find derivative of the function w.r.t b, if we take the derivative w.r.t b from both terms ‘yz’ and ‘ln(1+e^z)’ we get

its important to note the parenthesis here, as it clarifies how we get our derivative.

Taking the LHS first, the derivative of ‘wX’ w.r.t ‘b’ is zero as it doesn’t contain b! The derivative of ‘b’ is simply 1, so we are just left with the ‘y’ outside the parenthesis.

for the RHS, we do the same as we did when calculating ‘dw’, except this time when taking derivative of the inner function ‘e^wX+b’ we take it w.r.t ‘b’ (instead of ‘w’) which gives the following result (this is because the derivative w.r.t in the exponent evaluates to 1)

this term is simply our original

so putting the whole thing together we get

which we have already show is simply ‘dz’!

So that concludes all the derivatives of our Neural Network. We have calculated all of the following:

# Wrapping up

well, we can unpack the chain rule to explain:

Note that the term

is simply ‘dz’ the term we calculated earlier:

and the term

evaluates to W[l] or in other words, the derivative of our linear function Z =’Wa +b’ w.r.t ‘a’ equals ‘W’.

and finally the term in blue

is simply

‘da/dz’ the derivative of the the sigmoid function that we calculated earlier!

As a final note on the notation used in the Coursera Deep Learning course, in the result

we perform element wise multiplication between DZ and g’(Z), this is to ensure that all the dimensions of our matrix multiplications match up as expected.

# So there we have it…

… all the derivatives required for backprop as shown in Andrew Ng’s Deep Learning course.

Simply reading through these calculus calculations (or any others for that matter) won’t be enough to make it stick in your mind. The best way to learn is to lock yourself in a room and practice, practice, practice!

# What next?

If you got something out of this post, please share with others who may benefit, follow me Patrick David for more ML posts or on twitter @pdquant and give it a cynical/pity/genuine round of applause!

--

--

## More from Patrick David

Machine Learning , Deep Learning, AI, Python, Quant Trading -ML Researcher - follow for walkthroughs, tutorials, proofs, research etc. Also on twitter @pdquant Machine Learning , Deep Learning, AI, Python, Quant Trading -ML Researcher - follow for walkthroughs, tutorials, proofs, research etc. Also on twitter @pdquant