DL : Basic Concept of Neural Network

Part 1.1 of Deep Learning Specialization

1-layer Neural Network

= input layer + output layer

2-layer Neural Network

= input layer + hidden layer + output layer

4-layer Neural Network

= input layer + 3 hidden layers + output layer

L-layer Neural Network

= input layer + L-1 hidden layers + output layer

L = hidden layers + output layer

(input layer is not counted !!)

1. Elements in Neural Network

W[L] = weight vector of layer L

B[L] = bias vector of layer L

Z[L] = linear combination vector of layer L

= (W[L]• A[L-1]) + B[L]

A[L] = activation vector of layer L

= g (Z[L])

g() = activation function

A[0] = input vector

A[L] = output vector

2. Neural Network : Step-by-Step

initialize parameters W, B

train input (loop = num_iterations)

• forward propagation
• compute loss & cost function
• backward propagation (gradient descent)
• update parameters W, B

predict output

• forward propagation

Step 1 : Initialize Parameters W, B

W must be small, non-zero value vector to break symmetry of node’s weight

B can be a zero vector

Step 2: Forward Propagation

forward propagation ( link )

Step 3: Compute Loss & Cost Function

loss function provide a loss of being wrong per 1 training data.

cost function provide average loss on m training data.

Step 4: Backward Propagation (Gradient Descent)

backward cache (dZ[L])

use forward cache (W[L], B[L], A[L-1]) and backward cache (dZ[L]) to compute gradient descent (dW[L], dB[L], dA[L-1])

Step 5: Update Parameters W, B

alpha = learning rate

Step 6: Predict

Related Articles

• Neural Network — Part 1.1
• Neural Network — Part 1.2
• Neural Network— Part 2
• Neural Network — Part 3
• CNN — Part 4.1
• CNN — Part 4.2
• RNN— Part 5.1
• RNN— Part 5.2

Reference

Deep Learning Specialization: Neural Networks and Deep Learning (Coursera) (Youtube)