Neural Network 01 — Prerequisites
Welcome to my neural network blog. Here onwards you will be learning plenty of useful theories and concepts behind neural network and deep learning.
Before get into actual neural network, you need to prepare yourself and familiarize with some mathematical and statistical concepts. That will definitely help you to understand interesting intuitions behind this awesome neural network and deep learning concepts.
I will explain couple of useful Python tips and tricks which you should remember along the way.
Let’s get started!
Scalars, Vectors and Matrices
(If you are very good at Algebra, you can ignore this lesson.)
Scalar: Any real number is a “scalar”
Vector: Consists of a single column of real numbers. Number of elements of a vector called the dimension of the vector.
Matrix: Consists of n rows and m columns of real numbers.
Matrices can be multi-dimensional as well.
Vector addition
To add two or more vectors together, they must be in the same dimension.
Vector * Scalar multiplication
Vectors have two properties. Magnitude and direction. When doing multiplication between vector A and a scalar n > 0, nA will have the same direction but n times magnitude.
Linear combinations
A linear combination of vectors is a mathematical operation that involves multiplying each vector by a scalar and then adding the results together. The result of this operation is also a vector.
Here each “cᵢ” represents a scalar coefficient associated with vector “vᵢ”.
Linear combination of vectors play a crucial role in Neural network. For example: Weighted Sum, the linear combination of input vectors with learnable weights is effectively a weighted sum of these inputs. This weighted sum helps the network learn to assign different levels of importance to various input features.
Matrix multiplication
Important rule: The number of columns of first matrix must be equal to the number of rows of the second matrix.
Remember: We have to use “Transpose” operation when we perform vector/matrix multiplication when we do real coding in Python.
Matrix multiplication is a binary operation. The element in the resulting matrix C at the iᵗʰ row and jᵗʰ column is calculated by taking the dot product of the iᵗʰ row of matrix A and jᵗʰ column of matrix B.
A notation to memorize:
“Dataset” is a common term we are using in Machine Learning world. Dataset usually comes in tabular form which contains rows and columns. Columns represent different features (eg: diameter of a cell, average rainfall, or number of bathrooms of a house). Rows represent the observations (set of values for different features at different situations).
We use matrices to load data in and apply them in our program logic to perform various machine learning/neural network tasks. So, following notation will be using in this lesson series. Therefore worth to remember this notation.
Reshaping matrices
Shape of a matrix refers to the dimension of it. In neural network, we reshape/change the dimension of matrices for various purposes.
In Python we can reshape matrices using numpy library. If you don’t know about it, no worries. Just focus on reshaping concept. That’s enough at this stage.
We can use “-1” as a new dimension to reshape matrix into a row vector or a column vector.
Example usage: In image processing we need to transform Red, Green, and Blue value matrices into 1D vector.
Python Broadcasting
Broadcasting is a very useful feature in Python when dealing with neural network tasks. What is basically does is, when we want to add a value to elements of a matrix we can do it as a normal addition operation as follows.
There are separate set of mathematical operations defined inside Python libraries like numpy, Tensorflow, …
I encourage you to use those library specific functions rather than using classic math operations comes with Python. If you use library specific functions they will return objects in relevant datatypes which is essential for bugs free and hassle free program.
Vectorization
In neural network, specially in deep learning, the efficiency of the code is very very important. That means every millisecond counts.
Explicit For-loops makes your program inefficient. So, when you perform complex matrix calculations, vectorization is a important topic to consider.
What is vectorization?
Vectorization refers to arranging data in matrices or tensors like data structures and perform computationally expensive operations (such as matrix multiplication) on entire dataset simultaneously. Vectorization allows parallelization and optimization of computations.
This is a simple example of vectorization in Python.
In real world, deep learning computations consist of matrices with thousands of millions of elements. Those computations are extremely expensive. In that case vectorized and non-vectorized implementations affect the performance of your system.
End of the lesson 01. You can continue learning with me. Go to next lesson and learn more Neural Network towards Deep Learning. Good Luck! Keep Learning! :)