I am a senior Information Technology student at Thiagarajar College of Engineering, Madurai. My folks Revanth M S and Thilagraj are the senior students pursuing Civil Engineering and Production Engineering respectively at the National Institute of Technology, Trichy.
Harvard business review regarded “Data Scientist” as the sexiest job in the 21st century. GlassDoor named it as the best job in America for this year. These are not the only motivations that connected us in pursuing data science in our free time but also the business and societal impact that we can make with the data!
I am very happy to share that we are part of the 30 engineering students selected for the ACM Summer School of Data Science, Goa among 1000+ nationwide applicants. We thank ACM India, Robert Bosch Centre for Data Science and Artificial Intelligence (IIT Madras) and Goa University for the opportunity to explore this disruptive topic at the India’s most beautiful place ❤. This 2 week program covers topics ranging from mathematical foundations to advance topics in data science like deep learning and reinforcement learning with the blend of theory and hands-on sessions by the domain experts from various Indian Institute of Technologies. The video recordings of theory sessions will be made available to the public through the Swayam portal which is initiated by the Ministry of Human Resources and Development, Government of India. However we decided to share our *important* learning precisely on each day :)
TABLE OF CONTENTS:
1. Speaker Introduction
2. Why Learn Linear Algebra?
3. Vectors And Matrices
4. Vector Properties
== a. Scalar Multiplication
== b. Vector Addition
5. Vector Operations
== a. Distance Between Two Vectors
== b. Dot Product
== c. Scalar Projection
6. Linear Combination Of Vectors
7. Matrix Rank
8. Types Of Matrices
9. Solving System of Linear Equations
== a. Naive Gauss Elimination
== b. LU Decomposition
== c. QR Factorization
10. Eigen Values And Eigen Vectors
1. Speaker Introduction:
Dr. Sridharakumar Narasimhan is an Assistant Professor at the Department of Chemical Engineering, IIT Madras. He received his M.Tech (Integrated course) degree in Chemical Engineering from IIT Bombay and started his graduate studies in Chemical Engineering and obtained his PhD from Clarkson University in 2006. From 2006–2008 he was a post-Doctoral Scholar in the department of Chemical Engineering, NTNU, Trondheim, Norway. His research interests are in process operations (control, fault diagnosis), system identification and optimization.
2. Why Learn Linear Algebra?
A data set is typically represented as the NxD matrix where N is the number of observations and D is the number of dimensions and is usually large. As human beings tend to visualize only up to 3 dimensions, linear algebra helps us to extend our observations on smaller dimensions to larger dimensions.
For example, the image can be represented pixel by pixel in a matrix format. Linear algebra is also largely employed in solving the system of linear equations.
3. Vectors And Matrices
Vector is a quantity having magnitude and direction that determines the position of a point in space relative to another. It can be used to represent individual data points in the data set.
2D Representation: A vector v(1,2) in XY axes. Here the X component of the vector ‘v’ is 1 and Y component of ‘v’ is 2.
Matrix can be referred as a vector of vectors usually in the order of NxD.
4. Properties of vectors:
The vector space should satisfy the following properties:
a) Scalar multiplication:
It is nothing but multiplying a number and multiply with every element of a matrix. The resultant vector will be collinear with the old vector.
b) Vector addition:
Two or more vectors are added and the resultant sum is a vector. Every element of a matrix in the same row will be added to get a resultant element.
5. Vector Operations
a) Distance Between Two Vectors
Let us consider two vectors P(x1, y1) and Q(x2, y2) in the two dimensional vector space. The distance between them can be computed using the Pythagoras theorem. It states that the square of the hypotenuse (the side opposite the right angle) is equal to the sum of the squares of the other two sides in the right angled triangle.
b) Dot Product
The dot product is an operation that takes two vectors as input, and that returns a scalar number as output. The number returned is dependent on the length of both vectors, and on the angle between them. The dot product of two vectors a = [a1, a2, …, an] and b = [b1, b2, …, bn] is defined as:
c) Scalar Projection
The scalar projection (or scalar component) of a Euclidean vector a in the direction of a Euclidean vector b is given by:
6) Linear combination of vectors:
It can be expressed as a combination of the two properties of vector space.
y=α1x1 + α2x2 + α3x3 +…..+ αnxn
We can find whether the vector is linearly independent or not with the help of above equation.
But, wait! What is meant by linearly independent vectors? It is said that if the vector is linearly independent, it cannot be written in above form ( as a linear combination of vectors) It is not as easy as it seems to be. There are number of equations in this world which seem to be linearly independent but it is actually linearly dependent. There are some easy ways to say the vector is linearly independent,
α1 = α2 = …….. = αn = 0
When we write those vectors in matrix form, det(matrix) ≠ 0
Quick fact: Determinant of vectors of a 2D matrix represents the area enclosed by those vectors and volume is represented by vectors of 3D matrix.
7. Matrix Rank:
Row rank: Number of linearly independent rows.
Column rank: Number of linearly independent columns.
Row rank ≤ number of rows in a matrix
Column rank ≤ number of column in a matrix
Wondering why rank intrude in between, actually dependency of vectors can also be found using rank of matrix formed by the vectors.
When the equation is in the form of Ax=B,
Solutions are unique if rank(A) = rank([AB])
Infinitely many solutions if rank(A) = rank([AB])
No solution if rank(A) < rank([AB])
8. Types Of Matrices:
Matrices can be used effectively to solve a system of linear equations in a simpler way. The order of increasing difficulty in solving the system of equations is as follows:
a) Diagonal Matrix:
A matrix that has data entries only on the main diagonal (left diagonal), and all the other values of the matrix is zero is called a diagonal matrix. Unlike other matrices, one could visibly solve the equations that are represented in the form of a diagonal matrix.
b) Lower Triangular matrix:
A matrix that has data entries all over the lower triangle of the matrix i.e. all the points below the main diagonal has some value and all other entries in the matrix is zero, is called a Lower Triangular matrix. The system of linear equations can be solved through a method known as the ‘Forward Sweep’.
c) Upper Triangular matrix:
A matrix that has data entries all over the upper triangle of the matrix .i.e all the points above the main diagonal has some value and all other entries in the matrix is zero, is called an Upper Triangular matrix. The system of linear equations can be solved through a method similar to the ‘Forward sweep’ in the backward direction known as ‘Backward Sweep’.
d) Any other matrix:
A matrix that has entries all over the matrix is the toughest to solve the system of linear equations. It has various methods to solve but is usually a time consuming process.
9. Solving system of Linear Equations:
a) Gauss elimination:
It is a method of solving linear equations by eliminating some of the elements of a matrix and creating either lower or upper diagonal matrix so that the bottom or top of the matrix will exactly have one variable in the linear equation, the last but one have two variable and so on. The equations are solved with the help of solutions got from previous equation.
b) LU Decomposition:
LU Decomposition technique is one of the most common and easy way to solve the system of linear equations. One would appreciate this method of solving the equations when coding algorithms to solve the equations. L U decomposition of a matrix is the factorization of a given square matrix into two triangular matrices, one upper triangular matrix and one lower triangular matrix, such that the product of these two matrices gives the original matrix. This method comes in handy to solve same set of equations with only the constant term changing.
Steps in L U Decomposition:
1. Factorize the given square matrix into LU such that L is a lower triangular matrix and U is an upper triangular matrix.
2. Solve for y in Ly = b.
3. Solve for x in Ux=y
X is the required solution of the equation Ax = b. The main advantage of this method is that for any matrix if we can find the factors L and U, for any value ‘b’ we could easily solve the equations.
c) Q R Factorization:
Two vectors are said to be orthogonal if the are perpendicular to each other. That is their dot product is equal to zero.
Orthogonal vectors that have an unit magnitude is called orthonormal vectors.
This process is also called as the ‘Gram Schmidt process’. It involves in factorization of a matrix in the form Q R where Q is the orthonormal matrix .i.e that is all the vectors are perpendicular to each other and have a unit magnitude. R is an upper triangular matrix that has the dot product of the row vectors of the A matrix and the column vectors of the Q matrix.
The shortest distance between any two vectors is the perpendicular distance between them. When a matrix is over-determined .i.e when the number of equations are more than the unknowns, a unique solution can not be found all the time. In such cases, we more often get what is called the ‘Best Estimated solution’ with the method known as least Squares.
For the system A x = b ,the least squares formula is obtained from the problem ,
min x ‖ A x − b ‖ , the solution of which can be written with the normal equations,
x = ( A ^T A ) ^− 1 A^ T b
10. Eigen Values And Eigen Vectors:
A vector that does not get rotated after undergoing a matrix transformation is called EigenVector. Eigenvalues and eigenvectors are particularly useful when finding the nth degree of a matrix. If S is a matrix containing all the eigenvector and /\ be a diagonal matrix that contains all the eigenvalues as the entries of the diagonal elements, then we can prove that,
A^n = S /\^n S^-1
If the matrix is symmetric all eigenvalues are real, distinct and always independent. They are perpendicular to each other. In a singular matrix at least one eigenvalue is zero.
Its the time to verify your learning! Solve the problems in this link.
We referred some great contents from Wikipedia and Applied AI course for this write up. Thanks!