57.Numpy-Matrix and Matrix operations

Matrices and Matrix Arithmetic

Matrices are a foundational element of linear algebra. Matrices are used throughout the field of machine learning in the description of algorithms and processes such as the input data variable (X) when training an algorithm.A matrix is a two-dimensional array of scalars with one or more columns and one or more rows.

Defining a Matrix

We can represent a matrix in Python using a two-dimensional NumPy array. A NumPy array can be constructed given a list of lists. For example, below is a 2 row, 3 column matrix.

# create matrix

from numpy import array

A = array([[1, 2, 3], [4, 5, 6]])

print(A)

o/p:

[[1 2 3]

[4 5 6]]

Matrix Addition

# matrix addition
from numpy import array
# define first matrix
A = array([[1, 2, 3],[4, 5, 6]])
print(A)
# define second matrix
B = array([[1, 2, 3],[4, 5, 6]])
print(B)
# add matrices
C = A + B  # or  np.add(A,b)
print(C)
o/p:
[[1 2 3]
[4 5 6]]
[[1 2 3]
[4 5 6]]
[[ 2 4 6]
[ 8 10 12]]

Matrix Subtraction
# matrix subtraction
from numpy import array
# define first matrix
A = array([[1, 2, 3],[4, 5, 6]])
print(A)
# define second matrix
B = array([[0.5, 0.5, 0.5],[0.5, 0.5, 0.5]])
print(B)
# subtract matrices
C = A - B # or np.subtract(A,B)
print(C)
o/p:
[[1 2 3]
[4 5 6]]
[[ 0.5 0.5 0.5]
[ 0.5 0.5 0.5]]
[[ 0.5 1.5 2.5]
[ 3.5 4.5 5.5]]

Matrix Multiplication (Hadamard Product)

Two matrices with the same size can be multiplied together, and this is often called element-wise matrix multiplication or the Hadamard product. It is not the typical operation meant when referring to matrix multiplication.# matrix Hadamard product

from numpy import array
# define first matrix
A = array([[1, 2, 3],[4, 5, 6]])
print(A)
# define second matrix
B = array([[1, 2, 3],[4, 5, 6]])

print(B)
# multiply matrices
C = A * B # or np.multiply(A,B)
print(C)
o/p:
[[1 2 3]
[4 5 6]]
[[1 2 3]
[4 5 6]]
[[ 1 4 9]
[16 25 36]]

Matrix Division
# matrix division
from numpy import array
# define first matrix
A = array([[1, 2, 3],[4, 5, 6]])
print(A)
# define second matrix
B = array([[1, 2, 3],[4, 5, 6]])
print(B)
# divide matrices
C = A / B # or np.divide(A,B)
print(C)
o/p:
[[1 2 3]
[4 5 6]]
[[1 2 3]
[4 5 6]]
[[ 1. 1. 1.]
[ 1. 1. 1.]]

Matrix-Matrix Multiplication

Matrix multiplication, also called the matrix dot product is more complicated than the previous operations and involves a rule as not all matrices can be multiplied together.

C = A.B

or
C = AB
The rule for matrix multiplication is as follows:

The number of columns ($n$) in the first matrix (A) must equal the number of rows ($m$) in the second matrix (B).

For example, matrix A has the dimensions $m$ rows and $n$ columns and matrix B has the dimensions $n$ and $k$. The $n$ columns in A and $n$ rows in B are equal. The result is a new matrix with $m$ rows and $k$ columns.

$C(m; k) = A(m; n) . B(n; k)$

# matrix dot product
from numpy import array
# define first matrix
A = array([
[1, 2],
[3, 4],
[5, 6]])
print(A)
# define second matrix
B = array([
[1, 2],
[3, 4]])
print(B)
# multiply matrices
C = A.dot(B)
print(C)
# multiply matrices with @ operator
D = A @ B
print(D)
o/p:
[[1 2]
[3 4]
[5 6]]
[[1 2]
[3 4]]

[[ 7 10]
[15 22]
[23 34]]

[[ 7 10]
[15 22]
[23 34]]

Matrix-Vector Multiplication

A matrix and a vector can be multiplied together as long as the rule of matrix multiplication is observed. Speci cally, that the number of columns in the matrix must equal the number of items in the vector. As with matrix multiplication, the operation can be written using the dot notation. Because the vector only has one column, the result is always a vector.

$c = A . v$

# matrix-vector multiplication
from numpy import array
# define matrix
A = array([[1, 2],[3, 4],[5, 6]])
print(A)
# define vector
B = array([0.5, 0.5])
print(B)
# multiply
C = A.dot(B)
print(C)
o/p
[[1 2]
[3 4]
[5 6]]
[ 0.5 0.5]
[ 1.5 3.5 5.5]

Matrix-Scalar Multiplication

A matrix can be multiplied by a scalar. This can be represented using the dot notation between the matrix and the scalar.

C = A. b
# matrix-scalar multiplication
from numpy import array
# define matrix
A = array([[1, 2], [3, 4], [5, 6]])
print(A)
# define scalar
b = 0.5
print(b)
# multiply
C = A * b
print(C)
o/p:
[[1 2]
[3 4]
[5 6]]
0.5
[[ 0.5 1. ]
[ 1.5 2. ]
[ 2.5 3. ]]

Types of Matrices

A lot of linear algebra is concerned with operations on vectors and matrices, and there are many different types of matrices. There are a few types of matrices that you may encounter again and again when getting started in linear algebra, particularity the parts of linear algebra relevant to machine learning. In this tutorial, you will discover a suite of different types of matrices from the field of linear algebra that you may encounter in machine learning.

Square Matrix

A square matrix is a matrix where the number of rows (n) is equivalent to the number of columns (m).

Symmetric Matrix

A symmetric matrix is a type of square matrix where the top-right triangle is the same as the bottom-left triangle.It is no exaggeration to say that symmetric matrices S are the most important matrices the world will ever see in the theory of linear algebra and also in the applications.

Triangular Matrix

A triangular matrix is a type of square matrix that has all values in the upper-right or lower-left of the matrix with the remaining elements filled with zero values. A triangular matrix with values only above the main diagonal is called an upper triangular matrix. Whereas, a triangular matrix with values only below the main diagonal is called a lower triangular matrix.

# triangular matrices
from numpy import array
from numpy import tril
from numpy import triu
# define square matrix
M = array([[1, 2, 3],[1, 2, 3],[1, 2, 3]])
print(M)
# lower triangular matrix
lower = tril(M)
print(lower)
# upper triangular matrix
upper = triu(M)
print(upper)
[[1 2 3]
[1 2 3]
[1 2 3]]

[[1 0 0]
[1 2 0]
[1 2 3]]

[[1 2 3]
[0 2 3]
[0 0 3]]

Diagonal Matrix

A diagonal matrix is one where values outside of the main diagonal have a zero value, where the main diagonal is taken from the top left of the matrix to the bottom right. A diagonal matrix is often denoted with the variable D and may be represented as a full matrix or as a vector of values on the main diagonal.

Diagonal matrices consist mostly of zeros and have non-zero entries only along the main diagonal.

# diagonal matrix
from numpy import array
from numpy import diag
# define square matrix
M = array([[1, 2, 3],[1, 2, 3],[1, 2, 3]])
print(M)
# extract diagonal vector
d = diag(M)
print(d)
# create diagonal matrix from vector
D = diag(d)
print(D)
[[1 2 3]
[1 2 3]
[1 2 3]]

[1 2 3]

[[1 0 0]
[0 2 0]
[0 0 3]]

Identity Matrix

An identity matrix is a square matrix that does not change a vector when multiplied. The values of an identity matrix are known. All of the scalar values along the main diagonal (top-left to bottom-right) have the value one, while all other values are zero.

# identity matrix
from numpy import identity
I = identity(3) // or I=eye(3)
print(I)
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 0. 1.]]

ones

Return a new array of given shape and type, filled with ones.

x=(2,3)

np.ones(x)

[[1. 1. 1.]

 [1. 1. 1.]]

ones_like

Return an array of ones with the same shape and type as a given array.

x=np.array([[1,2],[3,4]])

print(np.ones_like(x))

[[1 1] 

 [1 1]]

zeros_like

Return an array of ones with the same shape and type as a given array.

x=np.array([[1,2],[3,4]])

print(np.zeros_like(x))

[[0 0] 

 [0 0]]

Orthogonal Matrix

Two vectors are orthogonal when their dot product equals zero. The length of each vector is 1 then the vectors are called orthonormal because they are both orthogonal and normalized.

# orthogonal matrix
from numpy import array
from numpy.linalg import inv
# define orthogonal matrix
Q = array([[1, 0],[0, -1]])
print(Q)
# inverse equivalence
V = inv(Q)
print(Q.T)
print(V)
# identity equivalence
I = Q.dot(Q.T)
print(I)
[[ 1 0]
[ 0 -1]]
[[ 1 0]
[ 0 -1]]
[[ 1. 0.]
[-0. -1.]]
[[1 0]
[0 1]]

Matrix Operations

Matrix operations are used in the description of many machine learning algorithms. Some operations can be used directly to solve key equations, whereas others provide useful shorthand or foundation in the description and the use of more complex matrix operations. In this you will discover important linear algebra matrix operations used in the description of machine learning methods.

Transpose

A defined matrix can be transposed, which creates a new matrix with the number of columns and rows flipped.The rows of A become the columns of A transpose

# transpose matrix
from numpy import array
# define matrix
A = array([[1, 2],[3, 4],[5, 6]])
print(A)
# calculate transpose
C = A.T # or np.transpose(A)
print(C)
o/p:
[[1 2]
[3 4]
[5 6]]

[[1 3 5]
[2 4 6]]

Inverse

Matrix inversion is a process that finds another matrix that when multiplied with the matrix,results in an identity matrix.Not all matrices are invertible. A square matrix that is not invertible is referred to as singular.Matrix inversion is used as an operation in solving systems of equations framed as matrix equations where we are interested in finding vectors of unknowns.

A good example is in finding the vector of coefficient values in linear regression.

# invert matrix
from numpy import array
from numpy.linalg import inv
# define matrix
A = array([[1.0, 2.0],[3.0, 4.0]])
print(A)
# invert matrix
B = inv(A)
print(B)
# multiply A and B
I = A.dot(B)
print(I)

Trace

A trace of a square matrix is the sum of the values on the main diagonal of the matrix (top-left to bottom-right).

# matrix trace
from numpy import array
from numpy import trace
# define matrix
A = array([[1, 2, 3],[4, 5, 6],[7, 8, 9]])
print(A)
# calculate trace
B = trace(A)
print(B)
o/p:
[[1 2 3]
[4 5 6]
[7 8 9]]

15

Determinant

The determinant of a square matrix is a scalar representation of the volume of the matrix.The determinant describes the relative geometry of the vectors that make up the rows of the matrix. More specifically, the determinant of a matrix A tells you the volume of a box with sides given by rows of A.

The determinant of a square matrix is calculated from the elements of the matrix. More technically, the determinant is the product of all the eigenvalues of the matrix.
The determinant of a square matrix is a single number.It tells immediately whether the matrix is invertible. The determinant is a zero when the matrix has no inverse.
The intuition for the determinant is that it describes the way a matrix will scale another matrix when they are multiplied together.
For example, a determinant of 1 preserves the space of the other matrix
# matrix determinant
from numpy import array
from numpy.linalg import det
# define matrix
A = array([[1, 2, 3],[4, 5, 6],[7, 8, 9]])
print(A)
# calculate determinant
B = det(A)
print(B)
o/p:
[[1 2 3]
[4 5 6]
[7 8 9]]

-9.51619735393e-16

Rank

The rank of a matrix is the estimate of the number of linearly independent rows or columns in a matrix.

An intuition for rank is to consider it the number of dimensions spanned by all of the vectors within a matrix. For example, a rank of 0 suggest all vectors span a point, a rank of 1 suggests all vectors span a line, a rank of 2 suggests all vectors span a two-dimensional plane. The rank is estimated numerically, often using a matrix decomposition method. 

A common approach is to use the Singular-Value Decomposition or SVD for short. NumPy provides the matrix rank() function for calculating the rank of an array. It uses the SVD method to estimate the rank. The example below demonstrates calculating the rank of a matrix with scalar values and another vector with all zero values.

# vector rank
from numpy import array
from numpy.linalg import matrix_rank
# rank
v1 = array([1,2,3])
print(v1)
vr1 = matrix_rank(v1)
print(vr1)
# zero rank
v2 = array([0,0,0,0,0])
print(v2)
vr2 = matrix_rank(v2)
print(vr2)
o/p:
[1 2 3]
1
[0 0 0 0 0]
0
# matrix rank
from numpy import array
from numpy.linalg import matrix_rank
# rank 0
M0 = array([[0,0],[0,0]])
print(M0)
mr0 = matrix_rank(M0)
print(mr0)
# rank 1
M1 = array([[1,2],[1,2]])
print(M1)
mr1 = matrix_rank(M1)
print(mr1)
# rank 2
M2 = array([[1,2],[3,4]])
print(M2)
mr2 = matrix_rank(M2)
print(mr2)
o/p:
[[0 0]
[0 0]]
0

[[1 2]
[1 2]]
1

[[1 2]
[3 4]]
2

Sparse Matrices

Matrices that contain mostly zero values are called sparse, distinct from matrices where most of the values are non-zero, called dense. Large sparse matrices are common in general and especially in applied machine learning, such as in data that contains counts, data encodings that map categories to counts, and even in whole sub fields of machine learning such as natural language processing. It is computationally expensive to represent and work with sparse matrices as though they are dense, and much improvement in performance can be achieved by using representations and operations that specifically handle the matrix sparsity.

A matrix is sparse if many of its coefficients are zero. The interest in sparsity arises because its exploitation can lead to enormous computational savings and because many large matrix problems that occur in practice are sparse.

Sparse Matrices in Machine Learning

Sparse matrices turn up a lot in applied machine learning. In this section, we will look at some common examples to motivate you to be aware of the issues of sparsity.

Data

Sparse matrices come up in some specific types of data, most notably observations that record the occurrence or count of an activity. Three examples include:

Whether or not a user has watched a movie in a movie catalog.
Whether or not a user has purchased a product in a product catalog.
Count of the number of listens of a song in a song catalog.

Data Preparation

Sparse matrices come up in encoding schemes used in the preparation of data. Three common examples include:

One hot encoding, used to represent categorical data as sparse binary vectors.
Count encoding, used to represent the frequency of words in a vocabulary for a document
TF-IDF encoding, used to represent normalized word frequency scores in a vocabulary.

Areas of Study

Some areas of study within machine learning must develop specialized methods to address sparsity directly as the input data is almost always sparse. Three examples include:

Natural language processing for working with documents of text.
Recommender systems for working with product usage within a catalog.
Computer vision when working with images that contain lots of black pixels.

If there are 100,000 words in the language model, then the feature vector has length 100,000, but for a short email message almost all the features will have count zero.

Working with Sparse Matrices

The solution to representing and working with sparse matrices is to use an alternate data structure to represent the sparse data. The zero values can be ignored and only the data or non-zero values in the sparse matrix need to be stored or acted upon. There are multiple data structures that can be used to efficiently construct a sparse matrix; three common examples are listed below.

Dictionary of Keys: A dictionary is used where a row and column index is mapped to a value.
List of Lists: Each row of the matrix is stored as a list, with each sublist containing the column index and the value.
Coordinate List:A list of tuples is stored with each tuple containing the row index,column index, and the value.

There are also data structures that are more suitable for performing efficient operations; two commonly used examples are listed below.

Compressed Sparse Row: The sparse matrix is represented using three one-dimensional arrays for the non-zero values, the extents of the rows, and the column indexes.

Compressed Sparse Column:The same as the Compressed Sparse Row method except the column indices are compressed and read first before the row indices.

The Compressed Sparse Row, also called CSR for short, is often used to represent sparse matrices in machine learning given the efficient access and matrix multiplication that it supports.

Sparse Matrices in Python

SciPy provides tools for creating sparse matrices using multiple data structures, as well as tools for converting a dense matrix to a sparse matrix. Many linear algebra NumPy and SciPy functions that operate on NumPy arrays can transparently operate on SciPy sparse arrays. Further, machine learning libraries that use NumPy data structures can also operate transparently on SciPy sparse arrays, such as scikit-learn for general machine learning and Keras for deep learning.

A dense matrix stored in a NumPy array can be converted into a sparse matrix using the CSR representation by calling the csr matrix() function. In the example below, we defi ne a 3x6 sparse matrix as a dense array (e.g. an ndarray), convert it to a CSR sparse representation,and then convert it back to a dense array by calling the todense() function.

# sparse matrix
from numpy import array
from scipy.sparse import csr_matrix
# create dense matrix
A = array([[1, 0, 0, 1, 0, 0],[0, 0, 2, 0, 0, 1],[0, 0, 0, 2, 0, 0]])
print(A)
# convert to sparse matrix (CSR method)
S = csr_matrix(A)
print(S)
# reconstruct dense matrix
B = S.todense()
print(B
o/p:
[[1 0 0 1 0 0]
[0 0 2 0 0 1]
[0 0 0 2 0 0]]
(0, 0) 1
(0, 3) 1
(1, 2) 2
(1, 5) 1
(2, 3) 2
[[1 0 0 1 0 0]
[0 0 2 0 0 1]
[0 0 0 2 0 0]]

The example below demonstrates how to calculate the sparsity of an array.

# sparsity calculation
from numpy import array
from numpy import count_nonzero
# create dense matrix
A = array([[1, 0, 0, 1, 0, 0],[0, 0, 2, 0, 0, 1],[0, 0, 0, 2, 0, 0]])
print(A)
# calculate sparsity
sparsity = 1.0 - count_nonzero(A) / A.size
print(sparsity)
o/p:
[[1 0 0 1 0 0]
[0 0 2 0 0 1]
[0 0 0 2 0 0]]

0.7222222222222222

Solving system of linear equations
let 2x1+3x2 +5x3= 10
3x1-2x2+x3=3
x1+5x2+7x3=8
the matrix representation is
Ax=b
where
A=[[ 2 , 3, 5],
[ 3, -2 ,1],
[ 1, 5 , 7 ]])
b=[10,3,8]
The following is the python code to solve the problem
import numpy as np
A=np.array([[ 2 , 3, 5],
[ 3, -2 ,1],
[ 1, 5 , 7 ]])

b=np.array([10,3,8])
x=np.linalg.solve(A,b)
print(x)

Write a python program to create two numpy arrays of random integers between 0 and 20 of shape (3, 3) and perform matrix addition, multiplication and transpose of the product matrix(University question)

import numpy as np

A=np.random.randint(0,21,size=(3,3))

B=np.random.randint(0,21,size=(3,3))

print(A)

print(B)

print("Sum")

print(A+B)

print("Product")

C=A.dot(B)

print(C)

print("Transpose of product")

print(C.T)