CORRELATION

CO Means Together while Relation Means Connection

Correlation analysis determines the relationship between two quantities known as variables, ‘x’ and ‘y’. Correlation also describes the strength of association between variables.

You use correlation when you want to study two variables directly or indirectly

. A correlation could be positive, meaning both variables move in the same direction,

· Negative, meaning that when one variable’s value increases, the other variables’ values decrease.

· Correlation can also be neural or zero, meaning that the variables are unrelated.

For Example; Staffs who are paid less and others higher as salary, we use correlation to determine staffs who are paid lesser and staffs who are paid higher and the regression predicts the salary of a new staff according to department.

SCATTER DIAGRAM

A scatter diagram is also known as a correlation chart, scatter graph or a scatter plot. This diagram is drawn with two variables, usually the first variable is independent X-axis and the second variable is dependent Y- axis on the first variable.

You are analyzing the pattern of highest car washed at laundry. You select the two variables: car washed and numbers of clean cars, and draw the diagram.

Once the diagram is completed, you observe that as the washed cars increases, the number of clean cars also goes up. This shows that there is a relationship between the washed cars and clean cars at the laundry.

TYPES OF SCATTER DIAGRAM

1. Scatter Diagram with No Correlation
2. Scatter Diagram with Moderate Correlation
3. Scatter Diagram with Strong Correlation

Scatter Diagram with No Correlation is also known as “Scatter Diagram with Zero Degree of Correlation”

This type of scatter diagram, data points are spread so randomly that no line can be drawn through them

Scatter Diagram with Moderate Correlation is also known as “Scatter Diagram with Low Degree of Correlation”.

If you watch closely the data points are little closer together and you can tell that some kind of relation exists between these two variables.

Scatter Diagram with Strong Correlation is also known as “Scatter Diagram with High Degree of Correlation”.

For this you will see that the variables are closely related to each other.

Take note you can also divide scatter diagram according to the slope, or trend, of the data points below:

• Scatter Diagram with Strong Positive Correlation
• Scatter Diagram with Weak Positive Correlation
• Scatter Diagram with Strong Negative Correlation
• Scatter Diagram with Weak Negative Correlation
• Scatter Diagram with Weakest (or no) Correlation

Why Correlation?

The use of Correlation coefficient, such as the Pearson Product Moment Correlation Coefficient, to test if there is a linear relationship between the variables. To quantify the strength of the relationship, we can calculate the correlation coefficient (r). The numerical value of the person correlation coefficient (r) ranges from +1 to -1.

r>0 indicates a positive linear relationship between two variables

r<0 indicates a negative linear relationship

r=0 indicates that no linear relationship exists.

REGRESSION

Regression is the function relation between variables to make future projections on events. Regression analysis also implies that the outcome depends on one or more other variables.

The latter are referred to as x, predictor, prognostic, explanatory or independent variables. The horizontal axis of a graph is used for plotting the independent variables. The outcome (y) may be called the dependent variable, criterion or response. It is plotted on the vertical axis.

Regression analysis is all about mathematical measure of an average relationship between x and y in terms of original units of data. The average mathematical relationship is utilized to help estimate the expected value of the dependent variable based on known value of the predictor. Which makes regression analysis a reliable tool for forecasting events. E.g. Fish Pie demand can be estimated using past records.

USE FOR CORRELATION AND REGRESSION

Correlation and regression are two analysis based on the distribution of multiple variables. They can be used to describe the nature and strength of the relationship between the nature and strength of the relationship between two continuous quantitative variables.

The whole essence of regression is to help and predict the value of the random variable based on the value of a fixed variables while the essence of correlation coefficient measures whether one random variable changes with another.

Three primary uses for Correlation and Regression:

1. To test hypothesis about cause and effect relationships

2. To test for association between two variables

3. Use of Linear regression helps estimating the value of the dependent variable corresponding to a particular value of the independent variable.

Summary

Correlation and regression are very interesting and can be used to analyze relationships. They are also commonly used techniques for investigating the relationship between two quantitative variables.

Project of Gitgirl Academy.

In Python Lambda is an Anonymous Function that is defined with out a NAME. Anonymous is refers to a function declared with no name which means Anonymous Functions are called Lambda Functions. Now syntactically they appear differently, lambda functions behave in the same way as normal functions that are declared using the def Keyword.

The Characteristics of Python Lambda Functions below are;

• Lambda codes is simple and clear
• A Lambda Function can take any number of arguments, but can only contain a single expression. Expressions are representations of value i.e number, a string.
• Lambda Function can be used to return function objects.

How to Use Lambda Functions in Python

Based on The Project Given in the Gitgirl Academy (Course Data Science) to Discover Insights Records of United State Domestic Flights that took place from January 1st — 15th of 2015. The data set contained 201664 rows and 14 columns.

Before we get started download python 3.7 — www.python.org and a code editor called anaconda — www.anaconda.com, with this you are good to start coding by either choosing your choice of Applications. For this project i am using Jupiter Note Book.

Let’s Proceed To Action

Import Libraries

Firstly Import Python Libraries by using Pandas and Numpy and the dataset into Jupiter note book script as shown below:

Import Dataset

Importing your dataset allows you to understand your data and what should be done.

Clean Dataset

Our Dataset is rough and its advised to clean our data to avoid error. To clean this data i am going to use the ‘fillna()’ which will help replace missing values with (np.nan)

Sort Value in Descending Order

Sort_value in descending order to identify the highest value to the lowest value. How value was identified is from the arr_delay this is due to the time frame (number of minutes)of flight delayed.

Using Lambda Functions to find how often delay occur from January 1st-15th?

For us to get delays that occurred during January 1st-15th use the arr_delay which represent the number of minutes of a given flight_delayed, below you will observe in the script i have lambda function (lambda f: f >0)this is to check if the value of arr_delay is greater than 0 (Zero) then returns “True” or “False”.

Use Value Count to Find Aggregate Trends and the Percentage of Flight_Delays?

The Value_Count() function returns object containing counts of unique values of flight delays and non flight delays. In this case if a flight is delayed it returns “True”, if flight not delayed it returns “False”. This gives us a specific value on aggregate trends of flight_delay number(98627)

Getting the percentage of flight delayed i had to use the float() because it defines a variable with a fractional value (decimal point). Dividing data flight delayed by total sum flight giving us our total percentage 49.

Numbers of Flights were Delayed Longer than 20 minutes?

To get the number of flight delayed longer than 20 mins, use the lambda function (lambda f: f >20) this checks the total delay greater than 20 mins. As shown below the number of flight result is 48679

Finally Lambda Function has given us a good result or insight that enables us understand every activities that happened in the United State Domestic Flight from January 1st — 15th, 2015.

Lambda Functions

What does Lambda actually mean? “Functions used as Data” remember data is everything. Lambda is known for its symbol Uppercase Λ / lowercase λ, this is the 11th letter of the Greek Alphabet used to represent the “l” sound in Ancient and Modern Greek. In Greek numerical is the system value is of 30. The symbol representation for wavelength.

In computer science world the most important defining characteristic of a lambda expression is that it is used as data. This means the function is passed as an argument to another function returned as a value for a function or assigned to variables or data structure. Lambda function or expression provides a way to generate small anonymous function. Anonymous function are functions without a name. These functions are described as throw-away functions. Lambda function that are seen in programming language are; Python, Perls, Java-script, Scheme, PHP, Swift, Ruby etc. The Lambda features was added to python due to the demand from Lisp programmer (Lisp is a family of computer programming language).

Now lets take a look at the general syntax of a Lambda Function

lambda arguments: expression

Example of Lambda Function

In the above program, lambda x: x * 2 is the lambda function. Here x is the argument and x * 2 is the expression that gets evaluated and returned.

This function has no name. It returns a function object which is assigned to the identifier double. We can now call it as a normal function. The statement

double = lambda x: x * 2

Why Use Lambda Functions?

Lambda functions are used when you need a function for a short period of time. This is commonly used when you want to pass a function as an argument to higher-order functions, that is, functions that take other functions as their arguments.

The use of anonymous function inside another function is explained in the following example:

def testfunc(num):  
    return lambda x : x * num

In the above example, we have a function that takes one argument, and the argument is to be multiplied with a number that is unknown. Let us demonstrate how to use the above function:

def testfunc(num):  
    return lambda x : x * num

result1 = testfunc(10)

print(result1(9))

Output

90

In the above script, we use a lambda function to multiply the number we pass by 10. The same function can be used to multiply the number by 1000:

def testfunc(num):  
  return lambda x : x * num

result2 = testfunc(1000)

print(result2(9))

Output

9000

It is possible for us to use the testfunc() function to define the above two lambda functions within a single program:

def testfunc(num):  
    return lambda x : x * num

result1 = testfunc(10)  
result2 = testfunc(1000)

print(result1(9))  
print(result2(9))

Output

90  
9000

Lambda functions can be used together with Python's built-in functions like map(), filter() etc.

Use of Lambda Function in python

In Python, we generally use it as an argument to a higher-order function (a function that takes in other functions as arguments). Lambda functions are used along with built-in functions like filter(), map(), reduce()etc.

The filter() function in Python is also another built-in function that actually filters any iterable object (sequence).The function is called with all the items in the list and a new list is returned which contains items for which the function evaluates to True.

The map() function in Python takes in a function and a list as argument. The function is called with a lambda function all the items in the list and a new list is returned which contains items returned by that function for each item.

The reduce() function in Python takes in a function and a list as argument. The function is called with a lambda function and a list and a new reduced result is returned. This performs a repetitive operation over the pairs of the list.

Parallel Processing

Definition of Parallel Processing

Parallel processing also called parallel computing this is a type of computation in which many execution of processes are carried out concurrently.

In computers, parallel processing is the processing of program instructions by dividing them among multiple processor with the objective of running a program in less time. In the earliest computers, only one program ran at a time. According to research Parallel computing is closely related to concurrent computing — they are frequently used together, and often conflated, though the two are distinct: it is possible to have parallelism without concurrency (such as bit-level parallelism), and concurrency without parallelism (such as multitasking by time-sharing on a single-core CPU). Parallel processing makes a program run faster because there are more engines (CPUs) running it. In practice, it is often difficult to divide a program in such a way that separate CPUs can execute different portions without interfering with each other.

Advantages Parallel Processing

• Parallel computing saves time, allowing the execution of applications in a shorter wall-clock time.
• Solve Larger Problems in a short point of time.
• Compared to serial computing, parallel computing is much better suited for modeling, simulating and understanding complex, real-world phenomena.
• Many problems are so large that it is impossible to solve them on a single computer, especially given limited computer memory.
• You can do many things simultaneously by using multiple computing resources.
• Can using computer resources on the Wide Area Network(WAN) or even on the internet.
• It can help keep you organized. If you have Internet, then communication and social networking can be made easier.
• It has massive data storage and quick data computations.

Disadvantages Parallel Processing

• Programming to target Parallel architecture is a bit difficult but with proper understanding and practice, you are good to go.
• The use of parallel computing lets you solve computationally and data-intensive problems using multi-core processors, but, sometimes this effect on some of our control algorithm and does not give good results and this can also affect the convergence of the system due to the parallel option.
• The extra cost (i.e. increased execution time) incurred are due to data transfers, synchronization, communication, thread creation/destruction, etc. These costs can sometimes be quite large, and may actually exceed the gains due to parallelization.
• Various code tweaking has to be performed for different target architectures for improved performance.
• Better cooling technologies are required in case of clusters.
• Power consumption is huge by the multi-core architectures
• Parallel solutions are harder to implement, they’re harder to debug or prove correct, and they often perform worse than their serial counterparts due to communication and coordination overhead

Multiprocessing refers to a computer system’s ability to support more than one process(program) at the same time. Multiprocessing operating system enable several programs to run concurrently . UNIX is one of the most widely used multiprocessing system, but there are many others, including OS/2 for high-end PCs. Multiprocessing systems are much more complicated than single-process systems because the operating system must allocate resource to competing processes in a reasonable manner.

May also refer to the utilization of multiple CPUs in a single computer system.

The maximum number of processes you can run at a time is limited by the number of processors in your computer. If you don’t know how many processors are present in the machine, the cpu_count() function in multiprocessing will show it.

import multiprocessing as mp
print("Number of processors: ", mp.cpu_count())

Difference Between Synchronous and Asynchronous execution

In parallel processing, there are two types of execution: Synchronous and Asynchronous.

1. A synchronous execution is one the processes are completed in the same order in which it was started. This is achieved by locking the main program until the respective processes are finished.
2. Asynchronous, doesn’t involve locking. As a result, the order of results can get mixed up but usually gets done quicker.

There are 2 main objects in multiprocessing to implement parallel execution of a function: The Pool Class and the Process Class.

1. Pool Class
2. Synchronous execution

• Pool.map() and Pool.starmap()
• Pool.apply()

1. Asynchronous execution

• Pool.map_async() and Pool.starmap_async()
• Pool.apply_async())

2. Process Class

Additional Resource

Parallel Processing — https://www.machinelearningplus.com/python/parallel-processing-python/

https://searchdatacenter.techtarget.com/definition/parallel-processing

Learn more on Parallel processing in python — https://stackabuse.com/parallel-processing-in-python/

Conclusion

Lambda (or anonymous) functions are a tool that is gradually getting more popular in Python programs. Reason why it is very important that you can understand what it means.Take note there is nothing that the lambda syntax allows you to do that it wouldn’t be possible to do without them. With lambda expressions, the goal is to minimize as many of the drawbacks to using a single class or anonymous inner class when implementing a functional interface, while at the same time maximizing the benefits. Data parallel programming uses automatic paralleling compilers which enables loop-level parallelization. Generally, this approach often will not yield high efficiency. The main reason for this is that a large portion of the existing code is in most cases inherently sequential.

This project is part of Git girl school were i currently attend.

Thank you Git girl.