Why Functions ?

1. Python Functions processing data and performing task as per the requirement in software development.
2. Once we write a function that can be reused as per the requirement. That’s why functions are also called as reusable code. Due to the reusability code redundancy is avoiding which is the main advantage. That means we can avoid writing same code again and again.
3. Python Functions representing modular programming approach. A task is dividing into sub tasks where each sub tasks called as a module. To represent each module we define separate function. Then the functions are called from main program to perform the complete task. This is known as modular programming and it makes programming easy.
4. In software development, code maintenance is becoming easy due to functions. For example, when a new feature is adding in an existing application, then the new function definition can integrate into the application. If a point is not required then we can delete the corresponding function.
5. At the time of application development, if any error arises then the corresponding function can update without affecting other functions. Hence code debugging is becoming easy.

There are four different modes to open a file:

• “r” — Read — Opens a file for reading, error if the file does not exist
• “w” — Write — Opens a file for writing, creates the file if it does not exist
• “a” — Append — Opens a file for appending, creates the file if it does not exist
• “x” — Create — Creates the specified file, returns an error if the file exists
• “t” — Text — Default value. Text mode
• “b” — Binary — Binary mode (e.g. images)

#Open a file and perform read operation
f1 = open("e:\demo.txt", "r")
print(f1.read())

Python is open source and freeware.

Python is platform-independent.

Python is robust, because python provides a mechanism that is exception handling mechanism which is used to handle exception.

Python is multi-threaded: that means multiple task executing simultaneously where each task is a separate independent thread.

Python is dynamically typed language, because in python we are not specifying basic data type like int, float, str etc. to define a variable. Here at the time of execution, python control understand the type based on value assigned to variable.

Python takes care of memory allocation and de-allocation on it’s own, that means the programmer need not have to allocate and de-allocate memory before constructing variables and objects. Over all we can say python handles garbage collection.

Python is considered as beginners language, because without any knowledge of any programming language, you can happily learn python.

Python supporting huge libraries that’s why python is considered as technology.

Python is easy to integrate with other languages like C, C++, JAVA etc.

Python is extensible.

• Functional programming approach from C.
• OOP(Object-Oriented Program) concept from C++.
• Scripting concepts from Shell Script and Perl.
• Modular programming approach from modula-3.
• Syntax concept from C and ABC language.

Python can specifically used to develop for the following applications.

• Desktop Application
• Web Application
• Database Application
• Networking Application
• Game Application
• IOT Application
• Data Science Application
• Machine Learning Application
• AI Application

Features :

• Python is simple & easy to learn.
• Python is freeware & opensource.
• Python is Interpreted.
• Python is dynamically typed.
• Python is extensible.
• Python is embedded.
• Python has extensive library.

1. Ease of Learning: Python is designed with a clear and readable syntax that resembles plain English, making it easier to learn and understand, especially for beginners.

2. Versatility: Python is a versatile language used in various domains, including web development, data analysis, scientific computing, machine learning, artificial intelligence, automation, and more.

3. Large Community and Resources: Python has a vast and active community of developers, which means there are ample learning resources, tutorials, forums, and libraries available to assist you in your learning journey.

4. Readability and Maintainability: Python’s clean and consistent syntax encourages well-organized and readable code. This makes collaboration with other developers and maintaining code over time more manageable.

5. Rapid Development: Python’s concise syntax and extensive standard library enable faster development cycles. This is particularly beneficial for prototyping and creating applications in less time.

6. Cross-Platform Compatibility: Python runs on various operating systems, including Windows, macOS, and Linux. This cross-platform compatibility ensures that your code can be executed on different environments without major modifications.

7. Diverse Applications: Python is used in a wide range of applications, from building websites (using frameworks like Django and Flask) to performing complex data analysis (with libraries like Pandas and NumPy) and creating machine learning models (using libraries like Scikit-Learn and TensorFlow).

8. Automation and Scripting: Python is often used for automating repetitive tasks, writing scripts, and performing system administration tasks due to its ease of use and availability on most platforms.

9. Data Analysis and Visualization: With libraries like Pandas and Matplotlib, Python is a go-to language for data scientists and analysts to manipulate and visualize data effectively.

10. Machine Learning and AI: Python has gained significant popularity in the field of machine learning and artificial intelligence due to its powerful libraries like TensorFlow, Keras, and PyTorch, which simplify the creation of machine learning models.

11. Job Opportunities: Python’s widespread use and demand across various industries result in a plethora of job opportunities for Python developers, data scientists, and machine learning engineers.

12. Community Support: As one of the most popular programming languages, Python enjoys a strong community of developers who are often willing to help and share their knowledge through forums, tutorials, and open-source projects.

Overall, learning Python can open doors to various career paths and personal projects, whether you’re interested in web development, data analysis, machine learning, or simply want to add a valuable skill to your repertoire. Its versatility, readability, and widespread adoption make it an excellent language to invest your time and effort in learning.

1. Smoothing:

It is a process that is used to remove noise from the dataset using some algorithms It allows for highlighting important features present in the dataset. It helps in predicting the patterns. When collecting data, it can be manipulated to eliminate or reduce any variance or any other noise form. The concept behind data smoothing is that it will be able to identify simple changes to help predict different trends and patterns. This serves as a help to analysts or traders who need to look at a lot of data which can often be difficult to digest for finding patterns that they wouldn’t see otherwise.

2. Aggregation:

Data collection or aggregation is the method of storing and presenting data in a summary format. The data may be obtained from multiple data sources to integrate these data sources into a data analysis description. This is a crucial step since the accuracy of data analysis insights is highly dependent on the quantity and quality of the data used. Gathering accurate data of high quality and a large enough quantity is necessary to produce relevant results. The collection of data is useful for everything from decisions concerning financing or business strategy of the product, pricing, operations, and marketing strategies.

For example, Sales, data may be aggregated to compute monthly& annual total amounts.

3. Discretization:

It is a process of transforming continuous data into set of small intervals. Most Data Mining activities in the real world require continuous attributes. Yet many of the existing data mining frameworks are unable to handle these attributes.
Also, even if a data mining task can manage a continuous attribute, it can significantly improve its efficiency by replacing a constant quality attribute with its discrete values.

For example, (1–10, 11–20) (age:- young, middle age, senior).

4. Attribute Construction:

Where new attributes are created & applied to assist the mining process from the given set of attributes. This simplifies the original data & makes the mining more efficient.

5. Generalization:

It converts low-level data attributes to high-level data attributes using concept hierarchy. For Example Age initially in Numerical form (22, 25) is converted into categorical value (young, old).

For example, Categorical attributes, such as house addresses, may be generalized to higher-level definitions, such as town or country.

6. Normalization:

Data normalization involves converting all data.

1. Type I error

2. Type II error

Type I Error:

It is defined as the incorrect or false rejection of the Null hypothesis. For this error, the maximum probability is alpha, and it is set in advance. The error is not affected by the sample size of the dataset. The type I error increases as we increase the number of tests or endpoints.

Type II error

Type II error is defined as the wrong acceptance of the Null hypothesis. The probability of type II error is beta, and the beta depends upon the sample size and value of alpha. The beta cannot be determined as the function of the true population effect. The value of beta is inversely proportional to the sample size, and it means beta decreases as the sample size increases.

The value of beta also decreases when we increase the number of tests or endpoints.

Importance of P-value

The importance of p-value can be understood in two aspects:

• Statistics Aspect: In statistics, the concept of the p-value is important for hypothesis testing and statistical methods such as Regression.

• Data Science Aspect: In data science also, it is one of the important aspect Here the smaller p-value shows that there is an association between the predictor and response. It is advised while working with the machine learning problem in data science, the p-value should be taken carefully.

In Statistical hypothesis testing, the P-value or sometimes called probability value, is used to observe the test results or more extreme results by assuming that the null hypothesis (H0) is true. In data science, there are lots of concepts that are borrowed from different disciplines, and the p-value is one of them. The concept of p-value comes from statistics and widely used in machine learning and data science.

• P-value is also used as an alternative to determine the point of rejection in order to provide the smallest significance level at which the null hypothesis is least or rejected.
• It is expressed as the level of significance that lies between 0 and 1, and if there is smaller p-value, then there would be strong evidence to reject the null hypothesis. If the value of p-value is very small, then it means the observed output is feasible but doesn’t lie under the null hypothesis conditions (H 0 ).
• The p-value of 0.05 is known as the level of significance (α). Usually, it is considered using two suggestions, which are given below:
• If p-value>0.05: The large p-value shows that the null hypothesis needs to be accepted.
• If p-value<0.05: The small p-value shows that the null hypothesis needs to be rejected, and the result is declared as statically significant.

In Statistics, our main goal is to determine the statistical significance of our result, and this statistical significance is made on below three concepts:

• Hypothesis Testing
• Normal Distribution
• Statistical Significance

Let’s understand each of them.

Hypothesis Testing

Hypothesis testing can be defined between two terms; Null hypothesis and Alternative Hypothesis. It is used to check the validity of the null hypothesis or claim made using the sample data. Here, the null hypothesis (H0) is defined as the hypothesis with no statistical significance between two variables, while an alternative hypothesis is defined as the hypothesis with a statistical significance between the two variables. No significant relationship between the two variables tells that one variable will not affect the other variable. Thus, the Null hypothesis tells that what you are going to prove doesn’t actually happen. If the independent variable doesn’t affect the dependent variable, then it shows the alternative hypothesis condition.

In a simple way, we can say that in hypothesis testing, first, we make a claim that is assumed as a null hypothesis using the sample data. If this claim is found invalid, then the alternative hypothesis is selected This assumption or claim is validated using the p-value to see if it is statistically significant or not using the evidence. If the evidence supports the alternative hypothesis, then the null hypothesis is rejected.

Steps for Hypothesis testing

Below are the steps to perform an experiment for hypothesis testing:

1. Claim or state a Null hypothesis for the experiment.

2. State the alternative hypothesis, which is opposite to the null hypothesis.

3. Set the value of alpha to be used in the experiment.

4. Determine the z-score using the normal distribution.

5. Compare the P-value to validate the statistical significance.

• For the 2 prediction classes of classifiers, the matrix is of 2*2 table, for 3 classes, it is 3*3 table, and so on.
• The matrix is divided into two dimensions, that are predicted values and actual values along with the total number of predictions.
• Predicted values are those values, which are predicted by the model, and actual values are the true values for the given observations.
• It looks like the below table:

The above table has the following cases:

• True Negative: Model has given prediction No, and the real or actual value was also No.
• True Positive: The model has predicted yes, and the actual value was also true.
• False Negative: The model has predicted no, but the actual value was Yes, it is also called as Type-II error.
• False Positive: The model has predicted Yes, but the actual value was No. It is also called a Type-I error.

Need for Confusion Matrix in Machine learning

• It evaluates the performance of the classification models, when they make predictions on test data, and tells how good our classification model is.
• It not only tells the error made by the classifiers but also the type of errors such as it is either type-I or type-II error.
• With the help of the confusion matrix, we can calculate the different parameters for the model, such as accuracy, precision, etc.

Example: We can understand the confusion matrix using an example.

Suppose we are trying to create a model that can predict the result for the disease that is either a person has that disease or not. So, the confusion matrix for this is given as:

From the above example, we can conclude that:

• The table is given for the two-class classifier, which has two predictions “Yes” and “NO.” Here, Yes defines that patient has the disease, and No defines that patient does not has that disease.
• The classifier has made a total of 100 predictions. Out of 100 predictions, 89 are true predictions, and 11 are incorrect predictions.
• The model has given prediction “yes” for 32 times, and “No” for 68 times. Whereas the actual “Yes” was 27, and actual “No” was 73 times.

What is AUC-ROC Curve?

AUC-ROC curve is a performance measurement metric of a classification model at different threshold values. Firstly, let’s understand ROC (Receiver Operating Characteristic curve) curve.

ROC Curve

ROC or Receiver Operating Characteristic curve represents a probability graph to show the performance of a classification model at different threshold levels. The curve is plotted between two parameters, which are:

• True Positive Rate or TPR
• False Positive Rate or FPR

In the curve, TPR is plotted on Y-axis, whereas FPR is on the X-axis.

TPR:

TPR or True Positive rate is a synonym for Recall, which can be calculated as:

FPR or False Positive Rate can be calculated as:

Here, TP: True Positive
FP: False Positive
TN: True Negative
FN: False Negative

Now, to efficiently calculate the values at any threshold level, we need a method, which is AUC.

AUC: Area Under the ROC curve

AUC is known for Area Under the ROC curve. As its name suggests, AUC calculates the two-dimensional area under the entire ROC curve ranging from (0,0) to (1,1), as shown below image:

In the ROC curve, AUC computes the performance of the binary classifier across different thresholds and provides an aggregate measure. The value of AUC ranges from 0 to 1, which means an excellent model will have AUC near 1, and hence it will show a good measure of Separability.