# A Beginners Guide to Pandas

## Your first step to get started with data analysis.

Since 2010, when pandas first became open-sourced, it has matured quite into a beautiful and extensive library for data analysis. It is often used in conjunction with computational and statistical libraries like NumPy, scikit-learn, matplotlib etc.

In this article, I’ll walk you through the most common functions in pandas so you’re ready to do some exploratory data analysis.

pandas has 2 data structures that are repeatedly used over and over: Series and DataFrame

# Series

Simply put, a series is a 1-d array of elements, but with an added feature: there’s an explicit index to address each element. To create a series, you need to pass an array (a list or a dictionary or even a tuple). You can also pass an explicit array for the index. When indexing a series, think of it as a hashmap, associative array or dictionary data structure.

>>> import pandas as pd

>>> series = pd.Series([1,2,3,4,5,6,7])

>>> series

0 1

1 2

2 3

3 4

4 5

5 6

6 7

dtype: int64>>> series.index

RangeIndex(start=0, stop=7, step=1)>>> #or pass an explicit index

>>> series2 = pd.Series([1,2,3,4,5,6,7], index['a','b','c','d','e','f','g'])

>>> series2

a 1

b 2

c 3

d 4

e 5

f 6

g 7

dtype: int64

# DataFrame

pandas is designed for working with tabular or heterogeneous data. This is achieved through the DataFrame data structure. It is like a table — it contains rows and columns which can be indexed. There are numerous ways to create a dataset, the most common of which would be to pass a dictionary. Let’s create one.

`>>> diction = {'name': ['Alex', 'Bob', 'Charlie', 'Jack', 'Melissa'],`

'age': [20, 18, 19, 20, 19],

'score': [109, 108, 99, 120, 115]}

>>> dataframe1 = pd.DataFrame(diction)

To illustrate the true power of pandas library, we will now start working with a small dataset from kaggle. This is a dataset of students’ performance on the exams and how it is affected by economic, personal and social factors. Download the dataset from here.

## Importing a dataset

To import a dataset in csv format using pandas, we use pandas read_csv function.

`>>> df = pd.read_csv('StudentsPerformance.csv')`

## Examining the data

Often when we’re working with a dataset, we are interested in knowing what columns it contains and what each column describes. This is easy with pandas head() and info() functions.

`>>> df.head()`

`>>> df.info()`

<class 'pandas.core.frame.DataFrame'>

RangeIndex: 1000 entries, 0 to 999

Data columns (total 8 columns):

gender 1000 non-null object

race/ethnicity 1000 non-null object

parental level of education 1000 non-null object

lunch 1000 non-null object

test preparation course 1000 non-null object

math score 1000 non-null int64

reading score 1000 non-null int64

writing score 1000 non-null int64

dtypes: int64(3), object(5)

memory usage: 62.6+ KB

## Checking for Null values

Datasets often contain null values which can become a nuisance to work with during ML. We can check for null values using the info() function or sum over each individual column to check for nulls through the columns.

`>>> df.isnull().sum()`

gender 0

race/ethnicity 0

parental level of education 0

lunch 0

test preparation course 0

math score 0

reading score 0

writing score 0

dtype: int64

## Describing the data

Another very useful function is describe(). This gives a 5 number summary of the data along with some other useful stats for interpreting the range in which our data would lie.

`df.describe()`

## Asking questions from data

The main reason for exploring a dataset is to be able to answer questions. For e.g. one would want to know by looking at the dataset, “How many students passed the Math exam?” Or, “How many students scored a percentage above 80?” Answering these question is easy with pandas.

>>> # how many students passed the math exam?

>>> passing_score = 40

>>> math_stats = df['math score'] >= passing_score

>>> print(math_stats.sum())

960>>> # how many students scored a percentage above 80?

>>> total_marks = df['math score'] + df['reading score'] + df['writing score']

>>> total = 3

>>> total_percentage = (total_marks/total)

>>> above_80 = total_percentage >= 80

>>> print(total_percentage[above_80].count())

198

## Sorting data by percentages

Now that we’ve calculated the percentages for all the students, we may wish to sort the data according to their rank. pandas lets you sort according to your desired column. For this, let’s add another column names ‘percentage’ in our dataset and sort this.

`>>> df['percentage'] = total_percentage`

>>> df.sort_values(by='percentage', ascending=False)

## Counting by values

Sometimes we may be interested in analyzing a specific column from our dataset. For instance, the column ‘parental level of education’ contains 5 unique values throughout the column. We may wish to count these values. pandas value_counts() function makes this very simple.

>>> df['parental level of education'].unique()array(["bachelor's degree", 'some college', "master's degree",

"associate's degree", 'high school', 'some high school'],

dtype=object)>>> df['parental level of education'].value_counts()some college 226

associate's degree 222

high school 196

some high school 179

bachelor's degree 118

master's degree 59

Name: parental level of education, dtype: int64

# Wrapping up

Here I’ve discussed just a few handy functions that are repeatedly used during exploratory data analysis. For a complete guide to using pandas, the best source is the documentation itself!