A Practical Guide to Pandas

Jordan Wells
Aug 17, 2020 · 6 min read

What is Pandas?

Pandas is an open-source data analysis and manipulation tool for Python.

The name? It comes from the econometrics term “panel data”, which is multi-dimensional data with measurements over time. It’s also pretty cute so that’s a bonus!

At its core, it allows us to easily use spreadsheet-like data. From there, you can clean the data, preform any additional modifications, and analyse it to gain some insight into your data.

Installation and Importing

If you have Python installed, you likely already have pip. If not, there are easy instructions to install it here. Pip allows us to easily install packages from the Python Package Index from the command line. Pandas can be installed with pip, along with its dependency NumPy.

Whenever you need to use pandas for one of your projects, it can be imported like so:

We import using the abbreviations np and pd to make it easier to call upon the functions and classes in the module. If you want to read more about how the import system works in Python or how to use NumPy, you can check out my blog posts on either subject.

The Basics

Reading files with Pandas

If you’ve got data you want to do something with, you probably don’t just have it memorized. Most likely, you’ve got it in some sort of file. Conveniently, Pandas lets us easily read data from a wide variety of ways it can be stored! This will bring the data into Pandas with one of Panda’s objects, which we will talk about later.

For example, the most common ways are:

As always, all of the code for this blog post can be found my GitHub.

Seeing our data

If you want to get a basic overview of what your data looks like, you can use the head and tail methods. They allow you to see a snippet of the first few and last few (default 5) entries of your data respectively.

For the purposes of this guide, I’ll be using a tool known as Jupyter Notebook so that we can easily see the results of our code. I used a data set from the Food Network show Chopped, courtesy of Jeffrey Braun. All of the data is stored in a CSV file called “chopped.csv.”

The Series

Creating Series

According to the Pandas documentation, a series is “a one-dimensional labeled array capable of holding data of any type (integer, string, float, python objects, etc.).” Importantly, the series is built off of the NumPy ndarray, so all of the functionality from that also carries over into our series.

Essentially, its like having a spreadsheet with two columns: labels, which we will refer to as the index, and values, which could be anything from countries, prices, or MySpace “Top 8” rankings.

Conveniently, there are several ways to create them:

Series are ndarray-like

Series act very similarly to NumPy ndarrays, and as such a lot of similar functionality can be used on a series.

Additionally, because of this property, Pandas Series can also take advantage of “vectorized operations.” Essentially, this means that we don’t have to loop through each element in a Series to preform operations on it. Instead, our operation can be “scaled” to the size of our Series and be done just like a matrix operation. For example, if we wanted to add 3 to each element in our Series, all we have to do is add 3 to the Series itself:

The scalar value 3 got “stretched” into a vector or array full of 3’s that was the same size as our original Series. Consequently, the computation was much easier and faster.

Series are also dictionary-like

Series also act very similarly to a Python dictionary, with the index label acting as a key.

The DataFrame

The DataFrame is a two-dimensional data structure with potentially different types of data in each of its columns. It is essentially a spreadsheet or table, just with a lot of added functionality. It is an incredibly key object in Pandas.

Data Frame Creation

Conveniently, it accepts many different types of data as input:

A DataFrame from several Series:

A DataFrame from a dictionary:

A DataFrame from a 2D array:

DataFrame getting and setting

You can treat a DataFrame exactly like a Python dictionary, with the keys being the names of each column, and the corresponding values being a Series. For example:

If you want to get a row in the DataFrame as a Series, you can use the built-in Pandas loc method. Additionally, you can use the method to get and set values for specific elements in the DataFrame.

If you need to add an entire row of new data to the DataFrame, this can be done with the append method.

However, this can be computationally expensive, so try your best to first gather all your data and then create a DataFrame.

One helpful feature of Pandas is the ability to do boolean indexing. Essentially, this allows us to get only the values which meet a certain condition. For example, if we only want to grab people who are over the age of 25:

There are many applications of boolean indexing, and I encourage you to try out several different ideas. (What if you wanted to grab all the values about the mean age?)

DataFrame cleaning

Often, data is not very clean. There will be typos, missing information, extraneous rows, mismatched types, and just general weirdness. One of the most important jobs of a data scientist is getting a data set into a state that is “clean” and devoid of all of this weirdness. That way, any insights taken from that data isn’t the result of bad data.

This blog post by Malay Agarwal does an amazing job of explaining how to clean data, and goes into much more depth than I could in this single guide.

DataFrame Manipulation

Finally, one of the most important things we can do with a DataFrame is manipulating its data to create new insights.

For example, imagine you work at Amazon and you have sales data from some technology items.

Lets say you wanted to find out how much sales was generated from each item. We could define a new column called “Total Sales” which simply takes in the product of “Unit Price” and “Quantity” for each item. Similarly, let’s say you wanted to find out which items made up the largest fraction of sales. You could define a “Ratio of Sales” which takes in the “Total Sales” for each item and divides it by the sum of “Total Sales” for all items.

This is straightforward to accomplish with DataFrames:

And now you can generate some insight as to which items are the most important for your overall profits! Look at you, a whole Data Scientist!

Conclusion

Pandas is a powerful tool for dealing with data. Whether you want to draw insights from Food Network shows, or drive a small business, Pandas is an integral part of the Python data science ecosystem.

I hope you were able to glean some information from this post! It was an interesting topic to look into! If there is any aspect of this post that you would like me to go into more depth with, please contact me.

Thanks for reading!

  • Jordan

The Startup

Get smarter at building your thing. Join The Startup’s +792K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

Check your inbox
Medium sent you an email at to complete your subscription.

Jordan Wells

Written by

A Chemical Engineering student who is incredibly interested in Data Science

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +792K followers.

Jordan Wells

Written by

A Chemical Engineering student who is incredibly interested in Data Science

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +792K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store