Python Pandas:- Part 1
(Advice:- It’s always good to know at least basics of NumPy/ Matplotlib before jumping to Pandas)
Key Contents:-
- 3 W’s of Pandas (What/ Why/ Where)
- Installing Pandas
- Data Structures (Data Frame, Series)- Basic
- Important Questions
3 W’s of Pandas:-
What is Pandas?
Pandas is an open source, BSD-licensed library which provides easy to use data structures, data analysis, data manipulation tools for the python programming language. It’s built on top of two core libraries of Python — matplotlib (used for data visualization) and NumPy (used for mathematical Operations).
Pandas is like excel of Python. Data Frame is analogous to a table and Series is analogous to rows and columns.
Why Pandas?
As Pandas is built on top of other Python libraries, and some part of it is implemented in C, which makes it really smarter and faster while execution . In short you can say it’s “fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive.”(mentioned in Pandas docs). With the help of these features you can save some of your time of computation and invest it in analyzing data and building models.
Where Pandas?
You can use Pandas almost everywhere where you are dealing with data. If you are working on a project where you are going to visualize data, analyze it and perform some operations with it, you will most probably love to use Pandas.
Installation:-
Note:- Officially it is supported by, Python 3.6.1 and above.
There are multiple ways to install and use Pandas. The easiest and most recommended way to install pandas is to install it as part of the Anaconda distribution, a cross platform distribution for data analysis and scientific computing. Another widely used way to install Pandas is to create a virtual environment and install Pandas in it.
Create virtual environment
(Assuming that you have already installed python3, pip and virtual env)
cd my-project #go to project directory
virtualenv -p python3 venv #create virtual env
source venv/bin/activate #Activate Virtal env
pip install pandas #install pandasIt’s advised to use Jupyter Notebook for learning Pandas, as it makes it really easy to visualize the data at each step, making you to debug easily and saves time.
Installing Jupyter Notebook:-
pip install jupyter #install Jupyter notebook
jupyter notebook #launching Jupyter notebook Data Structures:-
The fundamental data structures in pandas are DataFrame and Series. Fundamental behaviour of these data types are to indexing, axis labelling/alignment apply across all of the objects. We need to import NumPy and pandas to get started into namespace:-
Series
It’s a 1-D labelled array capable of holding any data type, ex:- integers, strings, floating point numbers, Python objects, etc. The data can be either a python dictionary, ndarray, a scalar value etc. Axis labels are called indexes.
Syntax :- s = pd.Series(data, index=index)
Instantiation of Series from:-
a) python dictionary
Example 1) When we don’t give index arg to Series method
Notice:- we have not passed the index arg but still we got the index as the key of dictionary.
Example 2) When we pass the index arg with limited keys
Notice:- here we have passed index arg to Series method. There are various occasions when we are working on large datasets and our dictionary will have many unwanted information but we can use the above method to get the series of only needed information.
Example 3) When we pass index keys which are not part of dictionary
Notice:- Here we can observe that “b” is one of the index key passed, but the dictionary doesn’t have that key. Hence, it returns NaN for “b” key
Note:- For, Python version ≥ 3.6 and Pandas version ≥ 0.23, Series index will be ordered by the dictionary’s insertion order but for Python version ≥ 3.6 and Pandas version ≥ 0.23, Series index will be the lexically ordered list of dictionary keys.
b) ndarray
Example 1) When we don’t give index arg to Series method
Notice:- In case of ndarray, if no index is passed, it will be created having values [0, … , len(data-1)]
Example 2) When we pass index arg to series
Example 3) When the index list has less no of keys then the length of ndarray
Notice:- It will give you an error because, here we have taken the authority of pandas to assign index and passed index of length not equal to length of data list
c) Scalar Value
In case of Scalar Value, an index must be provided and value gets repeated in order to match index’s length.
We can treat Series like a dictionary or ndarray.
*np.random.randn(4) creates a numpy.ndarray of length 4
Series like a ndarray:-
Series like a dictionary:-
While fetching key that doesn’t exist in Series will give us “KeyError”
One of the key difference between ndarray and Series is that operations between Series automatically align the data based on the labels. Hence, we don’t need to consider whether the Series involved have the same labels or not. If the operation between unaligned series takes place, then the union of the indexes involved is the result. If any label doesn’t match in other series, then the value of that index will be set as NaN. We can also drop labels with missing data via the dropna function.
Name attribute:-
Series can have a name attribute.
We can also rename the series using pandas.Series.rename() method.
DataFrame
It’s a 2-D labeled data structure with columns of different types, like a spreadsheet or SQL table, or a dictionary of series object. It accepts different kinds of inputs like Dictionary of 1-D ndarrays, lists, dicts or Series, 2 -D numpy.ndarray,, Structured or record ndarray, another DataFrame. It’s optional to pass index(row labels) and columns(column labels) args. A dictionary of Series plus a specific index will discard all data not matching up to the passed index.
Instantiation of DataFrame from:-
a) Dictionary of Series or Dictionary
If indexes are passed, then union of indexes of series will be the resulting index else ordered list of dictionary keys. If there are nested dictionaries, then these will be converted to Series first. Row and Columns labels can be accessed by accessing the index and columns attributes respectively. All the ndarrays involved in DataFrame instantiation should be of same length and if the index list is passed as an arg then it must be of same length as ndarrays. If no index is passed, then the resulting indexes value will be in [0, …, len(data) — 1].
Notice:- Here, we have created dictionary of Series and passed it as the only arg for DataFrame instantiation. So, the indexes are the union of keys of dictionary of Series and columns are the key of each Series or dictionary keys. The value of index “e” in first_series column is NaN as we don’t have that key for first_series key in dict.
Now we will pass the explicit indexes and columns:-
Getting Index and Columns of a DataFrame
b) From structured or record array
c) From list of dictionaries
Column Selection, Addition and Deletion
DataFrame can be treated like a dictionary of like-indexed Series object. It uses the same syntax as analogous to dictionary operations for getting, setting and deleting.
i) Column Selection
ii) Column Addition
iii) Column Deletion
Column deletion can be achieved by using either pop() or del
Notice:- While Deleting column using pop(), it returned us the deleted column details
Questions:-
- Is Numpy included in Pandas?
- What is faster Numpy or Pandas?
- Should I learn Numpy or Pandas?
- Why Pandas is so fast?
- What is Numpy used for?
References:-
Thanks for reading!. Hope, I helped you and served the purpose. Learning is a every moment process and I would like to learn from you as well. If you think any improvement is required or if I missed anything or for any query, you can also reach out to me on my mail id:- iamayushshekhar@gmail.com. If you wish, we can also connect on linkedin.
