Polars Takes on Pandas: Accelerating Python’s Data Analysis Horizons with Breakneck-Speed Processing Power

Published in

DataScience with Python — NishKoder

6 min readMar 30, 2023

Introduction:

Pandas has long been a popular library for data manipulation and analysis in Python. However, as the size of datasets increases, the need for faster and more memory-efficient solutions becomes more apparent. Polars is a relatively new library that aims to provide high-performance data manipulation capabilities with a familiar API. In this article, we will explore why Polars might be an excellent choice for your data manipulation tasks in Python and show you a series of code examples that demonstrate its capabilities.

Polars:

An Overview Polars is an open-source DataFrame library designed for efficient data manipulation in Python. Built on top of the Arrow and Rust ecosystems, it achieves high-performance and parallelism through its use of the Arrow memory format and Rust’s powerful performance characteristics. Some key features of Polars include:

High-performance: Polars is designed to be fast, thanks to its use of the Arrow memory format and Rust’s powerful performance characteristics.
Memory-efficient: Polars reduces memory usage by applying lazy evaluation and other optimization techniques.
Familiar API: Polars offers a similar API to Pandas, making it easier for existing Pandas users to transition.

Installing Polars

To get started with Polars, you can install it via pip:

pip install polars

Code Examples and Explanations

In this section, we will explore a series of code examples comparing Polars and Pandas, covering various data manipulation tasks. Each example will be followed by an explanation.

a) Importing the necessary libraries:

import pandas as pd
import polars as pl

This code snippet imports both Pandas and Polars libraries, allowing us to compare the syntax and functionality of the two libraries side by side.

b) Creating DataFrames:

# Pandas DataFrame
df_pandas = pd.DataFrame({
    'A': [1, 2, 3, 4],
    'B': [5, 6, 7, 8],
    'C': [9, 10, 11, 12]
})

# Polars DataFrame
df_polars = pl.DataFrame({
    'A': [1, 2, 3, 4],
    'B': [5, 6, 7, 8],
    'C': [9, 10, 11, 12]
})

This example demonstrates how to create DataFrames in both Pandas and Polars. As you can see, the syntax for creating a DataFrame is quite similar, which makes it easy for Pandas users to switch to Polars.

c) Filtering DataFrames:

# Pandas
filtered_pandas = df_pandas[df_pandas['A'] > 2]

# Polars
filtered_polars = df_polars.filter(pl.col('A') > 2)

In this example, we filter the DataFrames to select only the rows where the ‘A’ column’s value is greater than 2. The Polars syntax is quite similar to Pandas but uses the filter() function instead of direct indexing.

d) GroupBy and Aggregations:

# Pandas
grouped_pandas = df_pandas.groupby('A').agg({'B': 'sum', 'C': 'mean'})

# Polars
grouped_polars = df_polars.groupby('A').agg([pl.col('B').sum(), pl.col('C').mean()])

Here, we perform a GroupBy operation on the ‘A’ column and aggregate the ‘B’ and ‘C’ columns using the sum and mean functions, respectively. The syntax for Polars is similar to Pandas, with the main difference being that Polars uses a list of aggregation functions rather than a dictionary.

e) Adding a new column:

# Pandas
df_pandas['D'] = df_pandas['A'] * 2

# Polars
df_polars = df_polars.with_column(pl.col('A') * 2).alias('D')

In this example, we add a new column ‘D’ to both DataFrames, where the values are twice the values in the ‘A’ column. Polars uses the with_column() function to create a new column based on a calculation.

f) Renaming columns:

# Pandas
df_pandas = df_pandas.rename(columns={'A': 'X', 'B': 'Y'})

# Polars
df_polars = df_polars.rename({'A': 'X', 'B': 'Y'})

Both Pandas and Polars have a rename() function to change the names of DataFrame columns. The only difference is that Pandas uses a dictionary for renaming, while Polars uses a key-value pair.

g) Sorting by column:

# Pandas
sorted_pandas = df_pandas.sort_values(by='A', ascending=False)

# Polars
sorted_polars = df_polars.sort(pl.col('A').desc())

Sorting DataFrames by column values is another common task. In this example, we sort both DataFrames by the values in the ‘A’ column in descending order. The Polars syntax is similar to Pandas but uses the sort() function and the desc() method to specify the sorting order.

h) Selecting columns:

# Pandas
selected_pandas = df_pandas[['A', 'B']]

# Polars
selected_polars = df_polars.select(['A', 'B'])

Selecting specific columns from a DataFrame is a fundamental operation. In this example, we select columns ‘A’ and ‘B’ from both DataFrames. Polars uses the select() function instead of direct indexing like Pandas.

i) Applying custom functions to columns:

# Pandas
def double(x):
    return x * 2

df_pandas['A'] = df_pandas['A'].apply(double)

# Polars
def double_polars(array):
    return array * 2

df_polars = df_polars.with_column(pl.col('A').apply(double_polars).alias('A'))

In this example, we apply a custom function to the ‘A’ column in both DataFrames. The function doubles the values in the ‘A’ column. The Polars syntax uses the with_column() function along with apply() to apply the custom function.

j) Merging two DataFrames:

# Pandas
df1_pandas = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2_pandas = pd.DataFrame({'A': [1, 2], 'C': [5, 6]})

merged_pandas = pd.merge(df1_pandas, df2_pandas, on='A')

# Polars
df1_polars = pl.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2_polars = pl.DataFrame({'A': [1, 2], 'C': [5, 6]})

merged_polars = df1_polars.join(df2_polars, on='A', how='inner')

Merging DataFrames is a common operation when combining data from different sources. In this example, we merge two DataFrames based on the values in the ‘A’ column. The Polars syntax uses the join() function with the how parameter set to 'inner' for an inner join, which is equivalent to Pandas' merge() function.

k) Pivot tables:

# Pandas
import numpy as np

data = {'A': ['foo', 'bar', 'baz', 'foo', 'bar', 'baz'],
        'B': ['one', 'one', 'one', 'two', 'two', 'two'],
        'C': [1, 2, 3, 4, 5, 6]}
df_pandas = pd.DataFrame(data)

pivot_pandas = df_pandas.pivot_table(index='A', columns='B', values='C', aggfunc=np.sum)

# Polars
data = {'A': ['foo', 'bar', 'baz', 'foo', 'bar', 'baz'],
        'B': ['one', 'one', 'one', 'two', 'two', 'two'],
        'C': [1, 2, 3, 4, 5, 6]}
df_polars = pl.DataFrame(data)

pivot_polars = df_polars.groupby(['A', 'B']).agg(pl.col('C').sum()).pivot('A', 'B', 'C_sum')

Creating pivot tables is a common operation when summarizing data. In this example, we create a pivot table for both DataFrames using the ‘A’ column as the index, ‘B’ column as the columns, and the sum of the ‘C’ column as the values. The Polars syntax is slightly different, using a combination of groupby(), agg(), and pivot() functions.

l) Concatenating DataFrames:

# Pandas
df1_pandas = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2_pandas = pd.DataFrame({'A': [5, 6], 'B': [7, 8]})

concatenated_pandas = pd.concat([df1_pandas, df2_pandas], ignore_index=True)

# Polars
df1_polars = pl.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2_polars = pl.DataFrame({'A': [5, 6], 'B': [7, 8]})

concatenated_polars = df1_polars.vstack(df2_polars)

Concatenating DataFrames is useful when you need to append rows from one DataFrame to another. In this example, we concatenate two DataFrames in Pandas using the concat() function and in Polars using the vstack() function.

m) Filling missing values:

# Pandas
data = {'A': [1, 2, np.nan], 'B': [4, np.nan, np.nan], 'C': [7, 8, 9]}
df_pandas = pd.DataFrame(data)

filled_pandas = df_pandas.fillna(0)

# Polars
data = {'A': [1, 2, None], 'B': [4, None, None], 'C': [7, 8, 9]}
df_polars = pl.DataFrame(data)

filled_polars = df_polars.fill_nan(0)

Filling missing values is an essential operation when cleaning and preprocessing data. In this example, we fill missing values in both DataFrames with zeros. The Polars syntax is quite similar to Pandas, using the fill_nan() function.

n) Multi-indexing:

# Pandas
data = {'A': ['foo', 'bar', 'baz', 'foo', 'bar', 'baz'],
        'B': ['one', 'one', 'one', 'two', 'two', 'two'],
        'C': [1, 2, 3, 4, 5, 6]}
df_pandas = pd.DataFrame(data)

multiindexed_pandas = df_pandas.set_index(['A', 'B'])

# Polars
# Polars does not have a direct equivalent to Pandas multi-indexing.
# However, you can use groupby() and other operations to achieve similar results.

This example demonstrates multi-indexing in Pandas, where we create a multi-level index using columns ‘A’ and ‘B’. Polars does not have a direct equivalent to Pandas multi-indexing, but you can use groupby() and other operations to achieve similar results.

o) Resampling time series data:

# Pandas
import pandas as pd

data = {'Date': pd.date_range('2020-01-01', '2020-01-10', freq='D'),
        'Value': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}
df_pandas = pd.DataFrame(data)
df_pandas.set_index('Date', inplace=True)

resampled_pandas = df_pandas.resample('3D').mean()

# Polars
import polars as pl
from polars import functions as pf

data = {'Date': pd.date_range('2020-01-01', '2020-01-10', freq='D'),
        'Value': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}
df_polars = pl.DataFrame(data)

# Define a custom function for resampling.
def resample_polars(df, date_col, value_col, rule):
    df = df.with_column((pf.col(date_col) / rule).cast(pl.Int32).alias('interval'))
    return df.groupby('interval').agg([pf.col(value_col).mean()])

resampled_polars = resample_polars(df_polars, 'Date', 'Value', '3D')

Resampling is a common operation for time series data. In this example, we resample both DataFrames using a 3-day interval and calculate the mean value for each interval. Pandas uses the resample() function, while Polars requires a custom resampling function using groupby() and aggregation functions.

Conclusion:

In conclusion, Polars is a powerful and efficient alternative to Pandas for data manipulation in Python, especially when working with large datasets. Its high-performance capabilities, memory efficiency, and familiar API make it an attractive choice for developers looking to boost their data processing speed without sacrificing usability. While Polars may not encompass every feature available in Pandas, it shines in situations where performance and scalability are crucial. Embrace Polars and elevate your data manipulation game to new heights, ensuring a smoother and more efficient data analysis experience.