The most interesting things about the new release — pandas 2.1 was released on August 30th 2023. Let’s take a look at the things this release introduces and how it will help us improving our pandas workloads. It includes a bunch of improvements and also a set of new deprecations. pandas 2.1 builds heavily on the PyArrow integration that…

Leverage cloud-hosted VMs to speed up your training — You can use Coiled Run and Coiled Functions for easily running scripts and functions on a VM in the cloud. In this post we’ll use Coiled Functions to train a RandomForestClassifer on a cloud-hosted machine that has enough cores to speed up our training process. The model parallelizes very well…

Explaining how Copy-on-Write optimizes performance — Introduction The first post explained how the Copy-on-Write mechanism works. It highlights some areas where copies are introduced into the workflow. This post will focus on optimizations that ensure that this won’t slow the average workflow down. We utilize a technique that pandas internals use to avoid copying the whole DataFrame…

Explaining how Copy-on-Write works internally — Introduction pandas 2.0 was released in early April and brought many improvements to the new Copy-on-Write (CoW) mode. The feature is expected to become the default in pandas 3.0, which is scheduled for April 2024 at the moment. There are no plans for a legacy or non-CoW mode. This series of…

Scale out your workload to a big VM in the cloud. — We recently pushed out two new and experimental features Coiled Run and Coiled Functions for easily running scripts and functions in the cloud. In this post we’ll use Coiled Functions to process the 150 GB Uber-Lyft dataset on a single machine with DuckDB. Getting started We start with creating the SQL queries…

Adding a logical query planer to Dask DataFrame. Introduction Dask DataFrame doesn’t currently optimize your code for you (like Spark or a SQL database would). This means that users waste a lot of computation. Let’s look at a common example which looks ok at first glance, but is actually pretty inefficient. …

Seamlessly transition to a cloud-hosted GPU — We recently pushed out two new and experimental features: Coiled Run and Coiled Functions, a deviation of coiled run. We are excited about both of them because they: Allow users to scale up any given program on any hardware in the cloud. Make GPUs easily accessible without going through the…

Explaining the pandas data model and its advantages — Introduction pandas enables you to choose between different types of arrays to represent the data of your dataframe. Historically, most dataframes are backed by NumPy arrays. pandas 2.0 introduced the option to use PyArrow arrays as a storage format. Additionally, an intermediate layer exists between these arrays and your dataframe, Block…

Authors: Hendrik Makait and Patrick Hoefler — Getting notified of a significant performance regression the day before release sucks, but quickly identifying and resolving it feels great! We were getting set up at our booth at JupyterCon 2023 when we received a notification: An engineer on our team had spotted a significant performance regression in Dask. With…