Applying Custom Functions to Groupby Objects in Pandas

Sean Turner
Aug 13, 2017 · 2 min read

With the end of week three, I am already a quarter of the way through the General Assembly Data Science Immersive bootcamp. This week, the cohort again covered a combination of statistics (t-tests, chi-squared tests of independence, Cohen’s d, and more), as well as more pandas and SQL.

Image for post
Image for post

Over this past week, I encountered a tricky problem. A dataframe had a column named order_id, which contained repeated values (see left). The objective was to create a sub_id column, which indexed the line(s) within each order_id. I solved this problem by looping if-else statements, which checked that the order_id from the previous row matched the current row order_id. This worked, but was time-consuming to write.

Image for post
Image for post

Alternatively, the problem set solution involved a custom function, and was beautiful. The custom function is applied to a dataframe grouped by order_id. The function splits the grouped dataframe up by order_id. Working order_id group at a time, the function creates an array of sequential whole numbers from zero to the number of rows in each order_id, adds one to each element in the array, and finally fills the sub_id column with the array. This process is repeated for each order_id.

Image for post
Image for post

This approach is known as split-apply-combine. While effectively utilising groupby objects has been tricky, it’s something I intend to work on. At the very least, I have the foundation to use what I learned from this solution in future problems. Additionally, should working the custom function solution be too time consuming, I can always fall back on my original solution.

The important thing as always is to be able to solve the problem one way or another.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store