Dask

In my previous post I showed how to submit tasks to the scheduler in a loop. The future of that submit is appended to a queue. Below is the code of my second script. Basically it waits for oncoming futures in a queue. When a new future is received (i.e…

In some of my projects I actively use Dask in Python and today I would like to highlight some useful approaches to handle parallelism. Dask is a flexible library for parallel computing and data manipulation in Python. It works amazing parallelizing computation of huge dataframes however today I will be…

In this paper we will discuss pandas, a Python library of rich data structures and tools for working with structured data sets common to statistics, finance, social sciences, and many other fields. The library provides integrated, intuitive routines for performing common data manipulations and analysis on such data sets. It…

The challenge The aim of this experiment is to compare the time it takes to extract, transform and load large datasets using three different methods. The source data for this experiment is from the publicly available Stack Exchange Data Dump, specifically the Users and Posts sources. The Posts dataset…

Hello everyone today we will be covering two main topics. Parallel processing or parallel computing and Dask. Let’s get started. What is parallel computing ? Parallel processing means to executing multiple tasks at the same time, using multiple processors in the same machine. We mostly face two type of flow in our daily codes. First…

Most of the time software engineers deal with relatively small data. Read some records from a database and display them to the user here, perform some simple calculations there, maybe process a small file every now and again. …

A deep look at the assumptions of the Pandas interface — Written by Kevin Kho and Han Wang This is a written version of our most recent PyCon talk. Pandas-like Frameworks for Distributed Computing Over the last year and a half, we’ve talked to data practitioners who want to move Pandas code to either Dask or Spark to take advantage of distributed computing resources. Their workloads…

3 Reasons Why Dask is Better Suited for Data Science Projects (And When it is Not) — Introduction: Big Data in Data Science As a data scientist, you might encounter projects where your data doesn’t fit into memory. In these cases, popular tools such as pandas and NumPy cannot support your operations, even after optimization. Before going to Big Data solutions such as PySpark, there are 3 question you should ask yourself: