Processing Large Numeric Arrays in Python — Part II

In this second article Dima explains how he worked with numpy, pandas, xarray, cython and numba to optimally implement operations on large numeric arrays on the Quantiacs platform.

In the previous article we have shown how to optimally load the data needed to compute an exponential moving average of the prices. Here we will show different approaches for the computation.

Naive Computation

A naive implementation in Python can be done as follows:

The execution takes more than 5 minutes and it times out.

Naive Computation with numpy

As xarray is based on numpy, and we based our toolbox on xarray, we can try a naive implementation using numpy.

The evaluation reports:

3 minutes are better, but still too much.

Improved Naive Computation

The main problem is that we are reading and writing the data element by element. xarray, numpy and pandas use system calls to external libraries in order to access to the internal C-array, and these calls are slow.

One possibility is to extract the whole series to a python list, calculate the moving average and write the series back. Let’s check:

The computation takes about 2 minutes, still slow:

Using Slices

Another approach is work with slices and boolean masks:

Unfortunately the execution time is still too large:

Using Slices with numpy

Slice operations are very fast in numpy. Let us bypass xarray and work with numpy directly:

The computation is very fast, about 3 seconds:

Using cython

With cython we can improve speed, but coding is more tricky. A possible implementation is:

Here details are essential (try to see what happens by removing line 17). We need to convert and compile code:

and finally:

The execution takes half a second:

Using numba

Our preferred implementation uses numba. Numba allows us to compile python code with JIT at runtime. It adds some limitations on data types(numpy array and primitive types) and operations which can be used:

The performance is very good and the code is very clean:

Conclusions

• numpy with slices is fast enough for most cases;
• work directly with numpy and bypass xarray and pandas;
• you can compile your code with cython or numba to work directly with the underlying C-arrays in RAM;
• cython is not python and it is more tricky than numba. Numba requires additional time (often small) for JIT compilation.

Did you learn someting new? Please feel free to leave a comment in the Quantiacs Forum!