Processing Large Numeric Arrays in Python — Part II
A naive implementation in Python can be done as follows:
The execution takes more than 5 minutes and it times out.
Naive Computation with numpy
The evaluation reports:
3 minutes are better, but still too much.
Improved Naive Computation
The main problem is that we are reading and writing the data element by element. xarray, numpy and pandas use system calls to external libraries in order to access to the internal C-array, and these calls are slow.
One possibility is to extract the whole series to a python list, calculate the moving average and write the series back. Let’s check:
The computation takes about 2 minutes, still slow:
Another approach is work with slices and boolean masks:
Unfortunately the execution time is still too large:
Using Slices with numpy
Slice operations are very fast in numpy. Let us bypass xarray and work with numpy directly:
The computation is very fast, about 3 seconds:
With cython we can improve speed, but coding is more tricky. A possible implementation is:
Here details are essential (try to see what happens by removing line 17). We need to convert and compile code:
The execution takes half a second:
Our preferred implementation uses numba. Numba allows us to compile python code with JIT at runtime. It adds some limitations on data types(numpy array and primitive types) and operations which can be used:
The performance is very good and the code is very clean:
- numpy with slices is fast enough for most cases;
- work directly with numpy and bypass xarray and pandas;
- you can compile your code with cython or numba to work directly with the underlying C-arrays in RAM;
- cython is not python and it is more tricky than numba. Numba requires additional time (often small) for JIT compilation.
Did you learn someting new? Please feel free to leave a comment in the Quantiacs Forum!