Speeding up Python and NumPy: C++ing the Way

Matt Fowler
Mar 20, 2016 · 4 min read
np.std(lst)
Python: 0.06 seconds
NumPy: 0.39 seconds

NumPy and Array Size

The key comes in the data set this algorithm used. It had to calculate the standard deviation of very small chunks of data at a time, roughly 25 elements.

Speeding it up with C++

The algorithm was built on calculating the standard deviation of small arrays in a loop. There wasn’t a way to refactor to work with larger arrays at a time. That coupled with the fact that this calculation could happen several million times meant seemingly no way to optimize.

Conclusion

I eventually ported over this entire algorithm to C++. Given its highly iterative nature, overall performance sped up over 100 times.

Source Code

Source of the C extension and plots is available at:

Coding With Clarity

Topics in Software Engineering and Development

Matt Fowler

Written by

Boston based full stack polyglot software architect specializing in JVM languages, iOS and agile development.

Coding With Clarity

Topics in Software Engineering and Development