Python and Numerical Computing not ❤
Few years ago I started programming in Python and I was hooked. The syntax was beautiful, installing it was easy and third party packages amazing. The only problem? Python is not easily parallelizable and slow. Really slow. If you compare it to see C it is often times 1000x slower. Here is a graph for you:
You can’t train your Mutliple Billion Parameter Megatron Model when you are this slow.
This has some consequences. Every Numeric Computing (NC) library you use in Python is a very thin wrapper around C, Fortran or C++. For example:
- Numpy (51% C)
- Tensorflow (60% C++)
- PyTorch (51% C)
This makes it very difficult to deep dive into your program. At some point you face a wall, where your normal Python tools like your debugger simply do not work anymore. I call this the C-Wall and to be honest it sucks! It‘s great for executing all the high-level stuff really fast but if you want to know how Numpy calculates your Eigenvalues you are stuck. You can install C and Numpy from source and then debug it but really who wants to do that? (If I can’t pip install something I’m out!)
Could you imagine a language that is fast enough with Python’s syntax? Where every layer in the NC stack is implemented in this language? Where we all (Data Science and ML people) could live in a one-language-world once in for all!
I think this language could be SWIFT. Like me, you probably though SWIFT was this language that you would only use if you programmed iOS apps. But this turns out to be totally wrong. You can use it for basically anything. It was created by this compiler legend called Chris Lattner. The first time I heard about Swift for NC was in the Fast AI course ML from the fundamentals by Jeremy Howard (Great course and great guy!). I highly recommend his blog.
Swift for Tensorflow is one of Chris Lattner’s projects at Google the first big project, that I know of, that is trying this out. They are implementing Tensorflow from scratch in Swift. This could be amazing!
I still love Python and I think it will always have it’s use-cases. I think these use-cases will not include NC though. It simply wasn’t build for that. I think a new programming language will emerge (maybe Swift), that will be fast enough and easy to learn (just like Python) that could be used for the whole NC stack.
Let’s tear down the C-Wall!
