AppEngine on MacOS is a CPU Hog: Solve This Problem with Another Python Native Extension Module

Ivo Bellin Salarin
Apr 10, 2018 · 4 min read
Image for post
Image for post

Hey, have you read this gorgeous article from Filippo Valsorda about how to write a Python extension in Go? You definitely should.

When I first read it, I was searching for a solution to a problem affecting all the MacBooks in our open space. You have this very same problem if you develop AppEngine applications on MacOS X. You should notice it when you launch dev_appserver.py and your CPU cooler starts going crazy. At this point, you can feel a warm breeze flowing out of your laptop and your SSD starts to age faster than it should.

Analysis of the problem

But what is the problem, exactly? Well, it resides in the mtime_file_watcher.py part of the Google Cloud SDK. This is the default file watcher for MacOS X. The file watcher is the component that watches your project source code and restarts a module to reload the modified code. On Linux, the default file watcher is the inotify_file_watcher.py so you won’t experience that warm breeze unless you don’t specify--use_mtime_file_watcher=yes as a dev_appserver.py option. If you are desperate, you can also disable the watcher entirely --automatic_restart=no but soon you will be even more desperate. Another way to mitigate this phenomenon is to fine tune the watched paths with --watcher_ignore_re=".*/app/client/.*" but honestly, when you have more than one module to run this will not produce the effects you would expect.

When I first started wandering the net searching for a solution, I found some alternatives. The most impressive article was from zeekay. In its implementation, he was using another fsevents module in Python. I have tried to use it, but it segfaulted on my MacOS. It’s C code, and I didn’t feel like rebuilding and and debugging it. I have to admit that I am more interested in Go than ol’ school C (best TIOBE language of 2017 and my favorite in 2002–2004, though).

Enter Filippo. His article sufficiently sparked my curiosity and was complete enough to get my extension running. He covers the structure of a Python extension written in Go, with the minimal C infrastructure to expose it to the Python interpreter. He shows how to parse two integers and how to return them to the caller.

My aim was to parse a list of paths and call a Python callback when an event was raised on one of the given paths. Returning the modified file and the action flags (Created, Modified, Deleted and so on). I wanted to do that in a parallel way (ie. call the callback immediately, once the event is raised. goroutines are great for that.)

What you can learn from my experience

  • Go has a great development ecosystem. The most significative part of the go tooling comes from the compiler itself, but you will certainly find an extension for your favorite editor too.

Once the POC was complete, there was one more obstacle to avoid: packaging. That means getting rid of “it works on my machine!” and making it a *product*. Enter the Python scripts to backup, replace (and eventually restore) the mtime file watcher that comes with AppEngine.

Result

On my machine, the main Python process started by the dev_appserver.py was consuming 156% CPU, constantly. In early January, I was about to ask my manager for a new MacBook.
Today, the code in the https://github.com/nilleb/fsevents-watcher repository allows Python to consume only 0.3% CPU. And, of course, the AppEngine application modules are being reloaded when needed.

What’s next

I would really like to write some unit tests, and understand a little more about what makes the Python interpreter delivered with the OS binary incompatible with the home brew one (even if they share the same major.minor version numbers). Once this is done, I will then perhaps build a pip package and publish it on pypi.org.

Caveats

If you update the Google Cloud components, you have to re-replace mtime_file_watcher.py with the scripts provided with the module.

Special thanks

This project was a fruit of the first LumApps Polish Week, held from January 22 to January 26. Special thanks to Lou for having had this brilliant idea. What’s a polish week? That’s a week during which a part of LumApps takes a break from delivery deadlines, in order to improve the collective health. Everyone is free to suggest a problem and volunteer to solve it (eventually along with other problems).

LumApps Experts

The LumApps team shares their work processes and expertise…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store