Async/await for wxPython

Python 3 GUI apps with asynchronous functionality

Python 3’s support of async/await is the new hotness, allowing cooperative computation with reduced complexity and without needing threads — as long as you use async compatible libraries like aiohttp etc. You can’t use the famous requests library because it is not async aware and will block, but not to worry, those other libraries have you covered.

The wxPython project is a wonderful, full featured cross-platform GUI toolkit that is completely free to use in commercial projects — unlike some (cough, cough, umm, PyQt). Wouldn’t it be great if you could utilise async/await in wxPython? This would be great for the usual reasons:

• checking a server for information, in the background
• running a long process without locking up the UI
• etc

Well it seems you are out of luck, because wxPython already comes with its own messaging event loop, and you can’t have two event loops — can you?

By two event loops, I mean the Python 3 asyncio event loop and the wxPython main GUI event loop.

Blending event loops

I remember running into this problem a decade ago where I wanted to blend wxPython widgets into PyGame and both had their own event loops. Some interesting initial approaches were unsatisfactory so I ended up writing a meta-event loop that did a bit of wxPython and a bit of PyGame — giving them both a chance to run. I never published the code, but it worked.

Then when listening to an episode of the wonderful podcast Python Bytes, the topic of introducing asyncio to the PyQt GUI toolkit came up — again the solution has been to blend event loops — see asyncqt.

Whilst listening to the above podcast, it struck me as coincidental because I’ve recently been utilising a similar project for wxPython called wxasync which blends the wxPython event loop with Python 3’s asyncio loop. Result? I can now use Python 3’s async/await with wxPython!

wxasync is a library for using Python 3 asyncio (async/await) with wxPython.

A trio of similar solutions!

Benefits to my project — Pynsource

The ability to have asynchronous methods in my Python UML tool Pynsource has been a boon.

Pynsource UML for Python is now using async/await

I recently upgraded the entire open source Pynsource codebase to Python 3. Beside being a prudent upgrade, it has allowed me to be able to parse and reverse engineer Python 3 source code into UML class diagrams. Actually Pynsource can still parse Python 2 because I switched to the typed-ast parser, which only runs under Python 3, and has the advantage of being able to parse either syntax. In fact, Guido himself worked on this module recently.

Now that Pynsource was converted to Python 3, I wanted to take advantage of all that async/await goodness. :-)

Background app version check

The first thing I added was a new background version check function, to notify users when a new version of Pynsource was available. I now check a server for the latest version — in the background, a few seconds after startup — without affecting the UI.

Background http request to render PlantUML

Asynchronous programming also helped enormously with a new feature of Pynsource which is the ability to reverse engineer Python and render a PlantUML class diagram image. This involves calling an internet based rendering service which can take some time to run, depending on the complexity of the UML class diagram you are trying to generate.

My initial implementation blocked whilst waiting for the server response and the user had to wait - the UI was unresponsive. Not a good look.

After importing wxasync and converting my code to use async/await, the PlantUML rendering in Pynsource is now non-blocking. This is a much better user experience.

Pynsource rendering UML but GUI is still responsive

This also allowed me to implement a cancel feature: hittingESC will cancel the render, just in case it is taking too long or has locked up. Wonderful!

I even have a seconds counter in the status bar telling you how much time the internet call is taking. The counter is only possible because of async/await. The resulting PlantUML diagram soon arrives:

Final image retrieved from the server

How to convert your wxPython app to use wxasync

You should be able to convert any existing wxPython application to use wxasync as long as it is written in Python 3 of course, since the async/await technology is only available in Python 3.5 and later.

You should probably learn a little about the async/await approach before attempting this — there is a bit of a learning curve getting your head around this stuff. There are plenty of tutorials on it when you search.

I’ll be bold and try to sum up async/await up in a single paragraph:

Instead of traditionally invoking your Python program to run from top to bottom you must invoke it via a special loop which runs forever or till your program exits. Inside the loop, your code will run from top to bottom as usual but if you call an async function containing an await call to a long running function (e.g. a http request via aiohttp), your code yields to the main loop — allowing the loop to try to run other tasks. Tasks are just another name for these async functions that are in various states of suspension. It’s a cooperative thing — no threads or interrupts. When your code reaches the bottom, it of course stops and the only remaining code that will run in the cooperative loop are any leftover async tasks/functions. In the case of wxPython, you never reach the bottom of your code — the wxPython event loop runs forever, till you explicitly exit.

I’m definitely not an async/await expert, so apologies for any inaccuracies in my summation, above. It’s just meant to be an explanatory perspective that may help some Python programmers grok the idea. Remedial and official doco on Python 3 async/await event loops is here and broader official async/await doco is here.

Step 1

The first thing you do is import wxasync and change your app to subclass from WxAsyncApp instead of the traditional wx.App

from wxasync import AsyncBind, WxAsyncApp, StartCoroutine
class MainApp(WxAsyncApp):
# class MainApp(wx.App):

Then instead of starting your app with

def main():
    application = MainApp(0)
    application.MainLoop()

you start it with

from asyncio.events import get_event_loop
def main_async():
    application = MainApp(0)
    loop = get_event_loop()
    loop.run_until_complete(application.MainLoop())

Step 2

You should now be able to call async functions or methods, and still have the wxPython GUI remain responsive. For example here is my call to trigger the check for the latest version of my program — which happens in during app startup in OnInit():

StartCoroutine(self.check_for_updates, self)

The “coroutine” being called is simply my async function check_for_updates which first sleeps for a while using await asyncio.sleep(15), so that the version check doesn’t happen straight away and annoy the user.

Note that check_for_updates does not use the requests package to retrieve the url, it instead uses a call to my function url_to_data which uses aiohttp. Yeah, the syntax of theurl_to_datafunction is a bit gnarly but it does the job and I don’t have to think about it, now that it is written — it grabs the contents of a url, asynchronously, which means it yields to the wxasync event loop whilst waiting for the server response, allowing both wxPython and Python 3 tasks to have a bit of cpu time.

Notice that I had to explicitly decode the server response text from utf-8 before accessing it as a Python string.

Notice also that I cache the call to the server, so that next time, the server call is avoided for the same rendering job. I had to use a special version of lru_cache namely async_lru since the standard Python lru_cache does not work with async functions.

Rubberband app

Here is a fun wxPython app that uses async/await. You drag the mouse to create a selection rectangle. Note that at some point in the demo, I right click the mouse to start the async timer which displays in the status bar. The timer runs independently and asynchronously whilst I am dragging in the GUI — nice!

Fun asynchronous “rubber band” wxPython app

The full source code to this app is:

Other asynchronous approaches

To be fair, there are a few things you can do in wxPython to “loosen up” the event loop without using async/await.

You can call wx.SafeYield() at any time which allows the main wxPython event loop to “breathe” and process GUI events. However you can soon get into recursion problems if another wx.SafeYield() happens during the wx.SafeYield(). Plus you are not taking advantage of async/await and all the libraries that support it. Still, I use this approach during my diagram layout so that the incremental results of layout animate for the user:

Layout Animation using wx.SafeYield() in Pynsource

You can also use wx.CallAfter which queues a regular function or method to be run next time the wxPython loop gets a chance. That’s cooperative, like async/await is.

Then there are threads. There is a portion of my Pynsource app which does not use async/await during a long task — it uses threading. That is the deep layout algorithm feature for “laying out” UML diagrams more optimally. This works like normal layout but runs for longer, running the entire layout several times, using different parameters, then chooses the “best looking” layout. This takes several seconds to run and “think”.

Of course I wanted to run this deep layout and not lock up the GUI — the only solution five years ago was threading. It got a little complex and you can look at the Pynsource GitHub source code to see how I did it. It was based on the wxPython article on long running threads. Using wxasync is much easier to use than background threads and CallAfter’s. Presumably I could now revisit that code and use async/await instead — or I could just leave it, as it isn’t broken. :-)

Is async/await an anti-pattern?

Are there any downsides to async/await? Surely not.

Yet soon I hit a conceptual brick wall — it startled me and got me researching... It turns out that async/await has been around in other languages for a long time

The feature is found in C# 5.0, Python 3.5, Hack, Dart, Kotlin 1.1, and JavaScript, with some experimental work in extensions, beta versions, and particular implementations of Scala[1], Rust[2], and C++.

time enough for “real world” news from the trenches to emerge. Sure enough, googling around, I found that the brick wall I hit was a common problem which I call the “turtles all the way up” problem:

Anything calling an async function must also be marked with the async keyword. And so on. This can get out of control and spread through a codebase like wildfire.

For example, when I implemented the async loading of a particular http resource deep down in my app, I was forced to declare async/await all the way up my call hierarchy. At first, I started complying — just to get the thing to work, but after I’d added a dozen async calls I still wasn’t finished — so I stopped and regrouped. Something was wrong.

Some have criticised async/await as an anti-pattern for this very reason. The the wikipedia article on async/await refers to this phenomenon as a “zombie virus” — though mitigates this inflammatory description with the explanation:

it has been argued that this contagious nature of the code (sometimes being compared to a “zombie virus”) is inherent to all kinds of asynchronous programming, so await as such is not unique in this regard…

My “turtle avoidance” solutions

If you don’t mind the async keywords spreading through your codebase, or you can control its spread, then that’s great.

If however you have an existing app and you simply want to make a few key portions asynchronous without too much contagion of the async keyword and the async/await paradigm, then read on.

My two solutions revolve around the premise that often you don’t need to wait for the result, especially in a wxPython GUI context. Like when checking for a response from a server regarding what the latest version of your wxPython app is, so that the user can be notified of upgrades. The user doesn’t care when the notification message appears. So you don’t need to await. Luckily wxasync provides a method called StartCoroutine()which lets you call an async function without awaiting it.

Let’s review the important wxasync imports:

from wxasync import AsyncBind, WxAsyncApp, StartCoroutine

• AsyncBind—is an alternative to wxPython Bind() and is there to trigger an async method instead of a normal method, from an event (like a button click, mouse move or menu item click).
• WxAsyncApp we have already talked about — it’s the fundamental way you instantiate your wxPython application if you want to use async/await.
• And StartCoroutine()is how you can trigger an async function, otherwise you would have to await it.

Solution 1 — StartCoroutine()

So instead of async/await calls all over the place, it often might be enough for some bit of code to call the wxasync library’s method StartCoroutine(), thereby work happens and the GUI is still responsive. In fact the main wxasync example on their GitHub page demonstrates this approach. You simply trigger StartCoroutine() calls from any normal bit of wxPython code — even code triggered from button clicks bound with regular wxPython Bind. Nice. There is no need to use the async/await keyword, except for the async on the coroutine function itself, and presumably an await inside that coroutine. Thus no danger of async/await keywords spreading through any call stack.

Solution 2— Raise a custom event

The second variant of this idea has the same premise, but a different implementation approach. You generate a custom wxPython event and emit it from any place in your code base, to trigger an async method which is bound to that custom event.

Defining and raising a custom wxPython event which triggers an async function

See below for a full code sample. With this approach, any “normal” synchronous code can emit this event, and there is no need to contaminate your codebase with async/await keywords, except for the final async function doing the work.

In a sense, the use of an event (or a call to StartCoroutine) decouples and breaks the turtle hierarchy — as long as you don’t have to wait for the result.

If you do need to wait for the result, then you are back to the turtle problem again. The asynchronous use cases in my application Pynsource have so far gotten away with not needing to wait for the final asynchronous result. For example when a user switches to the “PlantUML” tab to see their UML diagram rendered as a PlantUML diagram image — the user can either watch and wait, or do other things in Pynsource whilst they are waiting, or hit ESC to abort at any point. The premise of “not needing to wait for a result” is satisfied, so luckily, I avoided the “turtles all the way up” problem in this case.

Of course I do have several call layers of async/await involved in my PlantUML rendering functionality — the point is, its isolated.

Another complete code example

Here is another complete wxPython app using wxasync and Python 3 — a more technical demo, which demonstrates async/await functionality and also shows the use of:

• Using wxasync with wxPython
• Using AsyncBind
• Calling coroutines
• Raising custom events
• Lots of async goodness

The buttons in the demo GUI app do the following, starting from top to bottom:

• Button click triggers an async method which displays messages. The display of messages is asynchronous, and GUI buttons are still active.
• Button click triggers an async method which is a Coroutine which loops forever or until the button is pressed again, showing a clock ticker at the bottom of the frame.
• Button click triggers an event, which is bound to a normal synchronous method which displays messages. Note the GUI becomes unresponsive and the clock ticker halts until the messages stop.
• Button click triggers an event, which is bound to an asynchronous method which displays messages asynchronously. Note the GUI and the clock ticker are still responsive.
• Button click causes another popup frame to appear. You can close this. The purpose of this was to debug wxasync when using multiple wxPython frames. Ensure you are using the latest version of wxasync to avoid exceptions when closing frames.

This code requires Python 3, wxasync (pip install wxasync) and wxPython (pip install wxPython), which happens to be Phoenix 4.04 as of the writing of this article. It runs on Mac, Windows and Linux. For Linux, to install wxPython, you will need to target a wheel specific to match your version of e.g. Ubuntu e.g. pip install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython see official wxPython installation instructions here.

A full wxPython application using wxasync

Final Words

The ability to use Python 3 asynchronous programming in wxPython has been a wonderful success. There is nothing worse than a GUI app “locking up” during long processes.

I hope you found this article useful. If you are a Python developer, please check out Pynsource my open source UML reverse engineering and UML class diagramming tool — and experience the power of a wxPython async application!

My Other Cool Software

• GitUML — generate UML diagrams instantly from any GitHub repository containing Python code (web app)

GitUML — generate UML diagrams instantly from any GitHub repository containing Python code (web app). A revolution in documentation — diagrams automatically update when you push code using git

• Pynsource — UML for Python (desktop app)
• Python to RPN — Run Python 3 on vintage HP calculators (web app)
• Print42 — Electron-Python app, non-invasive log file annotation tool that supports thermal printers

About Andy Bulka

• Main website and blog www.andypatterns.com
• Medium article: Building a deployable Python-Electron App — Using Electron as a GUI toolkit for Python.
• Medium article: Async/await for wxPython — Python 3 GUI apps with asynchronous functionality.
• Medium article: Getting Python and wxPython apps into the Ubuntu app store
• Github https://github.com/abulka