Async/await for wxPython
Python 3 GUI apps with asynchronous functionality
Python 3’s support of async/await is the new hotness, allowing cooperative computation with reduced complexity and without needing threads — as long as you use async compatible libraries like aiohttp etc. You can’t use the famous requests library because it is not async aware and will block, but not to worry, those other libraries have you covered.
The wxPython project is a wonderful, full featured cross-platform GUI toolkit that is completely free to use in commercial projects — unlike some (cough, cough, umm, PyQt). Wouldn’t it be great if you could utilise async/await in wxPython? This would be great for the usual reasons:
- checking a server for information, in the background
- running a long process without locking up the UI
Well it seems you are out of luck, because wxPython already comes with its own messaging event loop, and you can’t have two event loops — can you?
Blending event loops
I remember running into this problem a decade ago where I wanted to blend wxPython widgets into PyGame and both had their own event loops. Some interesting initial approaches were unsatisfactory so I ended up writing a meta-event loop that did a bit of wxPython and a bit of PyGame — giving them both a chance to run. I never published the code, but it worked.
Then when listening to an episode of the wonderful podcast Python Bytes, the topic of introducing asyncio to the PyQt GUI toolkit came up — again the solution has been to blend event loops — see asyncqt.
Whilst listening to the above podcast, it struck me as coincidental because I’ve recently been utilising a similar project for wxPython called wxasync which blends the wxPython event loop with Python 3’s asyncio loop. Result? I can now use Python 3’s async/await with wxPython!
wxasync is a library for using Python 3 asyncio (async/await) with wxPython.
A trio of similar solutions!
Benefits to my project — Pynsource
The ability to have asynchronous methods in my Python UML tool Pynsource has been a boon.
I recently upgraded the entire open source Pynsource codebase to Python 3. Beside being a prudent upgrade, it has allowed me to be able to parse and reverse engineer Python 3 source code into UML class diagrams. Actually Pynsource can still parse Python 2 because I switched to the typed-ast parser, which only runs under Python 3, and has the advantage of being able to parse either syntax. In fact, Guido himself worked on this module recently.
Now that Pynsource was converted to Python 3, I wanted to take advantage of all that async/await goodness. :-)
Background app version check
The first thing I added was a new background version check function, to notify users when a new version of Pynsource was available. I now check a server for the latest version — in the background, a few seconds after startup — without affecting the UI.
Background http request to render PlantUML
Asynchronous programming also helped enormously with a new feature of Pynsource which is the ability to reverse engineer Python and render a PlantUML class diagram image. This involves calling an internet based rendering service which can take some time to run, depending on the complexity of the UML class diagram you are trying to generate.
My initial implementation blocked whilst waiting for the server response and the user had to wait - the UI was unresponsive. Not a good look.
wxasync and converting my code to use async/await, the PlantUML rendering in Pynsource is now non-blocking. This is a much better user experience.
This also allowed me to implement a cancel feature: hitting
ESC will cancel the render, just in case it is taking too long or has locked up. Wonderful!
I even have a seconds counter in the status bar telling you how much time the internet call is taking. The counter is only possible because of async/await. The resulting PlantUML diagram soon arrives:
How to convert your wxPython app to use wxasync
You should be able to convert any existing wxPython application to use
wxasync as long as it is written in Python 3 of course, since the async/await technology is only available in Python 3.5 and later.
You should probably learn a little about the async/await approach before attempting this — there is a bit of a learning curve getting your head around this stuff. There are plenty of tutorials on it when you search.
I’ll be bold and try to sum up async/await up in a single paragraph:
Instead of traditionally invoking your Python program to run from top to bottom you must invoke it via a special loop which runs forever or till your program exits. Inside the loop, your code will run from top to bottom as usual but if you call an
async function containing an
await call to a long running function (e.g. a http request via aiohttp), your code
yields to the main loop — allowing the loop to try to run other tasks. Tasks are just another name for these async functions that are in various states of suspension. It’s a cooperative thing — no threads or interrupts. When your code reaches the bottom, it of course stops and the only remaining code that will run in the cooperative loop are any leftover async tasks/functions. In the case of wxPython, you never reach the bottom of your code — the wxPython event loop runs forever, till you explicitly exit.
I’m definitely not an async/await expert, so apologies for any inaccuracies in my summation, above. It’s just meant to be an explanatory perspective that may help some Python programmers grok the idea. Remedial and official doco on Python 3 async/await event loops is here and broader official async/await doco is here.
The first thing you do is import
wxasync and change your app to subclass from
WxAsyncApp instead of the traditional
from wxasync import AsyncBind, WxAsyncApp, StartCoroutine
# class MainApp(wx.App):
Then instead of starting your app with
application = MainApp(0)
you start it with
from asyncio.events import get_event_loop
application = MainApp(0)
loop = get_event_loop()
You should now be able to call async functions or methods, and still have the wxPython GUI remain responsive. For example here is my call to trigger the check for the latest version of my program — which happens in during app startup in
The “coroutine” being called is simply my async function
check_for_updates which first sleeps for a while using
await asyncio.sleep(15), so that the version check doesn’t happen straight away and annoy the user.
check_for_updates does not use the requests package to retrieve the url, it instead uses a call to my function
url_to_data which uses
aiohttp. Yeah, the syntax of the
url_to_datafunction is a bit gnarly but it does the job and I don’t have to think about it, now that it is written — it grabs the contents of a url, asynchronously, which means it yields to the wxasync event loop whilst waiting for the server response, allowing both wxPython and Python 3 tasks to have a bit of cpu time.
Notice that I had to explicitly decode the server response text from
utf-8 before accessing it as a Python string.
Notice also that I cache the call to the server, so that next time, the server call is avoided for the same rendering job. I had to use a special version of lru_cache namely async_lru since the standard Python lru_cache does not work with async functions.
Here is a fun wxPython app that uses async/await. You drag the mouse to create a selection rectangle. Note that at some point in the demo, I right click the mouse to start the async timer which displays in the status bar. The timer runs independently and asynchronously whilst I am dragging in the GUI — nice!
The full source code to this app is:
Other asynchronous approaches
To be fair, there are a few things you can do in wxPython to “loosen up” the event loop without using async/await.
You can call
wx.SafeYield() at any time which allows the main wxPython event loop to “breathe” and process GUI events. However you can soon get into recursion problems if another wx.SafeYield() happens during the wx.SafeYield(). Plus you are not taking advantage of async/await and all the libraries that support it. Still, I use this approach during my diagram layout so that the incremental results of layout animate for the user:
You can also use wx.CallAfter which queues a regular function or method to be run next time the wxPython loop gets a chance. That’s cooperative, like async/await is.
Then there are threads. There is a portion of my Pynsource app which does not use async/await during a long task — it uses threading. That is the deep layout algorithm feature for “laying out” UML diagrams more optimally. This works like normal layout but runs for longer, running the entire layout several times, using different parameters, then chooses the “best looking” layout. This takes several seconds to run and “think”.
Of course I wanted to run this deep layout and not lock up the GUI — the only solution five years ago was threading. It got a little complex and you can look at the Pynsource GitHub source code to see how I did it. It was based on the wxPython article on long running threads. Using wxasync is much easier to use than background threads and CallAfter’s. Presumably I could now revisit that code and use async/await instead — or I could just leave it, as it isn’t broken. :-)
Is async/await an anti-pattern?
Are there any downsides to async/await? Surely not.
Yet soon I hit a conceptual brick wall — it startled me and got me researching... It turns out that async/await has been around in other languages for a long time
time enough for “real world” news from the trenches to emerge. Sure enough, googling around, I found that the brick wall I hit was a common problem which I call the “turtles all the way up” problem:
Anything calling an async function must also be marked with the async keyword. And so on. This can get out of control and spread through a codebase like wildfire.
For example, when I implemented the async loading of a particular http resource deep down in my app, I was forced to declare async/await all the way up my call hierarchy. At first, I started complying — just to get the thing to work, but after I’d added a dozen
async calls I still wasn’t finished — so I stopped and regrouped. Something was wrong.
Some have criticised async/await as an anti-pattern for this very reason. The the wikipedia article on async/await refers to this phenomenon as a “zombie virus” — though mitigates this inflammatory description with the explanation:
it has been argued that this contagious nature of the code (sometimes being compared to a “zombie virus”) is inherent to all kinds of asynchronous programming, so await as such is not unique in this regard…
My “turtle avoidance” solutions
If you don’t mind the async keywords spreading through your codebase, or you can control its spread, then that’s great.
If however you have an existing app and you simply want to make a few key portions asynchronous without too much contagion of the async keyword and the async/await paradigm, then read on.
My two solutions revolve around the premise that often you don’t need to wait for the result, especially in a wxPython GUI context. Like when checking for a response from a server regarding what the latest version of your wxPython app is, so that the user can be notified of upgrades. The user doesn’t care when the notification message appears. So you don’t need to await. Luckily wxasync provides a method called
StartCoroutine()which lets you call an async function without awaiting it.
Let’s review the important wxasync imports:
from wxasync import AsyncBind, WxAsyncApp, StartCoroutine
AsyncBind—is an alternative to wxPython Bind() and is there to trigger an async method instead of a normal method, from an event (like a button click, mouse move or menu item click).
WxAsyncAppwe have already talked about — it’s the fundamental way you instantiate your wxPython application if you want to use async/await.
StartCoroutine()is how you can trigger an async function, otherwise you would have to await it.
Solution 1 —
So instead of async/await calls all over the place, it often might be enough for some bit of code to call the wxasync library’s method
StartCoroutine(), thereby work happens and the GUI is still responsive. In fact the main wxasync example on their GitHub page demonstrates this approach. You simply trigger
StartCoroutine() calls from any normal bit of wxPython code — even code triggered from button clicks bound with regular wxPython
Bind. Nice. There is no need to use the async/await keyword, except for the async on the coroutine function itself, and presumably an await inside that coroutine. Thus no danger of async/await keywords spreading through any call stack.
Solution 2— Raise a custom event
The second variant of this idea has the same premise, but a different implementation approach. You generate a custom wxPython event and emit it from any place in your code base, to trigger an async method which is bound to that custom event.
See below for a full code sample. With this approach, any “normal” synchronous code can emit this event, and there is no need to contaminate your codebase with async/await keywords, except for the final async function doing the work.
In a sense, the use of an event (or a call to
StartCoroutine) decouples and breaks the turtle hierarchy — as long as you don’t have to wait for the result.
If you do need to wait for the result, then you are back to the turtle problem again. The asynchronous use cases in my application Pynsource have so far gotten away with not needing to wait for the final asynchronous result. For example when a user switches to the “PlantUML” tab to see their UML diagram rendered as a PlantUML diagram image — the user can either watch and wait, or do other things in Pynsource whilst they are waiting, or hit ESC to abort at any point. The premise of “not needing to wait for a result” is satisfied, so luckily, I avoided the “turtles all the way up” problem in this case.
Of course I do have several call layers of async/await involved in my PlantUML rendering functionality — the point is, its isolated.
Another complete code example
Here is another complete
wxPython app using
Python 3 — a more technical demo, which demonstrates async/await functionality and also shows the use of:
- Using wxasync with wxPython
- Using AsyncBind
- Calling coroutines
- Raising custom events
- Lots of async goodness
The buttons in the demo GUI app do the following, starting from top to bottom:
- Button click triggers an async method which displays messages. The display of messages is asynchronous, and GUI buttons are still active.
- Button click triggers an async method which is a Coroutine which loops forever or until the button is pressed again, showing a clock ticker at the bottom of the frame.
- Button click triggers an event, which is bound to a normal synchronous method which displays messages. Note the GUI becomes unresponsive and the clock ticker halts until the messages stop.
- Button click triggers an event, which is bound to an asynchronous method which displays messages asynchronously. Note the GUI and the clock ticker are still responsive.
- Button click causes another popup frame to appear. You can close this. The purpose of this was to debug wxasync when using multiple wxPython frames. Ensure you are using the latest version of wxasync to avoid exceptions when closing frames.
This code requires Python 3, wxasync (
pip install wxasync) and wxPython (
pip install wxPython), which happens to be Phoenix 4.04 as of the writing of this article. It runs on Mac, Windows and Linux. For Linux, to install wxPython, you will need to target a wheel specific to match your version of e.g. Ubuntu e.g.
pip install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython see official wxPython installation instructions here.
The ability to use Python 3 asynchronous programming in wxPython has been a wonderful success. There is nothing worse than a GUI app “locking up” during long processes.
I hope you found this article useful. If you are a Python developer, please check out Pynsource my open source UML reverse engineering and UML class diagramming tool — and experience the power of a wxPython async application!