Electron + React + Python (Part 3) — Boilerplate 3/3

In the last article (written over a year ago) we showed a hacky way of glueing together Python and Electron using a well-structured architecture. The architecture itself was powerful but it was still complicated to use, so I ended up creating a library out of it. Today we shall discuss on how to use it properly using an example application and we will also talk about the changes made in package.json to make it shippable.

“On a side note, I never really expected people to take an interest in the project as it was an extreme use-case so I didn’t feel like continuing with the article series as it was too much work for too little reward. But now that I see people asking me to maintain the library and using it in their own projects, I feel obligated to return something back to the community.” Without further delay, let’s start…

We will talk about two things in this article:

• How to use electron-load-balancer to do background tasks in python
• How to package your app

Using the library

electron-load-balancer

The library we will be using is called the electron-load-balancer. It is a very minimal library that offers a couple of APIs for users to make use of the python realm for mainly two purposes:

1. Reusing python libraries that have not been implemented/ported in JS.
2. Taking advantage of python threads for convenient background processing.

An overview…

Before we move into the sample application that I have built for you guys, let me give you a quick tour of the API I have for you guys and the possible limitations that you need to keep in mind.

1. register( ipcMain, registeredPaths )
2. start( ipcRenderer, processName, values )
3. stop( ipcRenderer, processName )
4. job( ipcRenderer, processName, func, cleanup_func )
5. onReceiveData ( ipcRenderer, processName, func )
6. sendData ( ipcRenderer, processName, value )

That’s it. That is all there is to the library. The documentation for the APIs are heavily described with examples in the documentation. If you find anything confusing, feel free to raise an issue and I’ll address it ASAP.

Now let’s quickly talk about the limitation of the library.

• You need to register possible background tasks way ahead of time during the compilation time. This is not that big of a pain point but worth mentioning.
• There is no way for automatic garbage collection of unused/out of scope hidden processes. As of now, we don’t have that feature included in the library. The developer will be responsible for creating and destroying background tasks using suitable hooks and API calls provided. This is probably the only major limitation I have not addressed in the library so far.

The sample app (Factorial Calculator)

Go ahead and clone the repository on your machine.

AakashMallik/electron-react-boilerplate

First things first, take a look at package.json. Start off with npm install . This shall install all the needed dependencies, and also electron-load-balancer. If you wish to add this library to your project, you would do something like this in your project root.

npm install --save electron-load-balancer

There are many ways to think and approach the problem statement but I would like to show you two of my favourites. In both cases, we will be doing the same thing, but in one case we’ll do it in as a one shot task, in the other we will do it using a continuous preemptive loop.

If you start the application in development mode, you shall two buttons. The text on the two buttons should make it obvious which scenario they represent. Let’s take a look at what’s happening under the hood.

1. One shot task

In this approach, we create the background process only when the user clicks to start and destroy it immediately after we get a result. Let’s write it down formally for comparison.

start: When the user clicks on the button. Multiple clicks will create multiple background processes.

intermediate: The task is done in either a python main thread.

end: The choice of killing the process can be handed over to the user with a stop button. In this case, we are stopping and cleaning up using hooks. Basically, when the background process returns an output, we clean it up.

From the above points, this way of arrangement can be useful when tasks are relatively short-lived and not too complex.

Eg: Giving a pre-trained model a photo and asking it to label it.

2. Preemptive loop

This is one of my personal favourites as it lets you create a bridge between python and JS realm and can be used to solve very complicated real-time problems.

start: Usually started as soon as the electron application starts. Not mandatory and can be changed accordingly.

intermediate: Usually has a main loop that accepts commands from JS realm and spawns helper threads that keep sending back data to the JS realm. Very effective for solving real-time problems.

end: The background processes should be stopped right before the application exits. Not mandatory and can be changed accordingly.

For the above points, this way of arrangement can be useful when you have a background process doing computations in real-time and constantly updating the UI with data.

Eg: A pre-trained model that keeps extracting frames from a video stream and labels them in real-time.

Reflecting the concepts in code

Let’s take a look at the directory structure:

.
├── background_tasks
│   ├── one_shot.html
│   └── preemptive_loop.html
├── package.json
├── package-lock.json
├── public
│   ├── electron.js
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
├── scripts
│   ├── one_shot.py
│   └── preemptive_loop.py
├── src
│   ├── App.css
│   ├── App.js
│   ├── App.test.js
│   ├── Home.js
│   ├── index.css
│   ├── index.js
│   └── serviceWorker.js
└── utils

Alright, to demonstrate the two background approaches mentioned above, we’ll need two HTML files that will act as the hidden renderer process launching the python scripts in the background. You shall hence see two files in the background_tasks directory. Meaningful names have been given to the files.

background_tasks
|
|--one_shot.html
|--preemptive_loop.html

The contents of the files have been commented accordingly.

Let’s move to scripts. In this directory, you shall be writing all the python code that you wish to execute. I’ll have added two files here for the same reasons as above.

scripts
|
|--one_shot.py
|--preemptive_loop.py

Note: The communication between the JS realm and the python realm happens using the HTML files. They are more or less like bridges between electron and python scripts. Also, note that communication happens using stdin and stdout. To keep things structured I have used JSON as the serialization technique. This also helps with debugging in case you need it.

So our background tasks are ready. Now we need to modify the electron main thread to relay the messages that come from the background tasks. If you read the documentation of the library, you shall see note the mentions that this is a trial task that I have opted not to handle using the library.

Head over to the public directory and take a look into:

public
|
|--electron.js

Most of the code here is standard boilerplate code, I have commented the part that has been added. Keep in mind that we also need to register the background tasks here with all the needed details as mentioned in the library. You shall see two statements that register the two background tasks we have created. You shall also see a few event listeners that we have placed to bounce the message from background tasks to the UI thread.

Finally, move to src directory to write some standard ReactJS code. I have added some comments in App.js and Home.js. Pay close attention to them.

src
|
|--App.js
|--Home.js

The code should be fairly simple and like I have mentioned before, cleanup needs to be done by the developers. So make sure you understand how and when that is happening in the sample code.

Now with everything set. Let’s run each of the two techniques one by one.

1. One shot task

As you can see by clicking on the first button, you get one single output every time you click on it. This is basically attacking the problem using a one shot approach. The console shall give you a good idea of what we are receiving from the python realm.

2. Preemptive loop

He every time you click, you see a thread name “fact_thead_x” along with comma-separated values. So back in our python realm, we have a loop running waiting for data. We pass an array of integers from 0–15 and ask it to find the factorial of each of them. It returns the data one number at a time to mimic real-time application streaming data. That is just one thread in action. Clicking the button several times in quick succession will initiate new threads that run in parallel. Hence you can see a new row being printed with each new click. This is a more robust approach to attacking the problem.

I have tried to keep the article as exhaustive as possible but in case you still find yourself confused, feel free to drop an issue on either of the repositories mentioned above or just drop a comment here on Medium. I’ll reach out as soon as I can.

Packaging

Packaging

Everything that you will need in order to configure the packaging pipeline goes into the build key. In code, it would look something like this.

"build": {        
  "appId": "Factorial Calculator",        
  "productName": "F-Cal",        
  "copyright": "Copyright © 2019-2020 Mr. Mallik",        
  "asar": false,        
  "linux": {            
    "target": ["deb","snap"],            
    "category": "Utility"
  },        
  "snap": {            
    "confinement": "classic"        
  },        
  "win": {            
    "target": ["nsis"],            
    "legalTrademarks": "Copyright © 2019-2020 Mr. Mallik", 
    "signingHashAlgorithms": ["sha1","sha256"]
  },
  "nsis": {
    "oneClick": true        
  },        
  "mac": {            
    "category": "public.app-category.education",             
    "target": ["default"]        
  },            
  "files": [            
    "build/**/*",            
    "node_modules/**/*",            
    "utils/**/*",            
    "scripts/**/*",            
    "background_tasks/**/*"        
  ],        
  "directories": {            
    "buildResources": "assets"        
  },        
  "extends": null    
}

Note: This way of adding build config to package.json may be an outdated approach and to my knowledge, it is done using a separate JSON file. But the things you learn here shall be transferable with little to no changes.

Let’s try to understand the config one key at a time.

appID, productName, copyright

These are pretty obvious.

asar

The asar package aka Electron Archive is a standard mechanism for creating bundle out of your code. The reason for doing this is usually to prevent users from poking around in your code. Unfortunately, for the python binding to work, you cannot allow electron-builder to hide your python code inside an asar bundle. Hence, we will opt to keep it False.

linux, win, mac, snap, nisc

The names should have given you a hint that these are keys that let electron-builder know which platforms you are aiming for and what custom metadata you wish to pass along. For the sake of simplicity, we’ll focus on Linux. In Linux, I have opted to create deb (Debian) and snap packages.

For people who don’t know, there are a couple of ways you can install stuff on Linux machines. For Linux systems derived from debian, you can rely on .deb files. Snap happen to be a newcomer that runs apps in a sandbox like environment and you install snap packages from snap store.

You may notice another key for snap itself. This is because some packages can be given specific metadata. Here in snap, I have asked electron-builder to build a package that runs in classic mode.

Similar concepts shall follow for the rest of the platforms you wish to cover.

files

This key plays a very crucial role in making sure your python binding works as expected. You provide it with a list of paths that need to be copied over into your installation bundle as it is. No optimization, no post-processing. Here you can see we have passed all the directories that hold files that:

• are either not directly related to electron (build — this is generated when you create an optimized build from CRA — discussed in previous articles of this series),
• are written in python
• are written in pure JS and are not part of the react realm.

Electron-builder will copy them over with no changes while maintaining the directory structure (hence your relative paths will work just fine)

directories

This is a special directory that holds all your assets like icons and images.

extends

If you miss this out you will likely get an error.

How to modify the folder structure of app.asar when using electron-builder?

So the reason for this is when Electron-builder detects that you are using CRA, it will use a built-in config file for the Electron-React apps, but we have written our own config to create workarounds so that Electron-React apps work well with python. Hence we wish to override the build-in config. Hence, we’ll pass null to this.

Initiating the build via npm

So with all things in place, let take a quick look at the scripts key in package.json

{
.
.
. 
 
  "scripts":{
    .
    .
    .
    "react-build": "react-scripts build",
    "electron-build-linux": "electron-builder --linux -c.extraMetadata.main=build/electron.js",
    "build-linux": "npm run react-build && npm run electron-build-linux"
     .
     .
     .
  }
.
.
.

To start the build process you will run the following command:

npm run build-linux

This will first build the react app’s optimized version and put all of that in build directory, then electron-builder will be called and it will then use the config we just wrote in the previous section to generate the necessary deb and snap packages in a new dist directory.

Note: Sometimes electron-builder stalls out while downloading necessary packages. Please be a bit patient with it…

Honourable mentions

You may have seen electron-log being used everywhere in the sample code. That particular library will save you a hell lot of time with debugging across realms. Console.log will simply not work when you are working on pure electron side. You’ll have to rely on this library to take logs and fix bugs that may not be obvious to you at first glance.

electron-log