🤖 Deep Learning

Creating Desktop Apps For TensorFlow Models With Qt

Build a desktop app for a TensorFlow model with PySide6

Photo by Linus Mimietz on Unsplash

In the age of powerful web and mobile apps, desktop app development might seem like a decade old domain to work on. Alongside web and mobile apps, popular frameworks like Flutter and React-Native support building of desktop applications with a single codebase. In such an era, why would we need desktop applications for ML models? In this tutorial, we’ll be developing a desktop app that would run an image super resolution model in TensorFlow (also for PyTorch) with PySide6 which is identical to PyQt.

Features of Desktop Apps, Perks of using Qt and Python

• Security: Data privacy and security of user data have been popular concerns around software development and machine learning. Users fear that their data would be used for training a ML model without their consent. These concerns have given rise to cloud-less technologies which work offline on the users’ devices. Desktop apps would not require a persistent internet connection to operate and thus are safe from malicious attacks and data breaching

40% of organizations have suffered a cloud-based data breach

• Device-dependent performance: This point is subjective, as some of you might treat it as a disadvantage for desktop apps. If you don’t have a GPU on your system, running a web app that uses GPU from the cloud might be beneficial. Contrary, if you have a better GPU configuration on your system or some other optimized way of running the models, then a desktop app could utilize those resources better as it would directly run PyTorch/TensorFlow.

What are the benefits and challenges of developing web apps vs desktop apps?

• If the desktop app is written in Python, it would be super easy to run TensorFlow or PyTorch models as there’s no need of conversion to other optimized formats like tflite or ptl . Additionally, the preprocessing of input data also becomes easy with prebuilt packages available in Python like numpy or pandas or pillow . Also, using Qt with Python would give you more functionality and control towards the UI.

One of the options that you may consider while building a ML demo/app is Streamlit or Gradio. They are excellent tools for ML demos but provide a limited number of widgets and functionality. Also, both of these tools build apps for the web and bringing them to the desktop might not be possible. With Qt, you can have access to multiple widgets and which have seamless customization. Also, using Qt with Python will help us access all of our favorite Python packages and utilities.

GitHub Repository of the Project

Shubham Panchal

🤖 Deep Learning - Techniques, Methods and How To's

View list

17 stories

GitHub - shubham0204/TF_ESRGAN_Desktop_App: A desktop app built with Qt (PySide6) for super…

Before Starting

We’ll need Qt Creator, an IDE to develop apps in Qt with C++ or Python. The recommended installation includes Qt (the framework), Qt Creator and some other tools, but we only require Qt Creator to create .ui files that we’ll use in Python.

Download Source Package Offline Installers | Qt

If you’re using PyQt/PySide6 for building our app, why do we need to download Qt Creator? Why can’t we use the widgets included in PySide6 directly?

Qt Creator provides an easy-to-use drag-and-drop UI editor that can used to create Qt Forms represent by a .ui file. Once we’ve converted the .ui file to a .py file with the pyside6-uic tool, we can easily use the .py file in our program.

Contents

Step 1: Download packages for the Python project

Step 2: Designing the UI of the app with Qt Creator

Step 3: Using the .ui file in Python with PySide6

Step 4: Writing the App’s Logic

Step 5: Bundle the project into an executable with PyInstaller

Step 6: (Optional) Building apps for other platforms with GitHub Actions

Step 1: Download packages for the Python project

The first step is to download all the packages required for our project, in a virtual environment. First, create a virtual environment in the project directory,

$ cd tf_desktop_app
$ python -m venv project_env
$ .\Scripts\activate

Install PySide6, PyInstaller, Pillow and TensorFlow with TF Hub. You can customize the TensorFlow installation if you wish to run the model on the system’s GPU.

$ ( project_env ) pip install pyside6 pyinstaller pillow tensorflow-cpu tensorflow_hub

Step 2: Designing the UI of the app with Qt Creator

In order to create an UI, open Qt Creator, head to File -> New File -> Qt -> Qt Designer Form. Follow the steps ahead and a mainwindow.ui file will be opened (if you haven’t renamed the file) in the form editor.

You will then land up in the form editor where you can drag-and-drop widgets to build a simple UI for our image super-resolution app. The entire process of creating the UI couldn’t be included in this blog, so I’ve created a video in which I’m using two widgets QLabel and QPushButton . We simply drag-and-drop the widgets, modify their position and size, and change their IDs or object names. These object names are important, as they’ll be used in Python to address widgets

Once you’ve completed the UI, make sure you save it and move the resulting .ui file in the Python project.

The .ui file present in the GitHub repo may be different from the one shown in the above video. The difference is only that I’ve added CSS-like customization to the buttons to change their background color / borders.

Step 3: Using the .ui file in Python with PySide6

The .ui file provides XML definitions of the widgets that have been included in the form. To transform these definitions into PySide6 widgets, we use the pyside6-uic tool that gets shipped with the PySide6 package.

$ ( project_env ) pyside6-uic mainwindow.ui > mainwindow.py

On opening mainwindow.py , you observe the definitions are replaced with PySide6.QtWidgets ,

from PySide6.QtCore import (QCoreApplication, QDate, QDateTime, QLocale,
    QMetaObject, QObject, QPoint, QRect,
    QSize, QTime, QUrl, Qt)
from PySide6.QtGui import (QBrush, QColor, QConicalGradient, QCursor,
    QFont, QFontDatabase, QGradient, QIcon,
    QImage, QKeySequence, QLinearGradient, QPainter,
    QPalette, QPixmap, QRadialGradient, QTransform)
from PySide6.QtWidgets import (QApplication, QLabel, QMainWindow, QMenuBar,
    QPushButton, QSizePolicy, QStatusBar, QWidget)

class Ui_MainWindow(object):
    def setupUi(self, MainWindow):
        if not MainWindow.objectName():
            MainWindow.setObjectName(u"MainWindow")
        MainWindow.resize(1000, 700)
        MainWindow.setStyleSheet(u"background-color: #FFFFFF;")
        self.centralwidget = QWidget(MainWindow)
        self.centralwidget.setObjectName(u"centralwidget")
        self.input_img = QLabel(self.centralwidget)
        self.input_img.setObjectName(u"input_img")
        self.input_img.setGeometry(QRect(0, 0, 500, 500))
        self.output_img = QLabel(self.centralwidget)
        self.output_img.setObjectName(u"output_img")
        self.output_img.setGeometry(QRect(500, 0, 500, 500))
        self.select_image = QPushButton(self.centralwidget)
        self.select_image.setObjectName(u"select_image")
        self.select_image.setGeometry(QRect(175, 540, 150, 50))
        font = QFont()
        font.setFamilies([u"Ubuntu"])
        font.setPointSize(12)
        self.select_image.setFont(font)
        self.select_image.setStyleSheet(u"border-width: 2px;\n"
"border-radius: 16px;\n"
"background-color: #4285f4;\n"
"color: #FFFFFF;")
        self.save_image = QPushButton(self.centralwidget)
        self.save_image.setObjectName(u"save_image")
        self.save_image.setGeometry(QRect(675, 540, 150, 50))
        self.save_image.setFont(font)
        self.save_image.setStyleSheet(u"border-width: 2px;\n"
"border-radius: 16px;\n"
"background-color: #4285f4;\n"
"color: #FFFFFF;")
        self.super_resolve = QPushButton(self.centralwidget)
        self.super_resolve.setObjectName(u"super_resolve")
        self.super_resolve.setGeometry(QRect(420, 520, 200, 100))
        self.super_resolve.setFont(font)
        self.super_resolve.setStyleSheet(u"border-width: 2px;\n"
"border-radius: 16px;\n"
"background-color: #FF0000;\n"
"color: #FFFFFF;")
        MainWindow.setCentralWidget(self.centralwidget)
        self.menubar = QMenuBar(MainWindow)
        self.menubar.setObjectName(u"menubar")
        self.menubar.setGeometry(QRect(0, 0, 1000, 26))
        MainWindow.setMenuBar(self.menubar)
        self.statusbar = QStatusBar(MainWindow)
        self.statusbar.setObjectName(u"statusbar")
        MainWindow.setStatusBar(self.statusbar)

        self.retranslateUi(MainWindow)

        QMetaObject.connectSlotsByName(MainWindow)
    # setupUi

    def retranslateUi(self, MainWindow):
        MainWindow.setWindowTitle(QCoreApplication.translate("MainWindow", u"MainWindow", None))
        self.input_img.setText("")
        self.output_img.setText("")
        self.select_image.setText(QCoreApplication.translate("MainWindow", u"Select Image", None))
        self.save_image.setText(QCoreApplication.translate("MainWindow", u"Save Image", None))
        self.super_resolve.setText(QCoreApplication.translate("MainWindow", u"Super-Resolve!", None))
    # retranslateUi

With this Ui_MainWindow , we can achieve the same UI that we would get from the .ui file. We create another file, main.py , that will contain the logic of our super resolution app.

Step 4: Writing the App’s Logic

The first step is to create a class App that will contain methods for handling button press events and image (pre/post)processing for super resolution. Ui_MainWindow is imported and used as self.ui to connect functions with button press events and to handle other UI widgets.

# main.py
from mainwindow_qt import Ui_MainWindow
import PySide6.QtWidgets as widgets
import PySide6.QtGui as gui

class App( widgets.QMainWindow ):

    def __init__(self):
        super( App , self).__init__()
        self.current_secs = None
        self.ui = Ui_MainWindow()
        self.ui.setupUi(self)
        self.setWindowTitle("Super Resolution with ESRGANs")

The next step is to create three member functions that would be called by Qt when the buttons are pressed. We connect these functions with the buttons by using self.ui.<widget>.clicked.connect( <func> ) method and also restrict the resizing of the app’s window.

from mainwindow_qt import Ui_MainWindow
import PySide6.QtWidgets as widgets
import PySide6.QtGui as gui

class App( widgets.QMainWindow ):

    def __init__(self):
        super( App , self).__init__()
        self.current_secs = None
        self.ui = Ui_MainWindow()
        self.ui.setupUi(self)
        self.setWindowTitle("Super Resolution with ESRGANs")

        # Disallow window resizing
        self.statusBar().setSizeGripEnabled(False)
        self.setMaximumWidth( self.geometry().width())
        self.setMaximumHeight( self.geometry().height())

        self.ui.save_image.clicked.connect(self.on_save_image_click)
        self.ui.select_image.clicked.connect(self.on_select_image_click)
        self.ui.super_resolve.clicked.connect(self.on_super_resolve_click)

        self.loading_thread = None
        self.selected_image = None
        self.selected_image_name = None
        self.converted_image = None


    def on_select_image_click(self):
        ...

    def on_save_image_click(self):
        ...

    def on_super_resolve_click(self):
        ...

The on_select_image_click method is expected to open a file chooser dialog from where the user can select an image. The selected image also has to be resized so that it fits correctly in the input_img QLabel without losing its aspect ratio.

from mainwindow_qt import Ui_MainWindow
from PIL import Image
from PIL import ImageOps
from PIL import ImageQt
import PySide6.QtWidgets as widgets
import PySide6.QtGui as gui

class App( widgets.QMainWindow ):

    ...

    def on_select_image_click(self):
        """
        Opens a file select dialog and opens it with PIL.Image
        """
        selected_file_path = widgets.QFileDialog.getOpenFileName(self)
        path = selected_file_path[0]
        if path is not None:
            image_file = open(path, 'rb')
            image = Image.open(image_file).convert('RGB')
            self.set_image(image, self.ui.input_img)
            self.selected_image_name = os.path.basename(path)
            self.selected_image = np.asarray( image , dtype='float32' )

    def on_save_image_click(self):
        """
        Save the converted image to the directory
        """
        selected_file_path = widgets.QFileDialog.getExistingDirectory(self)
        if selected_file_path is not None:
            self.converted_image.save( os.path.join( selected_file_path , f'sr_{self.selected_image_name}.jpg' ) )

    def on_super_resolve_click(self):
        ...

    def set_image( self , image , label ):
        # ( 500 , 500 ) is the size of the `input_image` QLabel
        resized_image = ImageOps.contain( image , ( 500 , 500 ) )
        image = ImageQt.ImageQt( resized_image )
        label.setPixmap(gui.QPixmap.fromImage(image))

Once, the input image is selected by the user we can normalize the image and send it to the super resolution model. We’ll use the caption-pool/esrgan-tf/1 model from TF Hub and perform inference in the on_super_resolve_click . The method, hub.load , is a blocking function, meaning it would stop the execution of the program until it completes its execution. The model loading might take some time, and as we wish to perform it in the App ‘s constructor, we can create a new Thread (with the threading module) and load the model in it.

from mainwindow_qt import Ui_MainWindow
from PIL import Image
from PIL import ImageOps
from PIL import ImageQt
import PySide6.QtWidgets as widgets
import PySide6.QtGui as gui
import numpy as np
import os
import sys

import tensorflow_hub as hub
import tensorflow as tf

class App( widgets.QMainWindow ):

    def __init__(self):
        ...
        # Start loading the model in a worker thread
        self.loading_thread = None
        self.start_tf_model_loading()

        ...

    def on_super_resolve_click(self):
        low_resolution_image = np.expand_dims( self.selected_image , axis=0 )
        low_resolution_image = tf.cast( low_resolution_image , tf.float32 )
        # Normalizing the pixel values to the range [0 , 255]
        output = self.model( low_resolution_image )
        output = tf.clip_by_value( output , 0 , 255 ).numpy()[0]
        self.converted_image = Image.fromarray( output.astype( np.uint8 ) )
        self.set_image( self.converted_image , self.ui.output_img )

    def start_tf_model_loading(self):
        self.loading_thread = threading.Thread( target=self.load_tf_model , name="TF Model Loading" )
        self.loading_thread.start()

    def load_tf_model( self ):
        self.model = hub.load( "https://tfhub.dev/captain-pool/esrgan-tf2/1" )

In order to launch the app, we create an object of App and call the show method.

class App( widgets.QMainWindow ):

    ...


app = widgets.QApplication(sys.argv)
window = App()
window.show()
sys.exit(app.exec())

We’ve completed writing the logic of the app in main.py . The next step is to bundle our Python project into a portable executable or an app, with PyInstaller, that can run on Ubuntu (Linux), Windows and MacOS.

Step 5: Bundle the project into an executable with PyInstaller

In order to build an executable with PyInstaller, we’ll first create a .spec file that contains settings/options for building the executable. Using the pyi-makespec utility,

$ ( project_env ) pyi-makespec --onefile main.py

Observe that main.spec gets created. We’re good to go with the default options except that we add an icon to the executable by including the icon argument,

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None

a = Analysis(
    ['main.py'],
    ...
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)

exe = EXE(
    ...

    icon='app_icon.ico',
    name='Super Resolution',

    ...
)

Next, initiate the executable building process with,

$ ( project_env ) pyinstaller --clean main.spec

Once the build is complete, a portable executable will be added in a dist directory in our project. You may run the executable, to see the app running,

Step 6: (Optional) Building apps for other platforms with GitHub Actions

As you may observe in the GitHub repo of the project, three workflows have been setup to executables for Windows, MacOS and Linux. These workflows are triggered manually and deliver the executables with releases. Here’s the windows_app.yml file,

name: Build Windows Executable

on: [ 'workflow_dispatch' ]

jobs:
  build_app:
    runs-on: 'windows-latest'
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python 3.10
        uses: actions/setup-python@v3
        with:
          python-version: "3.10"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Build the executable with PyInstaller
        run: pyinstaller --clean main.spec

      - name: Upload executable as artifact
        uses: actions/upload-artifact@v1
        with:
          name: windows_executable
          path: dist/main.exe

      - name: Bump version
        id: tag_version
        uses: mathieudutour/github-tag-action@v6.1
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}

      - name: Create a new release
        uses: actions/create-release@v1
        id: create_release
        with:
          draft: false
          prerelease: false
          release_name: Windows App - ${{ steps.tag_version.outputs.new_tag }}
          tag_name: ${{ steps.tag_version.outputs.new_tag }}
          body_path: CHANGELOG.md
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Upload executable to release
        uses: actions/upload-release-asset@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          upload_url: ${{ steps.create_release.outputs.upload_url }}
          asset_path: dist/main.exe
          asset_name: windows_app.exe
          asset_content_type: application/vnd.microsoft.portable-executable

1. First, we setup Python 3.10 with actions/setup-python and installl the dependencies/packages with pip install -r requirements.txt
2. Next, we build the executable with pyinstaller --clean main.spec . The resulting executable is uploaded as an artifact.
3. The current release version is obtained and a new release is created. Finally, the executable is uploaded in the release with actions/upload-release-asset

The End

This brings us to the end of our tutorial. Qt is a powerful framework, and utilizing its combined power with Python, can open new frontiers in desktop apps for ML.

Hope you learnt something new. Feel free to share your comments/thoughts/suggestions in the comments below. Have a nice day ahead!