Model Migration on Ascend NPUs

Published in

Huawei Developers

6 min readJul 17, 2023

Introduction

Hello! Today, we will discuss the topic of model migration, which involves making artificial intelligence models compatible with different infrastructures, and explore Huawei’s model migration solutions for Ascend NPUs.

Let’s get started!

Model Migration

With the rapid advancement of artificial intelligence models today, model migration has become an important step in dealing with existing infrastructure and hardware limitations. But what exactly is model migration?

Model migration refers to the process of transferring a computational or data analysis model from one environment to another. During this process, the model’s operating environment, data sources, infrastructure requirements, or technological framework may change. But why do we need model migration? Let’s list and expand on some common reasons for model migration:

Infrastructure Changes: An artificial intelligence model may have been initially designed to work on a specific infrastructure. However, over time, technological advancements or organizational requirements may necessitate infrastructure changes. In such cases, model migration ensures continuity by making the model compatible with the new infrastructure.
Data Source Changes: AI models are often dependent on the data sources they were trained on. However, data sources can change over time, or there may be a transition to a different data source. In such situations, model migration allows the model to be updated and made compatible with the new data sources.
Performance Improvements: AI models are constantly evolving to enhance their performance. Model migration enables the adoption of a more advanced model, resulting in improved results compared to the current model. For example, migrating to a model that produces more accurate results and has fewer errors in image classification.
Scalability: AI models often require high computational power to process large datasets. Model migration increases scalability by transitioning to a more powerful infrastructure when the current infrastructure is insufficient. This enables the processing of larger datasets and the development of solutions for more complex problems.

As a result, model migration can occur for reasons such as performance improvement, scalability, infrastructure changes, changes in data sources, compliance and standards compliance in the field of AI.

So What Is This Ascend NPU?

NPUs (Neural Processing Unit) are a specialised hardware unit developed by Huawei. These units are designed to accelerate AI operations.

Huawei NPUs perform artificial intelligence calculations more efficiently. Thanks to its specially optimized hardware, it accelerates the training and inference of deep learning models.

Among the key advantages of these NPUs are high performance, low power consumption, on-device processing capability, and data privacy. With high performance, artificial intelligence operations can be carried out more rapidly, enabling support for complex applications.

For more information : Ascend

World of Huawei Ascend: Future with NPUs

Have you ever heard about NPUs? Now, you will meet the true power itself!

medium.com

Model Migration for NPU

When we come this far, we can easily realise that the purpose of our model migration is to adapt the existing artificial intelligence model to Huawei NPUs.

This process includes adapting the Huawei NPUs of the current model to the optimised hardware structure. In addition, there will be operations such as configuring the model and setting its parameters. In addition, the existing data processing flow may need to be reorganised to take full advantage of the parallel computing capabilities of NPUs.

The migration process may require the utilisation of specific libraries and tools designed for Huawei NPUs. These libraries are optimised functions that fully leverage the capabilities of the NPUs and ensure optimal performance.

The model migration process should include testing stages. Running the model on Huawei NPUs and analysing the results may be necessary to evaluate the success of the adaptation and verify the performance.

Considering that open-source deep learning models like Caffe, ONNX, and TensorFlow cannot be directly used for inference on Ascend AI Processor and Ascend AI Software Stack, it is necessary to convert these open-source models into specialized models tailored for the Da Vinci Architecture. For this purpose, Huawei provides us with a tool called ATC (Ascend Tensor Compiler).

Ascend Tensor Compiler (ATC)

ATC optimizes AI models to be run more effectively by Ascend AI processors. This compiler transforms AI models to make the most of the parallel computing capabilities of Ascend processors.

Converting the model to an optimized format enables efficient use of computational resources and improves performance. The compiler also takes full advantage of the special hardware units of Ascend processors, enabling artificial intelligence operations to be performed faster and more efficiently.

atc --model=./yolov3.prototxt \
    --weight=./yolov3.caffemodel \
    --framework=0 \
    --output=./yolov3_caffe_416_no_csc \
    --soc_version=Ascend310 \
    --insert_op_conf=./aipp_yolov3_416_no_csc.cfg

In the code script here, you see the bash command required to convert a yolov3 caffe model to an offline model, ie .om file, which is special for Huawei NPUs. Here — framework=0 means it’s a Caffe model. I do not want to tire you with the rest of the details, it will be enough for now if we understand what this command does. If you want to look at it in more detail, I’ll be leaving a guide for it below.

As an additional information, if our model is not in caffe, TensorFlow or
ONNX format, we must first convert it to ONNX format, then we can convert it to om with ATC.

Pre-Processing & Post-Processing

Pre-processing and post-processing refer to the stages of preparing and editing the input and output data of the artificial intelligence model.

Pre-processing is the process of transforming input data into a suitable format that can be processed by the model. In this stage, operations such as data cleaning, normalization, scaling, or transformation are performed. Additionally, data pre-processing methods such as handling missing data or correcting outliers may be applied. The goal is to prepare the dataset in a way that maximizes the performance of the model.

Post-processing refers to the operations applied to the outputs of a model. The output of the model can be in the form of a probability distribution, labels, or classification results. Various operations are performed to convert these outputs into meaningful results or bring them into the desired format. For example, determining the class with the highest probability, performing classification based on threshold values, or conducting operations like sentence correction or translation in natural language processing applications are carried out in the post-processing stage.

In the process of model migration, pre-processing and post-processing steps are important for the model to adapt to the new environment and produce accurate results. These steps allow the model to adjust to changes in data input and output, making them a crucial part of a successful migration process.

Conclusion

In this article, we talked about model migration, Huawei NPUs. We also talked about what processes we go through and what we pay attention to when migrating a model. In our next post, we’ll do a model migration with examples and dig deeper to understand its use.