👩‍💻How to use Huawei Moxing AI Framework ❓❗

Moxing Framework

Introduction 👔

Hi everyone👋😊 In this article, I will tell you about the moxing framework developed by Huawei. I will also talk about how you can use this framework in your AI projects. Enjoyable readings 👈😊☕🍫

MoXing provides some basic elements for using the framework. It can be used, for instance, to access Huawei Cloud’s Object Storage Service (OBS). It is independent of any particular AI engine and is compatible with all of ModelArts’ supported AI engines, including TensorFlow, MXNet, PyTorch, and MindSpore. Currently, the OBS operation component, or the mox.file APIs detailed in the following, is the major component of the MoXing Framework.

There are some services that are accessed by moxing framework. An object storage service is known as OBS. It is not possible to access OBS files in the same manner that one would a local UNIX file system. OBS files can only be read and written through network requests. For actions on the OBS directory in ModelArts, you are encouraged to utilize MoXing Framework (mox.file).

To open a local file, Python typically offers the following function:

with open('/tmp/a.txt', 'r') as f:
  print(f.read())

OBS provides many functions and tools, such as SDK, API, OBS Console, and OBS Browser. ModelArts mox.file provides a set of APIs for accessing OBS. The APIs simulate the operations of a local file system, allowing users to operate OBS files conveniently. For example, you can use the following code to open a file on OBS:

import moxing as mox
with mox.file.File('obs://bucket_name/a.txt', 'r') as f:
  print(f.read())

The following Python code lists a local path:

import os
os.listdir('/tmp/my_dir/')

To list an OBS path, you can use the following code of mox.file:

import moxing as mox
mox.file.list_directory('obs://bucket_name/my_dir/')

🔰 Calling mox.file

Enter the following code to implement the following simple functions:

1. Introduce MoXing Framework.
2. Create the demo folder in the existing modelarts-bucket/moxing directory.
3. Check whether the demo folder exists. If the folder exists, the preceding operation is successful.

import moxing as mox

mox.file.make_dirs('obs://modelarts-bucket/moxing/demo')
mox.file.exists('obs://modelarts-bucket/moxing/demo')

🔰 Write a string to a file

For example, write Hello World! into the obs://bucket_name/obs_file.txt file.

import moxing as mox
mox.file.write('obs://bucket_name/obs_file.txt', 'Hello World!')

✅ You can also open the file object and write data into it. Both methods are the same.

import moxing as mox
with mox.file.File('obs://bucket_name/obs_file.txt', 'w') as f:
  f.write('Hello World!')

Also, you can append content to an OBS file.

For example, append Hello World! to the obs://bucket_name/obs_file.txt file.

import moxing as mox
mox.file.append('obs://bucket_name/obs_file.txt', 'Hello World!')

🔰 API Mapping

● Python: local filež operation APIs of Python. The APIs can be shifted to the corresponding MoXing file operation APIs by one click.

● mox file : MoXing file operation APIs of MoXing Framework. The APIs correspond to the Python APIs.

● tf.gfile: TensorFlow APIs with the same functions as MoXing file operation APIs. In MoXing, file operation APIs cannot be automatically switched to TensorFlow APIs. The following table lists only the APIs with similar functions.

Comparing moxing file APIs and other APIs

In addition, you can copy the notebook codes files into obs bucket.

import moxing as mox
mox.file.copy_parallel("./train","obs://custom-algo1/code")

Conclusion 🎉

In this article, I have shared some scenarios about how the huawein moxing framework is used. In my upcoming articles, I will talk about the new features and usage areas of Moxing. I have shared the points that need attention with you. Don’t forget to follow me on Linkedin too 🤗🙌

References📜📚

Introduction to MoXing Framework_ModelArts_MoXing Developer Guide_Huawei Cloud