Open in app

Sign in

Write

Sign in

How to build MediaPipe on iOS.

Bestolkov Vlad
Deelvin Machine Learning
Published in
9 min readJun 1, 2021

MediaPipe is a framework for building multimodal (eg. video, audio, any time series data), cross platform (i.e Android, iOS, web, edge devices) applied ML pipelines. With MediaPipe, a perception pipeline can be built as a graph of modular components, including, for instance, inference models (e.g., TensorFlow, TFLite) and media processing functions.

Introduction

In this article I would like to share with you my experience of launching an application that demonstrates the work of MediaPipe graph on iOS ( Hello World! on iOS).

The required configuration is:

● MacBook Pro: Big Sur 11.3.1

● Xcode: 12.5

● iPhone 8: iOS 14.5.1

Preparation of MediaPipe

The instruction how to install Mediapipe on macOS can be found here.

1.Install Homebrew:

$ /bin/bash -c “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Command Line Tools for Xcode will be installed with Homebrew.

2. Install Bazel(Full instruction):

$ brew install bazel

For working with Mediapipe we need Bazel 3.7.2 version:

$ cd “/usr/local/Cellar/bazel/4.0.0/libexec/bin” && curl -fLO https://releases.bazel.build/3.7.2/release/bazel-3.7.2-darwin-x86_64 && chmod +x bazel-3.7.2-darwin-x86_64

3. Install OpenCV and FFmpeg:

$ brew install opencv@3#There is a known issue caused by the glog dependency. Uninstall glog.$ brew uninstall --ignore-dependencies glog

4. Install Python 3.7:

Python 3.7 is a preferred option for working with TensorFlow. For installation and configuration, we use Python version manager pydev.

  • Install pyenv:
$ brew install pyenv
  • Enable pyenv in your profile:
$ echo ‘export PYENV_ROOT=”$HOME/.pyenv”’ >> ~/.zprofile
$ echo ‘export PATH=”$PYENV_ROOT/bin:$PATH”’ >> ~/.zprofile
$ echo -e ‘if command -v pyenv 1>/dev/null 2>&1; then\n eval “$(pyenv init -)”\nfi’ >> ~/.zprofile

Install Python 3.7.5

$ pyenv install 3.7.5

Note that there exists a problem installing Python on macOS Big Sur with Xcode 12. If you get an error during installation, you need to install Python with flags (More here: https://github.com/pyenv/pyenv/issues/1643):

$ CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" \LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(xcrun --show-sdk-path)/usr/lib" \pyenv install --patch 3.7.5 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)
  • Rehash shim:
$ pyenv rehash
  • Specifying the Python version:
$ pyenv global 3.7.5
$ pyenv local 3.7.5
  • After performing all the manipulations, restart the terminal and check the Python version (it needs to be 3.7.5)
$ python --version

5. Install six library:

$ pip install –user future six

6. Install numpy:

$ pip install numpy

7. Clone the MediaPipe repository:

$ git clone https://github.com/google/mediapipe.git

8. Run the Hello World! in C++ example:

$ cd mediapipe
$ export GLOG_logtostderr=1# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
    mediapipe/examples/desktop/hello_world:hello_world

# Should print:
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!

9. Install Tulsi:

# cd out of the mediapipe directory, then:
$ git clone https://github.com/bazelbuild/tulsi.git
$ cd tulsi
# remove Xcode version from Tulsi's .bazelrc
$ sed -i .orig '/xcode_version/d' .bazelrc
# build and run Tulsi(12.5 - your Xcode version):
$ sh build_and_run.sh -x 12.5

Creation of a new project and preparation for working with Bazel and Tulsi

  • Create a new application: XCode > Create New Project > iOS >App. You’re your project, organization indicator, language — Objective-C.

We will work the ‘old way’ — without using SceneDelegate.

Make the following changes to the file plist:

  • Delete the key Application Scene Manifest
  • Bundle OS Type code = APPL
  • Add the key Privacy — Camera Usage Description for the future work with the camera.

As we are only working with AppDelegate, the following changes need to be made:

AppDelegate.h:

#import <UIKit/UIKit.h>@interface AppDelegate : UIResponder <UIApplicationDelegate>@property(strong, nonatomic) UIWindow *window;@end
  • AppDelegate.m:
#import “AppDelegate.h”@interface AppDelegate ()@end@implementation AppDelegate- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {// Override point for customization after application launch.return YES;}@end
  • To automatically create Provisioning Profiles for your application:

Project navigator > Select target > Select the “Signing & Capabilities” tab > Check “Automatically manage signing”, and confirm:

  • Save the project and change to the directory where it was saved. We need the following files from the project:

1. AppDelegate.h and AppDelegate.m

2. ViewController.h and ViewController.m

3. main.m

4. Info.plist

5. Base.lproj directory.

6. Assets.xcassets directory.

  • These files need to be copied into a directory with access to the Mediapipe source codes. For example, I created a folder called “edgeDetection” in “mediapipe/mediapipe/examples/” directory, hereinafter $APPLICATION_PATH.
  • In addition to these files, we also need provisioning profiles file for the previously created application. Copy the latest created file in “~/Library/MobileDevice/Provisioning\Profiles/” directory to $APPLICATION_PATH directory and rename it “provisioning_profile.mobileprovision”.
  • Create BUILD file in $APPLICATION_PATH directory with the following rules:
MIN_IOS_VERSION = "11.0"load(
    "@build_bazel_rules_apple//apple:ios.bzl",
    "ios_application",
)ios_application(
#Name of your application
    name = "EdgeDetectiondApp",
#Bundle_id – should coincide with the one mentioned in the application
    bundle_id = "vb.EdgeDetection",
    families = [
        "iphone",
        "ipad",
    ],
    infoplists = ["Info.plist"],
    minimum_os_version = MIN_IOS_VERSION,
#Path to your provisioning_profile file
        provisioning_profile = "//mediapipe/examples/edgeDetection:provisioning_profile.mobileprovision",
    deps = [":EdgeDetectionAppLibrary"],
)objc_library(
    name = "EdgeDetectionAppLibrary",
    srcs = [
        "AppDelegate.m",
        "ViewController.mm",
        "main.m",
    ],
    hdrs = [
        "AppDelegate.h",
        "ViewController.h",
    ],
    data = [
        "Base.lproj/LaunchScreen.storyboard",
        "Base.lproj/Main.storyboard",
        "//mediapipe/graphs/edge_detection:mobile_gpu_binary_graph",
    ],
    sdk_frameworks = [
        "UIKit",
        "AVFoundation",
        "CoreGraphics",
        "CoreMedia",
    ],
    deps = [
    "//mediapipe/objc:mediapipe_framework_ios",
    "//mediapipe/objc:mediapipe_input_sources_ios",
    "//mediapipe/objc:mediapipe_layer_renderer",
    "//mediapipe/graphs/edge_detection:mobile_calculators",
    ],
)

Create a new Xcode project with Tulsi

Tulsi integrates Bazel (Google’s build tool) with Apple’s Xcode IDE.

  • The first step is to create a Tulsi project by launching the Tulsi app.
  • Give your project a name and select the location of the Mediapipe Bazel WORKSPACE file, then click “Next”.
  • At this point you should see the “Packages” tab for your project. Click on the “+” button to add your BUILD file of directory $APPLICATION_PATH.
  • Go to the Config tab. Clicking the “+” button will allow you to add new generator configs. Double clicking on an existing config will allow you to edit it, and the “-” button can be used to permanently delete previously created configs.
  • Select a location to save the Tulsi project and give your Tulsi project name.
  • Select build targets, we need only iOS_application.
  • Do not change project options, click “Next”.
  • Choose source targets — directory your app, click “Save” and set name config.
  • Select your config, then press the Generate button below. You will be asked for a location to save the Xcode project. Once the project is generated, it will be opened in Xcode.
  • Tulsi created an Xcode project and will open it.

A simple camera app for real-time Sobel edge detection

The official instruction is here: Hello World! on iOS. In order not to duplicate the official instructions, the main points and ready-made program code with comments will be described below.

● Rename the file ViewController.m to ViewController.mm to support Objective-C++.

● Copy the following code in ViewController.mm:

#import "ViewController.h"#import "mediapipe/objc/MPPCameraInputSource.h"
#import "mediapipe/objc/MPPGraph.h"
#import "mediapipe/objc/MPPCameraInputSource.h"
#import "mediapipe/objc/MPPLayerRenderer.h"//Declare a static constant with the name of the graph, the input stream and the output stream
static NSString* const kGraphName = @"mobile_gpu";static const char* kInputStream = "input_video";
static const char* kOutputStream = "output_video";
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";@interface ViewController () <MPPGraphDelegate, MPPInputSourceDelegate>// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
// sent video frames on _videoQueue.
@property(nonatomic) MPPGraph* mediapipeGraph;@end@implementation ViewController {
    
// Handles camera access via AVCaptureSession library.
MPPCameraInputSource* _cameraSource;// Display the camera preview frames.
IBOutlet UIView *_liveView;
// Render frames in a layer.
MPPLayerRenderer* _renderer;
// Process camera frames on this queue.
dispatch_queue_t _videoQueue;
}#pragma mark - MediaPipe graph methods+ (MPPGraph*)loadGraphFromResource:(NSString*)resource {
  // Load the graph config resource.
  NSError* configLoadError = nil;
  NSBundle* bundle = [NSBundle bundleForClass:[self class]];
  if (!resource || resource.length == 0) {
    return nil;
  }
  NSURL* graphURL = [bundle URLForResource:resource withExtension:@"binarypb"];
  NSData* data = [NSData dataWithContentsOfURL:graphURL options:0 error:&configLoadError];
  if (!data) {
    NSLog(@"Failed to load MediaPipe graph config: %@", configLoadError);
    return nil;
  }// Parse the graph config resource into mediapipe::CalculatorGraphConfig proto object.
  mediapipe::CalculatorGraphConfig config;
  config.ParseFromArray(data.bytes, data.length);// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
  MPPGraph* newGraph = [[MPPGraph alloc] initWithGraphConfig:config];
  [newGraph addFrameOutputStream:kOutputStream outputPacketType:MPPPacketTypePixelBuffer];
  return newGraph;
}#pragma mark - UIViewController methods- (void)viewDidLoad {
    [super viewDidLoad];//  Initialize the _renderer object
    _renderer = [[MPPLayerRenderer alloc] init];
    _renderer.layer.frame = _liveView.layer.bounds;
    [_liveView.layer addSublayer:_renderer.layer];
    _renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
    
    //Initialize the queue
    dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
        DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
    _videoQueue = dispatch_queue_create(kVideoQueueLabel, qosAttribute);
    
    //The code initializes _cameraSource, sets the capture session preset, and which camera to use.
    _cameraSource = [[MPPCameraInputSource alloc] init];
    [_cameraSource setDelegate:self queue:_videoQueue];
    _cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
    _cameraSource.cameraPosition = AVCaptureDevicePositionFront;
    // The frame's native format is rotated with respect to the portrait orientation.
    _cameraSource.orientation = AVCaptureVideoOrientationPortrait;
    _cameraSource.videoMirrored = YES;
    //Initialize the graph,
    self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
    //Set the ViewController as a delegate of the mediapipeGraph object:
    self.mediapipeGraph.delegate = self;
    // Set maxFramesInFlight to a small value to avoid memory contention for real-time processing.
    self.mediapipeGraph.maxFramesInFlight = 2;
    //Start the graph when the user has granted the permission to use the camera in our app
    [_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
      if (granted) {
        // Start running self.mediapipeGraph.
        NSError* error;
        if (![self.mediapipeGraph startWithError:&error]) {
          NSLog(@"Failed to start graph: %@", error);
        }
        else if (![self.mediapipeGraph waitUntilIdleWithError:&error]) {
          NSLog(@"Failed to complete graph initial run: %@", error);
        }dispatch_async(self->_videoQueue, ^{
              [self->_cameraSource start];
        });
      }
    }];
}-(void)viewWillAppear:(BOOL)animated {
  [super viewWillAppear:animated];
    
    [_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
      if (granted) {
          dispatch_async(self->_videoQueue, ^{
              [self->_cameraSource start];
        });
      }
    }];
}#pragma mark - MPPGraphDelegate methods// Receives CVPixelBufferRef from the MediaPipe graph. Invoked on a MediaPipe worker thread.
- (void)mediapipeGraph:(MPPGraph*)graph
   didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
             fromStream:(const std::string&)streamName {
  if (streamName == kOutputStream) {
    // Display the captured image on the screen.
    CVPixelBufferRetain(pixelBuffer);
    dispatch_async(dispatch_get_main_queue(), ^{
        [self->_renderer renderPixelBuffer:pixelBuffer];
      CVPixelBufferRelease(pixelBuffer);
    });
  }
}#pragma mark - MPPInputSourceDelegate methods// Must be invoked on _videoQueue.
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
                timestamp:(CMTime)timestamp
               fromSource:(MPPInputSource*)source {
  if (source != _cameraSource) {
    NSLog(@"Unknown source: %@", source);
    return;
  }
  [self.mediapipeGraph sendPixelBuffer:imageBuffer
                            intoStream:kInputStream
                            packetType:MPPPacketTypePixelBuffer];
}@end
  • Go to Main.storyboard, add a UIView object from the object library to the View of the ViewControllerclass. Add a referencing outlet from this view to the _liveView object you just added to the ViewController class. Resize the view so that it is centered and covers the entire application screen.
  • Build and Run Application
  • You should get the following result:

Mediapipe examples

I would like to remind you that you can get acquainted with all ML solutions in MediaPipe supported on iOS. The “Mediapipe /” directory contains the MediaPipe.tulsiproj file with all the examples. Create Xcode project in threesteps:

1.Generate a unique prefix for the bundle IDs of our iOS demo apps:

$ python3 mediapipe/examples/ios/link_local_profiles.py

2.Open mediapipe/Mediapipe.tulsiproj using the Tulsi app.

3.Select the MediaPipe config in the Configs tab, then press the Generate button below. You will be asked for a location to save the Xcode project. Once the project is generated, it will be opened in Xcode.

Then you can choose any example you are interested in; be sure to generate a “provisioning profile” according to the following instruction for each application.

Examples of models:

IrisTracking

HandTracking

That’s all for today about my experience of working with mediapipe, I hope you found this instruction useful. Check out our Deelvin Machine Learning blog for more articles on machine learning.

Also I want to thank Anna Gladkova and Ildar Idrisov for their help in writing this article.

Mediapipe
IOS
Objective C

--

--

Bestolkov Vlad
Deelvin Machine Learning

Written by Bestolkov Vlad

9 Followers
Writer for

iOS Software Engineer at Deelvin.com

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams