Go to the profile of Fuse
Fuse
Fuse lets developers and designers build native apps on iOS and Android in a better and faster way.

Fuse Foreign Speech

by Remi Pedersen, VP of Products at Fuse

It’s been long overdue that I sat down and tried out foreign code — a way to integrate native platform APIs with Fuse. So, when I finally got around to trying it out I figured I’d turn it into a small blog experiment:

How hard would it be for someone with virtually no iOS developer experience (that’s me) to integrate a native platform feature in a Fuse app?

To make things even more interesting I wanted to target one of the brand new iOS 10 features that are still in beta, and of course: use a plain, off-the-shelf install of Fuse to do it.

I decided to go with the new speech recognition API since it’s an awesome feature and enables neat demos (with very little code required). As you can see from this video, it worked out quite well. :)

Man talks to phone, is pleased.

Foreign code

As mentioned above; foreign code is the mechanism which lets you integrate new Android and iOS features with Fuse. It does this by providing an easy way to write native code (Java, Objective-C) which can then be called from JavaScript & UX markup, the two main building blocks of Fuse apps.

This is really powerful and means that:

  • Any API features available to apps built with the standard native tools can also be accessed from Fuse.
  • You don’t have to wait for us (or anyone else for that matter) to create a “Fuse-version” of that special library you need.
  • You can benefit from all the existing documentation and resources out there on iOS and Android native development.
For more in-depth information on foreign code, you should check out this blog post from Chris and the official documentation.

Setting up

I started off by upgrading one of our test devices (an iPhone 5S) to the latest iOS 10 beta and also installed the Xcode 8 beta on my Mac. The latter was needed both to get access to the new iOS SDKs and to be able to deploy to iOS 10 devices in general. This was very straight-forward, all done according to Apple’s instructions without any Fuse-specific customization.

I also looked up one of the official example projects for the speech framework (the SpeakToMe sample), both to verify that my new iOS and Xcode beta installs were working as expected (they were!) and to use as a starting point for my own code.

Workflow

When exporting an iOS app, Fuse compiles all your UX markup to C++, combines it with any foreign code (Objective-C) you’ve written, and then produces an Xcode project which you can build & deploy just like any other native app.

Thus, after creating a new Fuse project, exporting it for iOS and opening it in Xcode, my workflow was as follows:

  • Write the small test app (using UX and JavaScript), as well as some thin foreign code wrappers and glue code, in Sublime (any text editor will do, of course).
  • Whenever I did a new export from Fuse, Xcode would automatically reload the changed files and be ready to deploy and debug on the attached iPhone.
  • Since the bulk of the foreign code was kept in a separate ObjC file I edited this in Xcode to benefit from its excellent autocompletion. This made it really easy to work with the native APIs, probably because this is the way you’re supposed to write Objective-C. :)

Of course, for any modifications to the UX and JavaScript I didn’t have to rebuild at all. As I exported in Fuse preview mode, those changes were instantly applied to the running app, in real time.

Implementation

My goal was to create a small app that you could talk to and which would then repeat back to you (as text on the screen) what you had just said. At that point, the API integration should be complete enough that I could expand the app further, for instance by making it respond to something the user said, just by writing some more JavaScript and UX markup.

A word of warning, though: As this was a quick proof of concept I allowed myself to focus on the main functionality rather than things like stability and error handling. This is not an example of best practices or production-ready code, but hopefully a useful starting point.

Wrappers and hooks

This part was taken more or less directly from the Fuse documentation for native interop. It’s a small set of Objective-C and Uno functions for invoking the actual speech functionality, as well as receiving callbacks and connecting it all to JavaScript in the form of a native module.

You can take a look at it here. It weighs in at around 60 lines of code, of which most is boiler-plate stuff. In other words: not exactly rocket surgery!

[Require()]

The .uno file also contains some macros to tell Fuse which frameworks and permissions the app needs.

When developing something from scratch in Xcode, you’d add things like permissions in the app’s info.plist file. In Fuse we have a nifty [Require()] macro which means we don’t have to do this manually every time the Xcode project is generated.

For instance, to add the Speech framework:

[Require(“Xcode.Framework”,”Speech.framework”)]

And to add permissions for audio recording and speech recognition:

[Require(“Xcode.Plist.Element”, “<key>NSMicrophoneUsageDescription</key> <string>We are good listeners.</string>”)]
[Require(“Xcode.Plist.Element”, “<key>NSSpeechRecognitionUsageDescription</key> <string>We would like to understand you.</string>”)]

Actual Functionality (TM)

The code that interacts with the iOS APIs was written completely in Objective-C, in a separate .mm file. It sets up microphone input and speech recognition and invokes a callback whenever the string representing the user’s words is updated.

Most of this was a straight-forward porting job from the original Swift sample mentioned earlier (all errors and omissions are my own, obviously).

It’s also possible to integrate Swift code with Objective-C, but since I’d rather focus on the actual app functionality (and it only took about 50 lines of ObjC) I didn’t really consider that route this time around.

Test application

This was very easy. I just included my new native module, wrote some JavaScript that used it, and created a (very) simple UI in UX markup.

The whole thing, including our guest-starring duck, looks like this:

All the UX and JavaScript we needed.

Conclusion

I think it all worked out very nicely — even surprisingly so, as you may have understood from the video! :)

Of course, there’s bound to be features that are harder to integrate than just relaying some function calls and callbacks, but I was still impressed by just how easy it was to play around with a brand new API like this.

You can find all the code here. It would be really cool if someone decides to pick this up and take it from proof-of-concept to a complete API integration. If you do, be sure to let me know! :)

To learn more about Fuse, check out our ever-expanding list of examples (with source code of course), join our community (we have a nice forum and a Slack group too) or simply follow us on Twitter or Facebook.