Is it only animated poo?

Published in

codequest

9 min readOct 2, 2017

A new iPhone X from dev’s perspective

Just a day after Apple’s conference on September 12, 2017 a colleague of mine from codequest asked me what do I, as an iOS developer for many years, think about the new iPhone X. Is it revolutionary? What will it change for us, developers?

Well, my first answer was: not so much, it’s just a new iPhone. For us the most important thing is the new iOS that is announced every year. But on a second thought a few facts started to come to me. We will have a new phone’s resolution and new hardware possibilities that comes hand in hand with iOS 11 software, some of which are actually quite interesting.

What is iPhone X?

A few basic facts for those who don’t follow every news that Coupertino giant announces: on September 12 Apple introduced 3 new phones: iPhone 8, 8+ and iPhone X (which is pronounced „ten”, if you don’t want to be a person that calls „gif” a „jiff”). It’s called „ten” because it was announced 10 years after the first iPhone release.

Models 8 and 8+ weren’t much of a surprise, but the iPhone X seemed revolutionary. Of course it has better hardware: 5.8-inch OLED display, A11 Bionic processor with neural network engine, wireless charging. But the more interesting part is that it doesn’t have a home button, which we know and love from the beginning of, well, iPhones, and it will be unlocked with your face, not your finger. It has two sets of powerful cameras: so called True Depth front camera and dual rear cameras.

But what does it mean for a developer?

Most changes that come along with iPhone X for a developer can be placed in three groups:

New screen size, layout and interface guidelines
Using FaceID
iPhone X cameras and iOS 11 frameworks, mostly ARKit

What’s the screen size?

The new iPhone X will have 1125x2436 pixels resolution, but for us developers the more important is that it’s 375x812 points. It means that it’s a first true @3x device. iPhones with a „+” (so 6+, 7+ etc) were using @3x assets as well, but the resolution from images was scaled down to match physical pixels on the device. So don’t worry, we will use the same @3x assets as before for iPhoneX. Width in points will be the same as iPhone 6/7/8, but 145 points higher.

Screeshot from https://developer.apple.com/ios/human-interface-guidelines/overview/iphone-x

What will happen to my old app?

Probably nothing, if you use autolayout and your app takes into account interface guidelines, layout should extend to fill the extra space. However, it is always advisable to check your app on iPhone X simulator before the phone is released.

The problem will occur if your app still uses png image as a Launch Screen, instead of a xib or storyboard. Then the app will stay at iPhone6/7/8 size with black areas on top and on the bottom of the screen until you add a launch screen image with appropriate for iPhone X resolution (the same trick that was used when iPhone 5 was released with old apps).

What is interesting and should be accounted for is that we won’t have a home button, but a home bar — a small 2 points high bar on the bottom of the screen, that by swiping up will be simulating home button behaviour. So the new guidelines say not to use any swipe gestures on the bottom of the screen(or on the edges in a horizontal app). We will have however the ability to hide home bar by overriding UIViewController’s method prefersHomeIndicatorAutoHidden() -> Bool, but Apple asks us not to do so if we don’t have any special agenda (e.g. like displaying full screen image slides).

Status bar in iPhone X won’t have 20 points of height and navigation bar in iOS 11 won’t be 44 points height, like we’re so used to. Any frame counting that takes into account these static values should be replaced, so our app won’t layout inappropriately.

You can read all layout guidelines in a specifically prepared tab for iPhone X on Apple’s site.

How can we use FaceID?

Apple made it easy for us to use TouchID authentication in apps, beginning from iOS 8. We have a LocalAuthentication framework and with a few lines of code can pop up a TouchID form to the user. Now it gets even better: the same API will be used in iOS 11 to authenticate a user by using FaceID. So your apps that use TouchID right now on iPhone X will use FaceID instead.

What is a True Depth front camera?

Screenshot from Apple conference: https://www.apple.com/apple-events/september-2017/

A True Depth camera is basically a little Kinect, to compare it to something we all know. By saying little I don’t mean „worse”, but only a physically smaller version. True Depth camera consists of an infrared camera, flood illuminator, proximity sensor, ambient light sensor, speaker, microphone, 7MP camera, and Dot projector that hits 30,000 points on your face. All this data processed through an A11 Bionic processor gives the possibility to create a fully animated 3D mesh of your face.

So can I create my own animojis?

Animojis are animated 3D emojis that mimic your face expressions, thanks to the front True Depth camera (among them animated poo — hence the title of this post). They were presented on the conference and will be available in iPhone X iMessage app.

Unfortunately, API to add your own animojis to the iMessage won’t be open for devs in iOS 11.

But the true power of iPhone X cameras for the developers comes with new iOS 11 frameworks, mostly — ARKit.

What is ARKit?

AR stands for Augmented Reality, which is a technology that uses a camera and specific software to put virtual images on top of real-life images in camera’s view that we see. Probably the most popular apps right now that uses AR are Pokemon Go and Snapchat.

Image from CNBC, https://www.cnbc.com/2017/08/29/iphone-8-augmented-reality-apps-from-ikea-food-network-and-more.html

How can a developer use True Depth camera with ARKit

In ARKit there is an API of „lower” and „higher” level used to unveil True Depth camera magic.

ARFrame returned by ARSession contains AVDepthData — a container for per-pixel distance or disparity information about things that are in front of the camera, so we have 3D information.

From Apple’s SDK info: when you run a face-tracking AR session (see ARFaceTrackingConfiguration), the session automatically adds to its list of anchors an ARFaceAnchor object when it detects the user’s face with the front-facing camera. Each face anchor provides information about the face’s current position and orientation, topology, and facial expression.

The geometry property of ARFaceAnchor have information about the whole 3D mesh of your face.

Screenshot from Apple Developer presentation: https://developer.apple.com/videos/play/fall2017/601/

The higher level API includes BlendShapes — facial features that are recognised by the SDK, such as jawOpen, eyeBlinkLeft, and eyeBlinkRight coefficients. There are more than 50 movements on your face recognised and returned as a float value from 0.0 to 1.0, to indicate the intensity of a movement.

How about rear cameras?

To obtain 3D information about objects that we see in front of a camera not only infrared technology can be used, but also stereoscopic vision. Dual rear cameras can do just that. These are not so revolutionary in iPhone X, because they were added to iPhone 7+ and 8+ now as well. However, it’s iOS 11 that gives us the possibility to capture depth map in AVFoundation framework’s AVCaptureDepthDataOutput.

What can we do with a depth map? When we combine AVFoundation’s depth map, CoreML, Vision and ARKit — only our imagination is left. For example, when placing an AR object, let’s say a cup, on a table, we don’t have any information of other objects that are placed there, so the cup can unnaturally occlude with a mug that is really standing on the table. Thanks to image processing from Vision and recognition from CoreML we could recognise and track the mug. Combined with 3D information from depth map, that gives us the distance from us to the mug, we could place our virtual cup in front or behind a mug. So the augmented reality things start to interfere with real objects.

What apps can we create for iPhone X?

Since developers will have access to a fully animated 3D mesh of a face probably there will be lots of clones of Snapchat filters. Snapchat spent millions on something that will be right now within a hand’s reach. Probably all the „apply-make-up” or „do-a-makeover” apps will flood the AppStore. Thanks to ARKit high level API for facial features games controlled by them could be created.

But let’s think further and less mainstream: apps or games for autistic kids to learn facial emotions or even sign language recognition apps, since we have depth map from the front camera — these will be now easier to be made than ever.

And many other apps, creative thinkers and developers will surely figure out something new and unexpected, thanks to the possibilities of iOS 11 and iPhone X.

Ok, but is it all revolutionary?

There is no direct answer to that. During my studies I was very focused on image processing and recognition and personally, for me, it probably is, and most certainly will be. Right now, in iOS 11, ARKit has some nice methods, but surely will be more powerful in time.

Inserting an infrared camera into a phone isn’t a new idea. The competition doesn’t sleep and to be fair — had the same thing done a long time ago. Google’s project Tango was started in 2012. But project Tango is more a hardware reference design plus the software for AR and since Google gives only the Android OS for phone producers it couldn’t make them to insert a special, quite expensive hardware into their phones. In 2016 one phone was released that had Google Tango Project in it: Lenovo Phab 2 Pro, but the market wasn’t too good for it. It exists more as an interesting device for developers, but not for everyday users.

A few months back Google announced ARCore — a framework that basically is the same as ARKit, just with some implementation differences. The only problem is that Google doesn’t produce their phones (except Google Pixel) and their users aren’t so used to spending a lot of money on them. So probably there won’t be any infrared or dual cameras, which give ARKit so much more possibilities and a better accuracy.

Probably Apple will insert True Depth cameras in all their next phones, which will give their technology (AR, computer vision processing, face recognition) a possibility to be spread among many users and can change the way we use our phones and create apps. I’m looking forward to it.

Links and references:

Layout:

FaceID:

https://developer.apple.com/documentation/localauthentication

iPhone X cameras + ARKit:

Is it only animated poo?

Written by Hanna Dutkiewicz