Hi, I’m Nikhil and I’m a software engineer on Adobe Capture. This article is for anyone who is interested in how Adobe Capture works from a technical standpoint.
“Creativity can happen anywhere” is the core mantra behind Adobe Capture — an app that transforms a mobile’s camera feed into design assets that show up magically in Adobe’s desktop apps. One can use it to capture assets such as shapes, colors, patterns, etc. from their surroundings with a few basic clicks. Each such asset, however, has its own transformation pipeline that twists and turns the raw camera feed in various ways, mixing it with some special ingredients along the way, to make them easily digestible later!
Despite such variations, the app aims to serve as a unique platform where users can switch these ingredients according to their whims and fancies. In technical terms, this involves switching the entire computation pipeline and UI options as the user swipes from one asset to the other, while keeping the camera feed intact and UI responsive.
During development, this meant that we needed to abstract out the aspects that were specific to a design asset and still provide a common enough interface for our camera module to relay its feed. Thus, we developed a CameraController and the concept of CameraClient — a platform-agnostic interface that could be used to configure it. The CameraClient serves as a single point of communication between the CameraController and its clients. Each client can provide its own implementation of the CameraClient interface to customize the entire computation stack along with its look and feel based on its requirements. CameraController itself is a standalone component which can directly be embedded into the view hierarchy and exposes a mechanism to configure a CameraClient.
Each section below describes a configuration parameter in CameraClient and its usage paradigm via sample code snippets in Swift. Read on if you are interested in how we made it work in Capture or you are an engineer at heart who just loves Swift and mobile camera feeds.
To develop a basic camera app, a developer simply requires a preview of the live feed and a mechanism to receive the captured image. As soon as the CameraController is embedded in the view hierarchy, the preview feed is setup automatically with a set of default UI options.
On capture, the CameraController processes the captured image and uses the CameraCaptureHandler to convey it to the client. CameraCaptureHandler, below, is the first configuration parameter that is provided as a part of the CameraClient interface. Thus, the standard camera setup is almost trivial with the provided CameraController that hides away all platform complexities.
Within Adobe Capture, extraction of assets such as “shapes” and “patterns” require a dozen filters to process each camera frame and replace the standard camera preview entirely with the respective filter outputs, while “colors” and “looks” simply overlay the processed content. So, our team’s developers wanted the ability to customize the camera feed for each design asset.
This meant that we needed a mechanism for each CameraClient to specify a set of GPU filters that could be easily plugged into the camera’s pipeline, while running the standard one in case none is provided. This led to the development of MetalPipeline on iOS and GLPipeline on Android, which took an array of MetalFilters or GLFilters respectively, where each filter could be added or removed dynamically.
Camera Feed Processing
Our developers at Adobe also wanted to process the camera frames asynchronously in order to perform CPU computations that could not be handled via MetalFilters or GLFilters. Hence, we needed a performant way to access camera buffers for offline processing. Thus, we decided that each CameraClient would provide a sink handler which would be notified whenever a new camera buffer arrived. On Android, we used Renderscript kernels that processed camera output written to a separate Renderscript surface to generate ByteBuffers from each camera frame. On iOS, this entailed processing respective CMSampleBuffers.
Further, our app developers wanted the ability to overlay custom content on camera’s live or transformed (via MetalPipeline/GLPipeline) feed. This was needed for design assets such as “colors” in Adobe Capture which required color values extracted from camera frame overlaid on the camera feed with a color strip at the top. So, we exposed the ability to provide an overlay component within the CameraClient that the CameraController simply embeds at the topmost level.
Our app developers also wanted customized options that either configure filter pipelines (e.g. ability to invert in “shapes” in Adobe Capture) or perform customized processing. This entailed the ability for each CameraClient to provide its own set of tools with respective delegates. The CameraController would then take care of their placement and notify the delegates as the user interacted with these options.
Using a CameraClient, our app developers were also able to provide entirely customized controls that are simply embedded inside the view hierarchy by the CameraController, thus, providing them complete control over the user interface.
Camera Client Switcher
Finally, Camera Centric design intended for Adobe Capture’s Max’18 release, required the ability to move between different design assets at run time. So, our developers wanted the ability to switch between different configurations. To achieve this, the CameraController exposes the ability to be set up with multiple CameraClients.
In a nutshell, we can now code up newer design assets at a blazing speed by simply creating new implementations of the CameraClient interface without worrying about camera intricacies and hope to provide our users with some more interesting ones soon enough! Stay tuned :)