A Random Encounter With VideoToolbox

Yesterday I was just doing some general work stuff on my MacBook when suddenly the CPU fan starts to go nuts. Taking a look the process in question was VTDecoderXPCService. Doing some quick googling wasn’t getting me anywhere so I decided to take the fun route: hacking.

nm is fun

First thing I did was check the process info in activity monitor. Unfortunately the process was called by launchd so I wasn’t able to figure out what it might have been interacting with. Instead I pulled the path up into a sudo-root terminal and did an nm dump on it. nm is a command line utility to “display name list (symbol table)”. Results are as follows:

# cd /System/Library/Frameworks/VideoToolbox.framework/XPCServices/VTDecoderXPCService.xpc/Contents/MacOS
# nm -a VTDecoderXPCService
U _VTRemoteDecompressionServer_ServiceEventHandler
0000000100000000 T __mh_execute_header
U _xpc_service_main
U dyld_stub_binder
0000000005614542–00 0000 OPT radr://5614542

Okay so this seems to be a pretty small app. So apparently this is an XPC service loop. Which of course brings up the next question.

What the hell is an XPC service

So I finally did some research through google and came to the osx documentation on XPC services. The benefits were explained here:

Let’s face it; applications sometimes crash. We don’t want it to happen, but it does anyway. Often, certain parts of an application are more prone to crashes than others. For example, the stability of any application with a plug-in API is inherently at the mercy of the authors of plug-ins.
When one part of an application is more at risk for crashes, it can be useful to separate out the potentially unstable functionality from the core of the application. This separation lets you ensure that that, if it crashes, the rest of the application is not affected.
Privilege Separation:
Modern applications increasingly rely on untrusted data, such as web pages, files sent by email, and so on. This represents a growing attack vector for viruses and other malware.
With traditional applications, if an application becomes compromised through a buffer overflow or other security vulnerability, the attacker gains the ability to do anything that the user can do. To mitigate this risk, Mac OS X provides sandboxing — limiting what types of operations a process can perform.
In a sandboxed environment, you can further increase security with privilege separation — dividing an application into smaller pieces that are responsible for a part of the application’s behavior. This allows each piece to have a more restrictive sandbox than the application as a whole would require.
Other mechanisms for dividing an application into smaller parts, such as NSTask and posix_spawn, do not let you put each part of the application in its own sandbox, so it is not possible to use them to implement privilege separation. Each XPC service has its own sandbox, so XPC services can make it easier to implement proper privilege separation.
For more information about sandboxing, see App Sandbox Design Guide.

Okay interesting, a bit of sandboxing and privilege separation. Seems like an interesting security feature. The main caveat of that is that you only get one unique XPS instance given the type of service. According to the man page this can be one of:

     •   Application
Each application will have a unique instance of this
     •   User
There is one instance of the service process created for
each user.
     •   System
There is one instance of the service process for the whole
system. System XPC Services are restricted to reside in
system frameworks and must be owned by root.

Looking over the plist file for the VTDecoder XPC Service:


So the decoder XPC service would be provided one unique service per application. Now what about this sandboxing?

XPC service sandbox

So next thing I check is what is supposed to constitute the sandbox. Under the package contents I come across a .sb file:

(version 1)
(deny default)
(import “system.sb”)
(import “com.apple.CodeSigningHelper.sb”)
(allow file*
(literal “/dev/dtracehelper”))
(allow file-read* file-write*
(regex #”^/private/var/db/auth\.db.*$”)
(subpath “/private/var/folders”))
(allow file-read*
(literal (string-append (param “XPCService_home”) “/Library/Preferences/com.apple.CoreGraphics.plist”))
(literal (string-append (param “XPCService_home”) “/Library/Preferences/com.apple.coremedia.plist”))
(literal (string-append (param “XPCService_home”) “/Library/Preferences/com.apple.corevideo.plist”))
(subpath “/System”)
(subpath “/usr/share”)
(subpath “/Library/Video/Plug-Ins”)
(subpath “/Library/Video/Professional Video Workflow Plug-Ins”))
(allow file-read-metadata)
(allow mach-lookup
(global-name “com.apple.CoreServices.coreservicesd”)
(global-name “com.apple.windowserver.active”)
(global-name “com.apple.SecurityServer”))
(allow ipc-posix-shm
(ipc-posix-name “apple.shm.notification_center”)
(ipc-posix-name “com.apple.AppleDatabaseChanged”))

Wait is that… LISP!? Okay given how OSX usually does things I was totally expecting XML to be honest. Anyways the sandbox restricted locations seem fairly straightforward for a video decoder. Now that I know this much I decided to check the process list a bit.

The processes

I found that there were already existing VTDecoderXPCServices:

Due to the fact that the parent process is launchd it’s hard to really tell what is precisely using the decoder services. As a general hunch I closed the TweetDeck tab in Twitter and one of the processes went away. Playing an HTML5 video on YouTube made the CPU start to rise in the topmost process. Flash video with hardware acceleration enabled doesn’t seem to have any effect on the processes, so I’m under the assumption it doesn’t use that (which considering the security fun of Flash I would think it would want to, but then again XPC services are one per app so that might be why).

I also took a nice sample of one of the processes as it was using CPU:

This looks to be where the decoded frame is being packaged up for the end client. Satisfied that I had more knowledge of this weird process than before I decided to call it a day. Would still be nice to figure out what is launchd calling each of these XPC services though.

If you’re interested in the VideoToolbox API I recommend taking a look at this 2014 WDC talk on the matter. Anyways, hope you enjoyed my screwing around for knowledge!