Binder Threading Model
Binder threads
In the HamKingServer
class of the HamKing sample project [1], I annotated all the service methods with @BinderThread
:
When an application is started by Android system, a Binder thread pool is allocated for the application to handle incoming Binder transactions. Note that a Binder thread is only used to serve incoming transactions on the server side of a Binder service. When an app initiates the transaction on the proxy side, the app usually fires off and wait for the transaction to finish on the UI thread. In the above example, the four annotated methods run when the client app calls the service methods remotely and they all run on Binder threads. This Binder thread pool frees main thread from caring about serving incoming Binder transactions so that it can focus on UI related works. However, for certain kinds of programs, if all it does is to listen and serve incoming transactions, it may not need a Binder thread pool at all. Because if the main thread spawns a thread pool to handle incoming calls, what should the main thread do? The Service Manager we talked about before is such a program where a Binder thread pool is not needed.
The Binder thread pool for an application is started early during process launching:
The AppRuntime
represents an Android Runtime instance used by an application. The onZygoteInit
is invoked during application launching. First of all, let's take a deep look into the ProcessState
class. The IPCThreadState
class we have seen in "Binder data model" is a class to manage the interactions with Binder driver on a per-thread basis. The ProcessState
instead manages the global Binder state of a process.
The most important task done by a ProcessState
during initialization is to open the Binder driver and use mmap
to create kernel buffer mapping. The gProcess
field is a global singleton of ProcessState
, it only gets initialized the first time ProcessState::self
is called. On line 42, open_driver
is called to open the /dev/binder
device. On line 29, an ioctl
system call is used to check for Binder driver version. Another ioctl
call on line 35 is used to set the maximum number of allowed Binder threads that driver can request user space to spawn. After driver is open, line 54 calls mmap
to setup a kernel buffer mapping for current process. The size of the buffer is about 1 MB.
The startThreadPool
method calls spawnPooledThread
to start a Binder thread. The isMain
indicates whether this thread joins thread pool voluntarily. In this case this is set to true for this reason. Binder driver can ask user space to spawn a new thread to join the thread pool when all threads in the pool are busy. isMain
will be false for these kind of threads. The maximum number of threads that kernel can request is set by BINDER_SET_MAX_THREADS
command in ioctl
. PoolThread
class inherits from the Thread
class in Android system utils library. The way this Thread
class works is that it will run code inside threadLoop
continuously until threadLoop
returns false. The PoolThread
class returns a false so the threadLoop
will only be called called once. But the joinThreadPool
method on IPCThreadState
will never return unless an error happens.
The joinThreadPool
method first uses a BC_ENTER_LOOPER
or BC_REGISTER_LOOPER
command to register itself to Binder driver then run in an infinite loop to process incoming commands unless an error happens. If an error happen then the Binder thread just uses BC_EXIT_LOOPER
to remove itself from the thread pool. We touched on talkWithDriver
in the article "Binder data model", basically this method uses a BINDER_WRITE_READ
command code with ioctl
to write and read data to and from Binder driver. executeCommand
then processes the data that are read from the driver. The BINDER_WRITE_READ
command is blocking, which means it may result the calling Binder thread to sleep and wait if there is currently no work for the Binder thread to do (which means the thread can read no data from driver). So in short, the way a Binder thread works is to write, fetch and process data in an infinite loop and that's why the term LOOPER
is used in the command name.
Now let’s look at the BC_*
and BR_*
command codes. Until now you have seen three levels of command codes. The top level is the command codes to identify the work type for an ioctl
system call. For example, the BINDER_WRITE_READ
, BINDER_SET_MAX_THREADS
and BINDER_VERSION
are several such codes we have seen so far. BINDER_WRITE_READ
command for ioctl
is a general command type to use when a thread wants to read and write data from and to the driver. So BINDER_WRITE_READ
defines a second level of BC_*
and BR_*
commands to indicate the action type of this BINDER_WRITE_READ
. ioctl
knows nothing about these BC_*
and BR_*
commands. Probably the most important BC_*
and BR_*
command type is BC_TRANSACTION
that is used when a process initiates a Binder transaction and BR_TRANSACTION
that is used when a thread processes incoming transactions. A Binder transaction defines a third level of commands to indicate the exact action to perform in the transaction. For example, the IHamKingInterface
Binder service in HamKing project implements three service methods: requestOrder
, pickupOrder
and cancelOrder
. Each method corresponds to a transaction command code. BINDER_WRITE_READ
knows nothing about these transaction command codes.
Now let’s look at the BC_*
and BR_*
command codes for BINDER_WRITE_READ
.
The acronym BC
stands for "Binder Command" which means the data goes from user space to Binder driver. The acronym BR
stands for "Binder Return" which means the data goes from Binder to user space. Let's take a Binder transaction for example, when a process initiates a transaction, source process sends Binder driver a BC_TRANSACTION
with a binder_transaction_data
data structure as it's payload. Binder driver will put a BR_TRANSACTION
command with payload data to the target process's todo list. A Binder thread in the target process will read this BR_TRANSACTION
from driver and process the transaction in user space. After the processing is done, the Binder thread will use BC_REPLY
to send a reply to driver together with reply data. The driver processes this BC_REPLY
by locating the source process of this transaction and put a BR_REPLY
into the source process's todo list. Once the source process receives the BR_REPLY
it knows the transaction is finished. The source process can send a BC_FREE_BUFFER
to request driver to free the kernel buffer used for the reply.
Threading in Binder driver
Going back to the Binder thread registration process. The BC_ENTER_LOOPER
or BC_REGISTER_LOOPER
command will be sent to kernel in the next talkWithDriver
, let's take a look at how kernel handles this command:
Line 17 reads out the command code for this BINDER_WRITE_READ
and adjusts the pointers on line 19 and 40. As we can see, the driver handles the two commands simply by setting a flag on the binder_thread
data structure representing the current thread. The only difference is that for BC_REGISTER_LOOPER
the driver keeps track of the number of threads it requested user space to start and the number of threads that are started in response to such requests. With that being said, a binder_thread
structure will be created to keep track of each thread that interact with Binder driver even if the thread is not a Binder thread. The main thread of a process is usually used to fire off a Binder transaction but the main thread is not a Binder thread. For example, when you use startActivity
to launch an Activity
on UI thread, the UI thread has a corresponding binder_thread
in Binder driver but the UI thread is not a Binder thread. However if a thread wants to read and process incoming data from Binder driver it must first register itself as a Binder thread, otherwise the driver won't consider the thread to schedule transaction works.
The binder_get_thread
function gets the binder_thread
data structure in current process or creates a new one if none exists. Let's first look at the declaration of binder_thread
:
The proc
field points to the binder_proc
structure it belongs to and the rb_node
field is used to link this binder_thread
structure onto the node
tree inside the binder_proc
. The waiting_thread_node
is used to link the structure onto the waiting_threads
list inside the binder_proc
. A Binder thread that is linked to this waiting list indicates that this thread is idle. The pid
field is the Linux process identifier of this thread. The looper
is a bitmap indicating the thread running states. looper_need_return
indicates whether an ioctl
should return to user space instead of waiting for work to do. For example, when a thread uses BC_ENTER_LOOPER
or BC_REGISTER_LOOPER
to register itself as a Binder thread, the calling thread isn't ready to block and wait for work. In this case, the ioctl
returns immediately instead of putting current thread to sleep and wait for work. transaction_stack
points to a stack of binder_transaction
items that the current thread is processing. It is a stack because Binder transactions can be recursive: A thread can fire off another transaction while in the middle of processing an incoming transaction. The wait
field is a Linux wait queue where the calling thread will sleep on when there is no work for the thread to do. The todo
field is a list of binder_work
items for this thread. The binder_proc
also has such a todo list. While scheduling a work item, the driver tries to select a best thread to schedule it onto. If such a best thread cannot be found, the driver will just put the work item onto the binder_proc
's todo list so that a Binder thread in that process can pickup the work item. Besides, if a thread is not a registered Binder thread no work shall be scheduled on to that thread's to do list, because such a thread will never read its todo list to process. Consider this code snippet in HamKing project:
You shouldn’t ever need to call linkToDeath
on a remote service if it is acquired through bindService
since you will get a callback in onServiceDisconnected
anyway when remote process dies. However we will just add a death listener for demonstration. After you add an order, kill the server app process and you will see a line of log like:
E/MainActivity: linkToDeath called on Binder:25869_3
The UI thread of the client app registers a death listener to the Binder driver. On initial thought it makes sense for Binder driver to deliver the death notification to the UI thread’s todo
, but the Binder driver cannot do that since the UI thread is not a Binder thread. Instead, the work item for the death notification is put on the todo
list of the client app process. A Binder thread inside the client app later picks up the work and handles the death notification.
We have looked into the binder_thread
data structure, let's now look inside binder_get_thread
function:
The binder_get_thread
function first tries to find a binder_thread
data structure in current process that has a match pid
. If such a binder_thread
doesn't exist, then create a binder_thread
structure and link it to the binder_proc
's tree.
The Binder driver may request user space to spawn new threads to join the Binder thread pool:
Before binder_thread_read
finishes, driver checks whether it should request user space to spawn a new thread to join the Binder thread pool. If so driver will put a BR_SPAWN_LOOPER
command in current thread's read buffer to let user space process it. Line 8 checks whether there is no idle Binder thread left in current process. Line 9 checks whether the number of requested Binder thread has reached the limit set by user space. After the ioctl
is returned, user space will handle it by spawning a Binder thread and join the thread pool:
We have seen the spawnPooledThread
earlier in this article. Note that the isMain
argument is set to false this time which means this Binder thread is created in response to a driver request. IPCThreadState
will use BC_REGISTER_LOOPER
command code to register the Binder thread instead.
It is important to understand that the action of writing data into the driver is not blocking. Which means the write action either succeeds or fails but will not put the calling thread to sleep. On the other hand, reading data from Binder driver is blocking. If there is no data for the current thread to do then the calling thread will be put to sleep. It is important that binder_ioctl_write_read
processes data write earlier than read, otherwise the system may dead lock:
Since Android framework doesn’t set the O_NONBLOCK
flag when opening /dev/binder
, the non_block
argument will always be false. binder_available_for_proc_work_ilocked
checks whether current thread is available to handle work items on binder_proc
's work list. If a thread is a registered Binder thread and not have work to do then this function returns true. binder_has_work_ilocked
checks whether there is work to do for current thread. The binder_wait_for_work
is where the current thread might be put to sleep. These steps are just typical steps to put a thread to sleep in Linux kernel. If there is no work for current thread to do then the current thread will be added to the wait queue in the binder_thread
structure and the binder_proc
's waiting thread list. When line 74 is called, a context switch will be called and the current thread will be put to sleep.
While an Android app is running, most of the Binder threads are just sleeping to wait for work to do. For example, this result is typical when you check the Binder threads state of an application:
The S
in the "S" column means the thread is sleeping and the binder_thread_read
means the thread is sleeping in binder_thread_read
function.
When a work item is added to a target thread’s todo list, the driver will wake up the target thread and then the target thread can continue reading data into the read buffer. For example, binder_proc_transaction
will be called during a transaction when Binder driver has found a target thread to perform the transaction:
The t
parameter in binder_proc_transaction
represents the current transaction to schedule. The proc
parameter represents the target process to schedule on and thread
parameter represents the target thread to schedule on. An empty thread
parameter means there is no best thread to schedule the transaction on, so binder_select_thread_ilocked
is called to select one thread to schedule the transaction on. The function just selects the first Binder thread that is idle. Line 13 to 16 adds this transaction item to the thread's todo list or the process's todo list. Then binder_wakeup_thread_ilocked
will be called to wake up the selected thread. After the thread is waken up, it can continue to run inside binder_thread_read
and pickup the transaction item in the thread's todo list or the process's todo list, write it the read buffer and return to user space.
Binder work items
During this article I keep using the phrase “work item”. Of course that term indicates some job for a thread or process to do. Let’s see how a work item is abstracted in Binder driver. The work item is represented by a binder_work
data structure in Binder driver:
The binder_work
structure is surprisingly simple. The entry
field is used to enqueue this work item onto a binder_thread
's todo list or a binder_proc
's todo list. Besides that there is only a type
field to represent the work type. For example, the BINDER_WORK_TRANSACTION
means the work item is a Binder transaction and BINDER_WORK_DEAD_BINDER
means the work item is a death notification. Since this structure is so simple, it cannot be the full story. The way this binder_work
works is to be embedded in other data structures:
Note that the binder_work
structure is just embedded in another data structure which mean the work
fields are not pointer types. To view it in an object oriented way, this means the four data structures inherit from binder_work
so they gain the capability of being scheduled onto a todo list. For example, during a transaction the driver will create a binder_transaction
and link it the target todo list. When the target thread processes its todo list, it will check the type of the binder_work
and "cast" the binder_work
to the container binder_transaction
. The actual code is simple:
Line 14 to 18 selects the target list and line 20 dequeue a binder_work
from the list. Line 21 checks the type of the work item then line 23 "casts" the binder_work
to binder_transaction
.
Binder polling
The benefit of using Binder thread pool is to free the main thread from serving incoming transactions. However if the main business a program does is just serving incoming transaction, then it doesn’t make sense to create a thread pool. The Service Manager is an example of this kind of program. The single core functionality of Service Manager is to serve other process’s requests of adding and fetching system services. It is a single threaded program where all Binder transactions are served on the main thread.
Note that on line 41 the size of the thread pool is set to zero and the main thread is responsible for serving all incoming Binder transactions. A Binder thread sleeps inside binder_thread_read
until a work item is enqueue onto its todo list. However Service Manager cannot act that way since there they other jobs the main thread needs to do, otherwise the main thread won't have a chance to do other jobs. Binder polling provides a mechanism for a process to receive a callback when there is something for it to read, instead of letting the thread sleep waiting for data. The Looper
is Android's solution of event notification and it is widely used in various places like message queue and input system. It's implemented with the help of the Linux epoll
system. Don't confuse it with the term LOOPER
used in BC_*
commands though. Looper
deserves its own article so we won't deep dive into it here. Basically you can add some file descriptors to a Looper
and receive a callback when something on the file descriptor happens. For example, line 10 to 14 says that when /dev/binder
has some data for me to read, invoke the callback cb
.
Under the hood, Looper
uses epoll
to poll on the state of /dev/binder
and the exact behavior is implement by Binder driver:
After the pollAll
method is called on line 49, Looper
will call into the kernel with epoll
and hit binder_poll
. Line 16 checks whether there are work for current thread to read. If there are, then EPOLLIN
will be returned and eventually it causes the handleEvent
function in BinderCallback
to be invoked. Otherwise 0 is returned indicating the device is not readable for calling thread and the calling thread may go to sleep. Yes, Looper
will still put the current thread to sleep, but the key is that you can observe multiple event sources not just the /dev/binder
readable event from the main thread. Line 14 adds the binder_thread
's wait queue to a poll table so that when there is work item for this thread to do the thread can be waken up and poll the Binder device again.
Let’s take a look at the setupTo
method in BinderCallback
. The setupPolling
method is called to prepare for polling on /dev/binder
:
Basically it prepares polling by registering the main thread to Binder driver. Otherwise the driver will never enqueue work the binder_thread
data structure of the main thread and the thread will never be waken up when polling. Even though the main thread also uses BC_ENTER_LOOPER
to register itself like the threads in Binder thread pool, how they interact with Binder driver is very different. At last, let's look at the handleEvent
method in BinderCallback
:
We have seen the getAndExecuteCommand
method before. This method just reads data from Binder driver and process it until all data in the input buffer is processed.