Unusual ways of using Android accessibility services

Maksym Dybarskyi
Apr 21, 2017 · 5 min read
Image for post
Image for post

The accessibility service is a feature of the Android framework designed to provide user interface enhancements to assist users with disabilities, or those who may temporarily be unable to fully interact with a device. In these cases, people might need additional or alternative feedback such as text-to-speech or haptic feedback. The accessibility services run in the background and receive callbacks from the system when accessibility events fired. Such events denote some state transition in the user interface, for example, the focus has changed, a button has been clicked, etc. Such the service can optionally request the capability for querying the content of the active window.

The accessibility services receive events from the system when UI of the active application is changed or user input appeared. Also they may retrieve content of the active window and perform actions on UI components.

What can we do by accessibility services?

Let’s check what the accessibility service is able to do with user interface.

The service may react to a bunch of events. The AccessibilityEvent class represents a system event and contains contextual information about what happened in the user interface, including event type, source of event, event time, etc. For example, there are several very useful event types such as TYPE_VIEW_CLICKED, TYPE_WINDOW_STATE_CHANGED, TYPE_VIEW_TEXT_SELECTION_CHANGED and others. You can read more here.

The accessibility service configuration allows us to filter events by type and by application package. In other words, it is possible to receive all accessibility events from all applications or some specific events from specific applications.

As far as interacting with the UI is concerned, you can interact with user interface in different ways. The accessibility service doesn’t use View class directly, but AccessibilityNodeInfo instead. This class allows you to click views (using ACTION_CLICK or ACTION_LONG_CLICK actions), scroll views (using ACTION_SCROLL_FORWARD and ACTION_SCROLL_BACKWARD actions) or make text selection operations (using actions like ACTION_SET_SELECTION, ACTION_COPY, ACTION_CUT, etc.). The whole list of actions is available in javadoc of the class.

Also it is possible to get view-related information from AccessibilityNodeInfo such as isScrollable, isClickable, etc. Considering ability to retrieve then window content and performing gestures, the accessibility services seem to be quite a powerful tool.

What can we use them for?

As I mentioned, the accessibility service is designed to assist users with disabilities by providing additional feedback. But possible usage is not limited by these user interface enhancements. The accessibility service may be developed as part of a bigger application and used to communicate with other components like services or activities. Hence, we can use them for our own needs. Let me take a couple of examples.

One of my previous projects was quite specific — an android application was part of big testing infrastructure developed for testing bluetooth connectivity of embedded software. The aim of the application was emulating user actions such as accept bluetooth pairing request, reject bluetooth pairing request, allow remote bluetooth device to access phonebook/messages, and so on. For these purposes, I used the accessibility service to perform actions with system dialogs.

The second example of using accessibility services (you can find several suggestions at Stackoverflow) is obtaining response of USSD code by retrieving content of system dialog. These codes, also known as “quick codes”, are usually used for communicating with GSM operator. Often they look like *100#, *123#, etc. Here is a demo application I wrote for this post. Below I will describe development details.

I created a list of possible usage scenarios:

1. Automating/emulating user actions (may be used for testing purposes)

  • Accepting/rejection runtime permissions

2. Reacting to system events if there is no broadcast event for them

  • Receiving USSD code result

3. Retrieving information from UI elements of other application (including systems)

  • GoogleNow-on-tap like functionality (analysing textviews content of any app and making a google search)

4. Dispatching gestures

  • Controlling other application by performing gestures (kind of user action emulation)

5. Soft keyboard state listening and controlling

  • Gathering key pressing statistics

As you can see, there are several ways of unusual usage of accessibility services. I wrote ‘unusual’ because all these scenarios are different from the original purpose of accessibility services. They are a bit tricky, but I believe we can use widely our own services as part of user interaction testing (together with espresso and uiautomator).

Any restrictions?

The accessibility service seems to be kind of security issue. But they are not. Firstly, because user must manually activate an accessibility service in the corresponding settings screen. Secondly, the services run in user space, not system space and not middleware. So, simple adb shell pm dump com.yourapp.package command will show all information about accessibility service, even classname. Moreover, to get content of some textview you should know its package, id and other information (the same as using uiautomator).

Show me an example

I won’t go into details of how to develop and configure accessibility services in general. You can find a good guide at android development documentation website (one, two). Instead, I will focus on approaches from the list above.

The demo application allows user to add USSD code to the list, run it by clicking on list item and reach USSD code response, dismiss dialog and show response content within application. The application may be downloaded from play store and source code is available at github.

This video shows how the application works:

The accessibility service must be able to do the following:

  • Receive accessibility event when the dialog with USSD code response appears;

So, firstly, we have to configure the accessibility service to receive events from the application which performs USSD codes (basically this is the application we use to make calls). Also we need to receive only events that represent the opening of dialogs. In order of do that we have to know package of phone application and corresponding accessibility event type. Second, to reach USSD response text we need to know id of dialog textview where response is shown. And finally we need to know id of OK button to emulate click to dismiss the dialog.

There is uiautomatorviewerpackage of application and id of textview and button. Here is a good tutorial on how to use it.

The configuration file of accessibility service looks like this:

There are a couple of important lines in this configuration:

  • The android:packageNames="com.android.phone" line tells the system that we are interested only in events from application with such package. On my Nexus 6 device this is Phone application made by Google. Other manufacturers (like Samsung, LG, etc) may change package of phone application.

Now, let’s implement the accessibility service itself. Below is a code snippet which contains overridden onAccessibilityEvent method of AccessibilityService class. It shows how to get text from textview and how to perform actions on button.

Please, let me know if you use accessibility services in your apps and for what purpose. Also, add comments if you find another unusual scenarios of accessibility service usage.

Tooploox

We're on a quest to build great teams and amazing products.

Maksym Dybarskyi

Written by

http://dybarsky.info

Tooploox

Tooploox

We're on a quest to build great teams and amazing products. Here we share our thoughts about web and mobile development, data science, IoT, product design and startups.

Maksym Dybarskyi

Written by

http://dybarsky.info

Tooploox

Tooploox

We're on a quest to build great teams and amazing products. Here we share our thoughts about web and mobile development, data science, IoT, product design and startups.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store