A Complete Guide to Accessibility Service Part 2 — Android

In the first part of the series, we have discussed about accessibility service what is it, how to create and allow accessibility service for an application. If you haven’t gone through that series I would suggest going through that before starting this.

In this series, we will be creating a class that extends AccessibilityService. Before jumping into the actual code, first, we should know what are the functions which we need to override from these AccessibilityService classes.

onServiceConnected

onAccessibilityEvent(AccessibilityEvent event)

OnServiceConnected:

This function is called when the user switches on the Accessibility Service for our app from the phone settings. Also in this function, we specify the type of events that can be handled by accessibility service and also other information that was done through @xml/serviceconfig file in the previous article. We do this by creating an object of type AccessibilityServiceInfo. Here is a sample code to demonstrate how we perform this operation.

@Override
protected void onServiceConnected() {
        System.out.println("onServiceConnected");
        AccessibilityServiceInfo info = new AccessibilityServiceInfo();
        info.eventTypes = AccessibilityEvent.TYPE_NOTIFICATION_STATE_CHANGED;
        info.eventTypes=AccessibilityEvent.TYPES_ALL_MASK;
        info.feedbackType = AccessibilityServiceInfo.FEEDBACK_ALL_MASK;
        info.notificationTimeout = 100;
        info.packageNames = null;
        setServiceInfo(info);
}

Let’s understand one by one.

First, We create an object of the type AccessibilityServiceInfo.

Second, On this object we call different static variable to suit our purpose. Like we specify the type of events our service will handle the packege name it will work upon etc.

Third, We pass this info object into the function to setServiceInfo which registers our service with the specified attributes.

All the attributes given above are already discusses in the previous article(under the heading How to build Accessibility Service?). I’ll suggest you to go through it to have more insights about the present context.

onAccessibilityEvent(AccessibilityEvent event)

This function is called whenever there is an event on the screen. These are the events which we mention under @xml/serviceconfig(in the previous article) or the ones we mention in the onServiceConnected Function.

What are the types of events that occur on the screen?

The events that occur on screen are: TYPE_VIEW_LONG_CLICKED, TYPE_VIEW_SCROLLED, TYPE_VIEW_SELECTED, TYPE_VIEW_TEXT_CHANGED, TYPE_WINDOWS_CHANGED, TYPE_WINDOW_CONTENT_CHANGED, TYPE_WINDOW_STATE_CHANGED, TYPE_VIEW_CLICKED, TYPE_NOTIFICATION_STATE_CHANGED and as such many more events.

How do we read these events from our onAccessibilityEvent?

Well the parameter of AccessibilityEvent type passed to the function onAccessibilityEvent contains information about the AccessibilityEvent. We can retrieve this information using the function getEventType which returns type of events occurring. Here is a sample code to displays how you can read the notifications.

if (event.getEventType() == AccessibilityEvent.TYPE_NOTIFICATION_STATE_CHANGED) {
    if (event.getPackageName().toString().equals("com.whatsapp")){
        StringBuilder message = new StringBuilder();
        if (!event.getText().isEmpty()) {
            for (CharSequence subText : event.getText()) {
                message.append(subText);
            }
            if (message.toString().contains("Message from")){
                name=message.toString().substring(13);
            }
        }
    }
}

Let’s understand this code here the block under if is executed when the event is of the type TYPE_NOTIFICATION_STATE_CHANGED. event.packageName() returns the name of the package for which the notification is targeted. In this code it is WhatsApp. The variable message contains the text returned by the notification message(for example in case of WhatsApp message the variable message will contain “Message from <sender’s name in you contacts>”. And the variable name will contain the sender’s name.

What are the type of actions that we can perform user’s behalf using AccessibilityService?

The event that we can perform on user’s behalf include ACTION_SET_TEXT, ACTION_CLICK, ACTION_LONG_CLICK, ACTION_CUT, ACTION_COPY, ACTION_SCROLL_FORWARD, ACTION_SCROLL_BACKWARD, ACTION_SELECT.

How do we perform these actions using AccessibilityService?

Before understanding this we need to understand how do we parse through the xml layout. For these we need to understand the concept of nodes. All XML UI’s contains a hierarchy of views and layouts. where the basic Layout can be considered as a node and all the sub-views or sub-layouts can be considered as it’s child’s. Similarly each of these sub-views or sub-layouts in turn contain there own child’s this creates a hierarchy which can be used to parse the XML.

How does Accessibility returns these XML elements?

For accessing the XML we need to have the parent node which represents the layout of the current window. This can be stored in an AccessibilityNodeInfo Object. Using, the following code:

AccessibilityNodeInfo currentNode=getRootInActiveWindow();

Now this currentNode object contains the parent layout and we can get it’s child using the function getChild(<index>). Several operations that can be performed on this currentNode object or it’s childs are: getChildCount(), getClassName(), getContentDescription(), getText(), getParent(), getInputType(). I would suggest you to go through the list of these operations from the documentation.

Thus by now you have achieved our first milestone, how to read the layout. Congratulate yourself!!

Great job!!

Now we are heading towards our next milestone.

How to perform actions on users behalf?

Now since we know that the object currentNode holds reference to our views and layouts. We can use this object to perform actions and for doing this we call the function performAction(<type of action>) on the current node object. Here is a sample code to demonstrate the working.

if (event.getEventType()==AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED){
    if (source.getPackageName().equals("com.whatsapp")) {
            AccessibilityNodeInfo currentNode=getRootInActiveWindow();
            if (currentNode!=null && currentNode.getClassName().equals("android.widget.FrameLayout") && currentNode.getChild(2)!=null && currentNode.getChild(2).getClassName().equals("android.widget.TextView") && currentNode.getChild(2).getContentDescription().equals("Search")) {
                currentNode.getChild(2).performAction(AccessibilityNodeInfo.ACTION_CLICK);
        }
    }
}

Let’s understand this code.

Whenever you will open WhatsApp, onAccessibiliyEvent function will give an event of type AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED and the package name as “com.whatsapp”. Next we get reference to the current window using

AccessibilityNodeInfo currentNode=getRootInActiveWindow();

Next we parse throught this layout if currentNode has a frameLayout and it contains a child at index=3. Then we move further to check whether the child at index=2 is a text view with content description “Search” then at this text view(i.e on currentNode.getChild(2) ) we call the function performAction. With the type of operation as an argument. This will perform a click on that TextView.

(This is just a code snippet taken out of my project. I encourage you to try this child parsing yourself. And to perform actions on view and layouts.)

So this was the basic tutorial on understanding and implementing AccessibilityService. If you have learned how to parse the nodes and perform actions then you have definitely mastered AccessibilityServices.

Yes you have completed it.Give yourself a pat!!

Hope you have enjoyed reading the article and grabbed a lot from this. If you did, then please don’t forget to give claps on this articles.

Happy Coding!!