Training Data for App Classifier
Oct 24, 2018 · 2 min read open sourced an AI classifier for Appium at SeleniumConf last week (setup article). This post will show you how to build the classifier yourself, customize them for your application, and add it to Appium.

Before you begin, you will first need to install TensorFlow on your machine. TensorFlow is the open source AI framework from Google.

pip install --upgrade tensorflow

Classify Elements

Checkout the repo ( on github, which will include all the necessary components (images, classifier, building script), and cd into the sample_run directory.

We will run the classifier on a cart:

Run the script:

python --image cart.png

This outputs — Top results: cart — 0.7937941

The 0.7937941 tells you that the AI is 79.38% sure that it is a shopping cart. From our experience, anything above a confidence of 0.3 is usually confident enough.

Build A Classifier

Run the following command from the repo root (cd ..) to start the training:

python --image_dir training_images/ --output_graph output/saved_model.pb --output_labels output/saved_model.pbtxt --how_many_training_steps 4000 --learning_rate 0.30 --testing_percentage 25 --validation_percentage 25 --eval_step_interval 50 --train_batch_size 2000 --test_batch_size -1 --validation_batch_size -1 --bottleneck_dir /tmp/bottleneck

NOTE: The initial training will take 2+ hours on a modern MacBook Pro.

How to Add to the Data Set

If you wish to add a new image that isn’t in the data set, such as a “refund_button”. Create a new folder called “refund_button” under training_images/, and fill it with images of refund buttons, followed by re-running the training.

python --image_dir training_images/ --output_graph output/saved_model.pb --output_labels output/saved_model.pbtxt --how_many_training_steps 4000 --learning_rate 0.30 --testing_percentage 25 --validation_percentage 25 --eval_step_interval 50 --train_batch_size 2000 --test_batch_size -1 --validation_batch_size -1 --bottleneck_dir /tmp/bottleneck

Updating Appium to use your Data

Now that you have an updated version of the classifier, it’s time to add it to Appium.

Convert the classifier to tensorflowjs. Install tensorflowjs.

pip install tensorflowjs

Run the conversion:

tensorflowjs_converter --input_format=tf_frozen_model --output_node_names=final_result  output/saved_model.pb web_model

This creates a web_model folder. Copy the contents of this folder to the npm dependency install for appium-classifier-plugin under the model folder (/usr/local/lib/node_modules/appium/node_modules/test-ai-classifier/model/).

Help Improve the Data Set

We will regularly update the data set and encourage community involvement to improve it. Feel free to send any pull requests with additional images/classes to add to the data set. For more information on how to do a github pull request, check out:

Finally, there is a chance that tests won’t break on every random build…

— Chris Navrides, Engineer at


Mobile app development, test automation, artificial…

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store