Solving CAPTCHA with TensorFlow and Go
A robot that proves it’s not a robot way better than I do
This article was first published at GopherAcademy. The Medium version has been edited and slightly restructured.
Before the Blacklight event at DevFest Siberia, I was tasked with play-testing a specific challenge to prove that it’s possible to crack in a reasonable time. That challenge was one I face occasionally in my everyday life, not always with success: solving a bunch of captchas.
At DevFest, people had to break into a room without surveillance cameras capturing the break-in attempt. Disabling the camera required entering a four-digit security PIN into a CAPTCHA-protected form.
Provided were a TensorFlow SavedModel in the binary ProtoBuf format, trained to recognize that particular captcha (tensorflow-savedmodel-captcha.zip, 27.7 MB), and a link to the camera interface.
An input of a TensorFlow model requires doing some TensorFlow!
A Few Words about TensorFlow
TensorFlow is an open-source software library for Machine Intelligence, used mainly for machine learning applications such as neural networks.
TensorFlow runs computations involving tensors. There are many sources to understand what a Tensor is, and this article is definitely not a sufficient one: it only has the bare minimum to make sense of what the code does. Tensors are awesome and complex mathematical objects, and I encourage you to take the time to learn more about them.
For our purposes, here is the explanation from the TensorFlow website:
A tensor is a generalization of vectors and matrices to potentially higher dimensions. Internally, TensorFlow represents tensors as n-dimensional arrays of base datatypes.
A tensor is defined by the data type of the values it holds and its shape, which is the number of dimensions and number of values per dimension.
The “Flow” part in TensorFlow comes to describe that essentially the graph (model) is a set of nodes (operations), and the data (tensors) “flows” through those nodes, undergoing mathematical manipulation. You can look at, and evaluate, any node of the graph.
A Few Words about TensorFlow + Go
On the official TensorFlow website, you can find a page dedicated to Go describing the recommended use:
TensorFlow provides APIs for use in Go programs. These APIs are particularly well-suited to loading models created in Python and executing them within a Go application.
It also warns that the TensorFlow Go API is not covered by the TensorFlow API stability guarantees, although to the date of this post, everything is still working as expected.
When going to the package page, there are two more warnings:
- The API defined in this package is not stable and can change without notice.
- The package path is awkward and can change in the future as well:
github.com/tensorflow/tensorflow/tensorflow/go.
In theory, the Go APIs for TensorFlow are powerful enough to do anything you can do from the Python APIs, including training (Asim Shankar has a good example of training a model in Go using a graph written in Python). In practice, some of workflows, particularly those for model construction, are very low level and certainly not as convenient as in Python.
For now, it generally makes sense to define the model in TensorFlow for Python, export it, and then use the Go APIs for inference or training. So while Go might not be your first choice for working with TensorFlow, they do play nice together when using existing models.
Note: thanks to Asim Ahankar from the TensorFlow team for pointing out it is possible to train models with Go. We will collaborate further to make the documentation around this more accessible.
Let’s Break In
The interface I was facing seemed pretty close to your regular captcha-protected form. My TO DOs were:
- Inspect the model.
- Start with a boilerplate.
- Load the model.
- Fetch the captcha.
- Predict the captcha.
- Try to log in with the predicted captcha and a given PIN code.
- Iterate over all possible PIN codes.
1. Inspect
SavedModel is the universal serialization format for TensorFlow models.
— TensorFlow documentation
We don’t know anything about the model, so our first step would be figuring out the input and output nodes for prediction. The SavedModel format allows storing this information, called a signature, as metadata; the tool for inspecting signatures is SavedModel CLI. Here’s the command and its output:
$ saved_model_cli show --dir ./tensorflow_savedmodel_captcha --allMetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:signature_def['serving_default']:The given SavedModel SignatureDef contains the following input(s):
inputs['input'] tensor_info:
dtype: DT_STRING
shape: unknown_rank
name: CAPTCHA/input_image_as_bytes:0The given SavedModel SignatureDef contains the following output(s):
outputs['output'] tensor_info:
dtype: DT_STRING
shape: unknown_rank
name: CAPTCHA/prediction:0Method name is: tensorflow/serving/predict
What we learn from this are the node names used with the tag serve:
- input:
CAPTCHA/input_image_as_bytes, - output:
CAPTCHA/prediction.
2. Start
Let’s define constants for the camera URL and for the error messages we already know:
We’ll add a log file support to our main() function to make sure everything is logged properly:
And implement a logResponse function for our login attempts. If the response is unknown, we’ll output it in full. Otherwise we’ll add a short entry to either the logfile or stdout.
3. Load
Now let’s load the model with LoadSavedModel. The full signature is:
func LoadSavedModel(exportDir string, tags []string, options *SessionOptions) (*SavedModel, error)The function takes 3 arguments: path, tags and session options. Explaining tags and options can easily take the entire post and will shift the focus, so for now you should know that serve is the tag used to serve TensorFlow models, and session options are not required in our case.
4. Fetch
Now let’s get the captcha image.
In order to keep the session with the generated captcha, we will first open a cookie jar. Even though it’s the first time I am writing about cookies publicly, I will spare cookie jokes as part of the Christmas spirit.
5. Predict
To run a prediction we need to supply inputs, called feeds (operations to feed our data to, mapped to tensors containing the data), and outputs, called fetches (operations to fetch the data from).
We only have one feed (input):
- the feed operation is
CAPTCHA/input_image_as_bytes, - the feed tensor is a string containing the CAPTCHA image as bytes.
Likewise, there is only one fetch (output): CAPTCHA/prediction.
After we run the model with our feeds and fetches, we receive the output — the captcha prediction.
6. Log In
Once the values — the PIN code and the captcha prediction — are there, let’s POST the request to disable the camera.
If the captcha prediction failed, we retry with the same PIN code.
The parseResponse function checks and reports whether the response is one of the known error messages, which I’ve found out manually by guessing PIN codes and making both correct and incorrect captcha predictions.
7. Iterate
The PIN code only has 4 digits, so we’ll just go over all the combinations. Additionally, in each iteration the loaded model is required for the prediction operation, and of course some logs.
To Wrap This Up
The full code after everything is composed together is available on GitHub: Pisush/break-captcha-tensorflow.
TensorFlow has many models which can all be used with Go. There is a great list of those at tensorflow/models.
The pre-trained model provided for the challenge is based on emedvedev/attention-ocr.
Online challenges can be an awesome way to learn, whether it’s coding, security or sports. The combination of putting in practice your knowledge and having a mission creates a fun environment where you can work on improving your skills. Consider joining such a challenge as your new year’s resolution.
Thanks a lot to Ed for reviewing this PR and to Asim Shankar from the TensorFlow team for his input on the TensorFlow + Go part of this article.
If you want to chat more about this, tweet me, or meet me at Gophercon Iceland!
Even more stories about solving captchas, picking locks and investigating conspiracies: @blacklightai on Twitter.
