Crack those CAPTCHAs with opencv and python (part 1)

Have you ever felt like a bot after failing a captcha? Because I have!

Captchas are sometimes so annoying and waste much time. So, I have been thinking about automatically solving a captcha. The first thing that came to mind was to use deep learning. However, I didn’t find a data set that is large enough to train a model such as a CNN, which requires a significant amount of data. So, I decided to use OpenCV with python to solve this problem. Here are some examples of the captchas that we will be solving:

1. PREPARE THE ENVIRONMENT

To be able to follow the steps of this tutorial, you need to have python installed on your machine and also opencv-python. You can install the latter by running the following command:

Also, I will be using cv2 in this tutorial, for more details visit :

OpenCV-Python Tutorials - OpenCV-Python Tutorials 1 documentation

we will also need numpy and matplotlib, so please install them!!!

2. STARTING WITH A SIMPLE EXAMPLE

To be able to crack the the types of captchas I showed you before, I gathered some of them and split each one of them into letter. Thus, for each letter I got a bunch of pictures, here is an example:

Of course, I stored each picture of each letter in a separate file. By doing that, I would be able to use OpenCV feature matching. For example, I can look for the letters that match with its letters with this captcha.

Let’s try to match two variants of the letter ‘a’ with the letters of the previous captcha, the results are shown in figures below:

We looked for a match of the letter in the left, notice there are two matches in this captcha but it only took the nearest one.

In this one, there isn’t a strong match between the letter in the left and the letters of the captcha. Matching at one point isn’t enough to conclude an overall match.

So basically we will be using this technique to look for letters inside a captcha and the letter with the highest match will win.

3. IMPLEMENTATION

It’s time to write some code. First of all, import needed libraries:

OpenCV uses different algorithms to detect edges, curves and critical points of an object. Those are called features. After computing the features of the letter and the objects inside the captcha, the algorithm matches similar ones. For more details about this, visit the documentation page of OpenCV:

Feature Detection and Description - OpenCV-Python Tutorials 1 documentation

For our example, we will use an algorithm called SIFT, and we will look for all matches between a specific letter and a complete captcha, then keep only those who have some matching features more significant than the minimum we fixed. The following code resumes all these steps.

The code might look complicated but this is a very typical code that will be used each time doing this task. With that, you are now able to reproduce all the experimentation we have done in the example.

4. WHAT’S NEXT?

In this first part, we have seen how to match a letter with different captcha letters. The strategy looks simple, but much work is yet to create an effective captcha solver. We need to look for all possible letters, handle the case of rotation and scaling, and handle the case of multiple matches or no matches at all, which would be the next part’s goal.

See you in part 2 where we enhance this and developed it more!