App for Automatic Number Plate Recognition (ANPR)

Image credit: https://www.123rf.com/

I got motivated by success stories of brilliant AI enabled mobile apps sometimes developed by kids under the age of 15 appearing in news and adverts every now and then. Told me: “I should build one — it isn’t as easy as it sounds”. Picked up Automatic Number Plate Recognition as a use case. Over 2 weeks of part-time research and development (deployment not included) for the pet project I didn’t come across end-to-end AI app examples (not saying I was thorough with my search). Hence sharing my experience with the planning and coding steps that may help fellow aspirants and can possibly bring down the average age of developers 😊.

If you prefer to try the app than reading further, feel free to delve right into the code.

Planning the app

Frontend

(1) What should it do?

• Select an image by accessing camera or photo roll
• Post image to backend API
• Receive and display the JSON response
• Run on iOS and Android

(2) How should it look?

Well, something like this:

Low fidelity wireframe

(3) Which language/framework to use?

Chose React Native as it is possibly the most popular multiplatform app development framework per git surveys. Run-time and libraries will be covered in the coding section (as I didn’t plan for it until I was coding).

Backend

(1) What should it do?

• Expose an API to post an image
• Detect number plate within the image
• Convert image to text
• Return the text as JSON

(2) Which language/framework to use?

Python is my de-facto choice for Machine Learning. But need to drill-down further.

• For API enablement, chose Flask.
• Number plate detection — I was happy to do some basic image processing with OpenCV (instead of looking for Object Detection models). The results may not be accurate but it’s a start.
• Converting image to text — It is an Optical Character Recognition (OCR) problem within Machine Learning space. I didn’t want to build/train a massive Neural Net model of my own. Not only because I am lazy but to operate within the constraints of my laptop which is slow to say the least. Plus, I wanted to use Tesseract-OCR engine (and its corresponding python implementation pytesseract) which is quite popular.

With the plan above, I have eliminated the need of any Machine Learning pipeline for build/train/test/deploy models or large image datasets (some will be required to test the app). It’s time to get coding.

Coding the app

Though the code pre-requisites and dependencies are mentioned, I won’t cover how to setup the local environment for development. There should be plenty of resources available online if you need help with setup. And if you still struggle to run the project then drop a line in comments and I will do my best to respond.

Backend

As my use-case is backend heavy, let’s start with it. First order of business is to look for how others have tackled ANPR to get ‘inspired’. That’s when I stumbled upon this PyImage post that has the solution for detecting the number plate in the image and converting to text within my planned parameters — bingo! I recommend going through the post if you have difficulty to follow my explanation (it’s more detailed than mine).

Runtime and dependencies:

• I used Python 3.6
• For dependencies refer to requirements.txt

Note: Tesseract binary needs to be installed before pip install pytesseract.

Code structure:

Didn’t care much, so ended up with:

VehicleIdentifier
|_ anpr.py
|_ test
    |_ test.py
    |_ *.png
|_ images

where:

• anpr.py is the backend API with resource /anpr to support posts
• test.py is the is a test script to confirm backend works
• *.png are a bunch of images to test with (supports jpeg and gif as well)
• images directory is reserved for future use

Code flow

I refer back to my plan to go step by step.

(1) Expose an API to post an image involves

• Initiate a Flask app
• Run the embedded development server (host is set to my IP so that it is accessible to devices in same network which will come handy while testing)
• Define the endpoint
• Accept and open image file (if I uncomment line #20 we have a functional endpoint that returns image size)
• Convert image to numpy array and resize to a standard

Above gets me ready for the image processing tasks at hand.

(2) Detecting number plate within the image involves

• Setting up some constants and a debug function — the fundamental assumption is license plates have an aspect ratio between 4 & 5.

• Function locate_license_plate_candidates perform a set of operations on the image to identify potential number plate candidates — fundamental assumption being license plates have dark characters in light background.

• Identify most likely contour containing the license plate based on the defined aspect ratio earlier.

Note:

a) Both locate_license_plate_candidates and locate_license_plate functions expect images in gray scale and I haven’t transformed the input image (which may be in RGB).

b) I haven’t plugged in the above functions in the main flow (process_image function) yet.

I do these in the next steps.

(3) Convert image to text involves:

• Tesseract parameters setup— contents of number plate is considered as a single word.

• Convert the input image to gray scale, invoke locate_license_plate_candidates and locate_license_plate functions from step #2 and then perform OCR.

(4) Return text as JSON involves:

• Clean up function to filter out any special characters in the output

• Plug find_and_ocr function to main flow (process_image function) and return output (below snippet gets added after line #27 from step #1)

At this point I have the back-end coded.

Testing

I use this simple test.py script to test the endpoint stand-alone as front-end is not ready (could have used a tool like Postman instead).

And the output is:

{'vechileId': 'KL5SR2473'} 200

Now, we are talking!

Frontend

Let me turn the focus on the experience components.

Runtime and dependencies:

• Used Node runtime (Yarn should work too)
• Initiated a blank Expo app
• For dependencies, refer to package.json

Code structure:

For brevity, I will cover my code files — rest is setup by Expo.

vehicle-identifier
|_ anpr-mapp
    |_ App.js
    |_ components
        |_ ImageSelector.js
    |_ constants
        |_ Colors.js
    |_ navigation
        |_ ScreensNavigator.js
    |_ screens
        |_ SelectPhoto.js
        |_ GetResults.js

anpr-mapp is the project root for the frontend. I will go through underlying files in the next section.

Code flow:

For UI, I generally start with the structure (how should it look as per my planning section) and then place the functions (what should it do) in the placeholders.

(1) I decide a color scheme for header-background(secondary), header-text (primary) and buttons (primary) and configure them as constants (and leave rest of the components to defaults).

(2) Next I need 2 screens and ability to navigate through the screens as per my lo-fi wireframes.

• Create screens directory as the container for screens
• Screen 1 selects a photo — so create file SelectPhoto.js
• Screen 2 displays the results of ANPR — so create file GetResults.js
• Create navigation directory
• Configure ScreenNavigator.js — It should create a stack of above 2 screens using React Navigation. Also added the default header style.

• Link ScreenNavigator to App so the first component in the navigation stack (SelectPhoto in above snippet) is rendered when the app is launched.

(3) I haven’t configured my screens yet — lets start with SelectPhoto. It should have some text and the 2 buttons as per the wireframes.

Didn’t want to clog this component directly with all the code related to accessing camera, photo-roll and permissions — so I set up a new ImageSelector component (within components directory) which will return the 2 buttons with all the functions to select a photo.

Lets inspect the above snippet. ImageSelector component has 2 buttons — Click Photo and Camera Roll (line 51–69). On press of either, it invokes buttonPressHandler function with different parameters that returns a promise when it’s tasks are completed, the tasks being:

• check which button is pressed
• accordingly seek permission to access camera or photo roll
• launch the native camera or photo app if permission is granted
• invoke onSelect function passed as props to ImageSelector component with the uri of selected photo as a parameter (line 48)

With ImageSelector component setup, all’s left to do in SelectPhoto is to combine the body text and ImageSelector.

I like to touch on selectedImageHandler function (line 7) which is passed to ImageSelector component as a prop. It simply navigates to GetResults screen and shares the uri for the selected image as a parameter.

(4) So far I have completed the ‘Intro’ screen per wireframe along with ‘Select an image by accessing camera or photo roll’ function. Time to focus on GetResults screen with posting the image and displaying the response from backend. Finally it is where everything comes together.

On load, GetResults screen displays the image (based on the uri passed from SelectPhoto component) and Confirm button (shown as GetNumber in wireframe) to post the image to backend. I felt a Cancel button will also add to usability if user wants to go back to SelectPhoto screen (though it is also possible based on the back button on the header as default capability of React Navigator).

I have created a placeholder for SetResult component (line 82)— which will display the response of backend. Here’s how — pressing Confirm button will invoke confirmHandler function (line 12). The function will:

• initialize a FormData component with the image
• post it to /anpr endpoint of my backend
• wait for response
• set a state variable vehicleId using React UseState Hook

SetResult component in turn will render the value of vechileId if set.

That’s pretty much it for the Frontend.

Testing

Moment of truth! I have the backend up and luckily Expo does the rest. I just scanned the bar code on my phone (needs Expo app installed) — feel free to run it on iOS/Android simulators on your computer if you prefer. And here’s how it looks finally.

Clone the code and try it yourself. And hope you find the post useful.

Limitations of the app

1. ANPR results are not up to the mark. With car.PNG test image in my git, it mistakes a ‘5’ for ‘S’ if you notice carefully. Also for images where the Aspect Ratio min/max assumptions are not valid, you will see ‘unable to decode’ message.
2. Lack of structure and acceptable practices (far from best practices 😊) in anpr.py bothers me — doesn’t make it much readable.

I plan to address above (and #1 will need some work) — not sure how soon though.