Receipt manager with OCR support written in dart.

William Todt
Oct 24, 2020 · 7 min read
Image for post
Image for post

Keeping track of your receipt is pretty hard. You need to update the shop names, receipt date and a total of every bill. Why do you not use the digital advantage to your advantage?

Thus there was no application which could parse receipts and manage receipts and is trusted, so I decided to create my own application, which is written in dart using the flutter framework. I use flutter because it massively decreases the code development time, it is simple to use and cross-platform.

Design choices

There was a lot of design choices, but I decided to create a client-server architecture since mobile devices have an unacceptable performance.

All receipt get stored in a SQL database, which allows to save receipts permanently. I use moor because it is type safe, easily extendable and fast in combination with the BLOC pattern.

The BLOC pattern is a state management system for Flutter recommended by Google developers. It is strongly related to the MVC/MVP (model view controller and model view presenter pattern.

Image for post
Image for post
BLOC pattern

If you are familiar with the BLOC pattern, you noticed that you have to define your events. I define database events e.g and InsertEvent, UpdateEvent and DeleteEvent.

Image for post
Image for post

To translate an database event to the corresponding state, I used the following method.

Image for post
Image for post
map event to state

If e. g. an InsertEvent is raised. The repository performs database access using a data access object (DAO). This scheme gives you even more power. If you have a new application, you can easily define new events, this will save a lot of time.

Lesson learned: Use design patterns. If you are an app developer and have a database application, use the BLOC pattern.

But back to the main application. I quickly created a minimalist and simple app design. I made the mistake that I did not use a theme manager. Moreover, I define colours in each widget. In the end, this was more work than required and expected.

Lesson learned: create a theme manager and use an SQL database for main applications. You won’t regret this.

I decided to create my main theme like this. At the first look, it looks like a monster is born but the theme manager allows defining colours in widget globally.

Image for post
Image for post

But there was still a lot of work. Flutter has some fancy user input forms which also support validation.

I use the TextFormFields and I use a custom regex. Regex is basically and sequence of characters, to define a search pattern. To check if the given total is correct, I check if the total contains only digits and two decimal places. You can express this condition using a regular expression.

Image for post
Image for post

If the regex matches the string the user input was correct (or the regex was wrong :)).I created a simply receipt form like this.

Image for post
Image for post
Main user input form

At this point, the application was ready to insert receipts. Thus you can’t edit any bills. I used the flutter slidable library to create a custom slider. Each slider event corresponds to a bloc event which is translated to a database event e. g. insert, edit or delete.

Image for post
Image for post

I crafted the history widget using the previously show slider. If an InsertEvent, UpdateEvent or DeleteEvent is raised the HistoryWidget is automatically updated. I used some free logos to design each category to give the HistoryWidget more vivid look. The HistoryWidget features a simple design but you get still a quick overview of your expenses.

Image for post
Image for post
history widget

The settings widget is crafted using the fancy flutter package settings_ui but at the moment you can only change locals and the server address.

Image for post
Image for post

The application now supports receipt parsing, which is a very challenging task.

Client

  1. At first the user takes an photo of the receipt.
  2. The photo is uploaded to your server via a POST request

You don’t have to worry about third party companies, since there aren’t any.

Now, the photo is stored at the server.

Server

  1. Server increase the contrast of the given image
  2. Blur is applied to the image
  3. The server uses pytesseract to parses the text
  4. Now, the parsed text is send as a JSON response

Client

  1. The application convert the JSON response to a receipt object
  2. The application auto fill the text-fields
Image for post
Image for post

The main problem is the image quality of the smartphone camera. To tackle this issue, I enhance the image using a variety of techniques. First, I resize the image.

Image for post
Image for post

Where FX and FY describes the scaling factor of the image.In this case, INTER_CUBIC generally performs better than other alternatives, though it’s also slower than others. Since we care about quality I do not use the INTER_LINEAR.

What got the ball rolling was the gaussian blurring which uses the Gaussian kernel instead of a normalized box filter, for convolution. In our case, the dimensions of the kernel and the standard deviations in both directions can be determined independently. It is very useful to remove noise from the image thus it does not preserve the edges in the input.

To remove the salt and peppter noise in the image. I applied a median blurring. The central element in the kernel area is replaced with the median of all the pixels under the kernel.

Image for post
Image for post

After, I use convert to increase the contrast. This is very important since it massively increase the outcome but this is covered in the receipt parser library.

Now, the configuration is read by the parser. In the configuration the language and the fuzzy keywords are defined. Since OCR is an supreme discipline, tesseract is not able to parse the image without any artifacts. This is how the output looks like:

Image for post
Image for post

Therefore an fuzzy parser is required to identify shop names, total, date and category. My example config looks like:

Image for post
Image for post

At first this might be some work but at the end it is worth or you use the default configuration but you might want to change the current language.

If the server can parse the receipt. It return a JSON string which contains the required information about total, shop name and date. The application convert this JSON string to an receipt object and autofill the form. You can now change the every field if required or store the receipt directly in the application.

The Startup

Medium's largest active publication, followed by +752K people. Follow to join our community.

William Todt

Written by

Currently a computer science student at KIT. Happy to meet new people and learn new skills and I love science

The Startup

Medium's largest active publication, followed by +752K people. Follow to join our community.

William Todt

Written by

Currently a computer science student at KIT. Happy to meet new people and learn new skills and I love science

The Startup

Medium's largest active publication, followed by +752K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store