Receipt manager with OCR support written in dart.
Keeping track of your receipt is pretty hard. You need to update the shop names, receipt date and a total of every bill. Why do you not use the digital advantage to your advantage?
Thus there was no application which could parse receipts and manage receipts and is trusted, so I decided to create my own application, which is written in dart using the flutter framework. I use flutter because it massively decreases the code development time, it is simple to use and cross-platform.
Design choices
There was a lot of design choices, but I decided to create a client-server architecture since mobile devices have an unacceptable performance.
The database
All receipt get stored in a SQL database, which allows to save receipts permanently. I use moor because it is type safe, easily extendable and fast in combination with the BLOC pattern.
The BLOC pattern is a state management system for Flutter recommended by Google developers. It is strongly related to the MVC/MVP (model view controller and model view presenter pattern.
If you are familiar with the BLOC pattern, you noticed that you have to define your events. I define database events e.g and InsertEvent, UpdateEvent and DeleteEvent.
To translate an database event to the corresponding state, I used the following method.
If e. g. an InsertEvent is raised. The repository performs database access using a data access object (DAO). This scheme gives you even more power. If you have a new application, you can easily define new events, this will save a lot of time.
Lesson learned: Use design patterns. If you are an app developer and have a database application, use the BLOC pattern.
App design
But back to the main application. I quickly created a minimalist and simple app design. I made the mistake that I did not use a theme manager. Moreover, I define colours in each widget. In the end, this was more work than required and expected.
Lesson learned: create a theme manager and use an SQL database for main applications. You won’t regret this.
I decided to create my main theme like this. At the first look, it looks like a monster is born but the theme manager allows defining colours in widget globally.
But there was still a lot of work. Flutter has some fancy user input forms which also support validation.
I use the TextFormFields and I use a custom regex. Regex is basically and sequence of characters, to define a search pattern. To check if the given total is correct, I check if the total contains only digits and two decimal places. You can express this condition using a regular expression.
If the regex matches the string the user input was correct (or the regex was wrong :)).I created a simply receipt form like this.
At this point, the application was ready to insert receipts. Thus you can’t edit any bills. I used the flutter slidable library to create a custom slider. Each slider event corresponds to a bloc event which is translated to a database event e. g. insert, edit or delete.
I crafted the history widget using the previously show slider. If an InsertEvent, UpdateEvent or DeleteEvent is raised the HistoryWidget is automatically updated. I used some free logos to design each category to give the HistoryWidget more vivid look. The HistoryWidget features a simple design but you get still a quick overview of your expenses.
The settings widget is crafted using the fancy flutter package settings_ui but at the moment you can only change locals and the server address.
Receipt parsing using tesseract
The application now supports receipt parsing, which is a very challenging task.
Client
- At first the user takes an photo of the receipt.
- The photo is uploaded to your server via a POST request
You don’t have to worry about third party companies, since there aren’t any.
Now, the photo is stored at the server.
Server
- Server increase the contrast of the given image
- Blur is applied to the image
- The server uses pytesseract to parses the text
- Now, the parsed text is send as a JSON response
Client
- The application convert the JSON response to a receipt object
- The application auto fill the text-fields
The main problem is the image quality of the smartphone camera. To tackle this issue, I enhance the image using a variety of techniques. First, I resize the image.
Where FX and FY describes the scaling factor of the image.In this case, INTER_CUBIC generally performs better than other alternatives, though it’s also slower than others. Since we care about quality I do not use the INTER_LINEAR.
What got the ball rolling was the gaussian blurring which uses the Gaussian kernel instead of a normalized box filter, for convolution. In our case, the dimensions of the kernel and the standard deviations in both directions can be determined independently. It is very useful to remove noise from the image thus it does not preserve the edges in the input.
To remove the salt and peppter noise in the image. I applied a median blurring. The central element in the kernel area is replaced with the median of all the pixels under the kernel.
After, I use convert to increase the contrast. This is very important since it massively increase the outcome but this is covered in the receipt parser library.
Now, the configuration is read by the parser. In the configuration the language and the fuzzy keywords are defined. Since OCR is an supreme discipline, tesseract is not able to parse the image without any artifacts. This is how the output looks like:
Therefore an fuzzy parser is required to identify shop names, total, date and category. My example config looks like:
At first this might be some work but at the end it is worth or you use the default configuration but you might want to change the current language.
If the server can parse the receipt. It return a JSON string which contains the required information about total, shop name and date. The application convert this JSON string to an receipt object and autofill the form. You can now change the every field if required or store the receipt directly in the application.
If you are curious about the project, please check out my application at GitHub and the server. The project is completely open-source. I would be happy if you share ideas, feedback or give the project an star. This would help to help to increase the popularity.
The application is available at F-DROID and precompiled binaries are available at the Github release page.
Projects
I’m glad to read your feedback.
