Receipt Recognition — How to Extract Data from Paper Receipts
The term “Receipt Recognition” describes the process of converting paper receipts to machine readable data structures.
Today most businesses use paper receipts in their shops. For example if you go shopping for groceries in Germany, or when you have dinner in a restaurant, you will most likely receive a paper receipt:
Some businesses have started to replace paper receipts with digital receipts. The main purpose is to reduce the number of printouts. This leads to cost savings and at the same time protects our environment.
Even though the evolution of the receipt has started, you will still get a paper receipt most of the time. If you get a digital receipt, you will most likely get a PDF. Having a PDF can be considered as digital, on the other hand a PDF is not per default machine readable. ‘Machine readable’ means that a computer software can directly access the data of the receipt. For example read and process the total amount, the shop information or the products the customer has bought. For this use case a special pre-processing of the receipt is needed.
Before we evaluate the step of digitizing paper receipts, let us have a look at the benefits of having the data of the receipt in a machine readable format.
Expense Management
Receipt recognition is the ideal solution for apps or software to help customers to manage daily expenses, travel expenses and so on.
Feedback from customers
If you have a shop or business and want to get feedback from your real customers, you can use receipt recognition to easily identify them by letting them upload a receipt. So they can provide feedback to products and you have the certainty that they have really bought it.
Marketing and Loyalty
If you want to reward your customers for shopping at your store or buying products from your brand, but you do not want to check every single receipt, then this process can be automated with receipt recognition.
Others
You can think of many other solutions like:
- Eliminating manual input of receipt data during financial administration
- Cost split apps with the help of line item detection on receipts
- Automation of tax reporting
- Reimbursement tools for companies
Now, as we have seen the potential of receipt recognition, let us go to the next step and understand how you can get meaningful data out of a paper receipt.
Get meaningful data out of a paper receipt
OCR or Optical Character Recognition is the first step towards receipt recognition. Special algorithms are used to convert text from images to machine-encoded text. Before the OCR algorithm can process the image, it is recommended to run several pre-processing steps on the image to increase the success rate of the OCR. Wikipedia offers an article with some further details: https://en.wikipedia.org/wiki/Optical_character_recognition
OCR is not only used for receipt recognition. There are several use cases like license plate detection, online translation tools for mobile phones with cameras, book digitization and many more. Therefore there are also several OCR implementations available to support building new applications on top.
Just to mention a few of them:
- Amazon Textract (https://aws.amazon.com/de/textract)
- Google Vision AI (https://cloud.google.com/vision)
- Microsoft Computer Vision (https://azure.microsoft.com/en-us/products/cognitive-services/computer-vision)
- Tesseract (Open Source) (https://github.com/tesseract-ocr/tesseract)
Let us use a receipt from the German drugstore ‘dm’ to see the result of Amazon Textract.
As you can see, the OCR algorithm provides a quite good result for the single words. In the next step the single words need to be grouped and they need to get a ‘meaning’ so that the software can work with the data. This means that for example the algorithm needs to understand that the main part of the receipt is a table with products and prices. Each line contains one product with a corresponding price. Additionally a line can also contain multiples of the same product.
Amazon offers different interpretations of the data. They support the user for example with the detection of tables or forms. Actually the receipt can be interpreted as a table, so let us have a look.
In principle the approach works quite well. It detects the area where we, as humans, would expect the table too (product list). It also separates the products from the price which is good. Unfortunately our receipt is a photo taken with a mobile phone and therefore the paper was not flat. So the assumption of having just a grid projected on the receipt does not work here. You can see that due to the wavy paper the text is not following the line. In this example this leads to a mix of the line two and three. The table on the right side shows a product name ‘4x-1,85 Früch. Müsli 375g’ which is actually the text of line 2 in the beginning and the text of line 3 in the end. Additionally ‘dmBio Kinder’ is then a new product in the table.
If you want to cover such use cases, it makes sense to take the raw data of the OCR result and optimize your own algorithm to the specific need. Amazon offers for this case the data as a JSON structure with coordinates of the text blocks so the blocks can be processed individually.
If you do not want to write your own logic for receipt recognition because you just want to use it for your next awesome app, you can have a look at specialized APIs like:
- Eagle Doc from S2Tec GmbH (https://s2tec.de/receipt-recognition-api.html)
- Taggun
- Verify
- Nanonets
- Klippa
- FormX
- Mindee
- Tabscanner
- Base64
- Cradl
- Dataleon
- … and many more, just check with your favorite search engine
Example: How to use Eagle Doc from S2Tec
As a next step we want to have a look at how you can use the Receipt Recognition Service Eagle Doc from S2Tec (https://s2tec.de/receipt-recognition-api.html).
If you just want to have a look in general, you can also check our website where you can upload a receipt and the algorithm shows you the extracted information:
https://s2tec.de/receipt-recognition-api.html
Another way is to download the ‘mallet app’ from the app store and see the integration of the receipt recognition service live in an app:
If you want to test the Receipt Recognition Service on API level, the easiest way is to go to RapidAPI and get a free test account:
https://rapidapi.com/s2tec-gmbh-s2tec-gmbh-default/api/receipt-recognition
In case the offered plans do not fit your needs please contact us directly to discuss your requirements in detail. We are happy to find the best solution for you: info@s2tec.de
On RapidAPI you will also find a drop down menu with example code for your favorite programming language to easily integrate it in your application.
For our example we will use the web interface of RapidAPI and upload a receipt to see the result. Let us use the same image as we have used previously with Amazon to show the improvement.
In the result (JSON) we can see that the algorithm has successfully detected the shop name, the total price, the line items and other useful information. Now, as we do have a machine readable document we can forward it to the next step in an automated processing chain.
Summary
In this article we have provided an overview of how receipt recognition works and explained the need for it. Furthermore we have provided an easy way of using receipt recognition for your own project with the Eagle Doc solution from S2Tec. In case you have questions, please feel free to contact us: info@s2tec.de
