OCR is not data extraction

OCR converts an image into characters. Extraction decides which of those characters mean something — that this number is the total, that string is the vendor, those digits are a date. They are separate problems with separate failure modes, and a tool can be excellent at one and mediocre at the other.

Nearly every “the scanner got it wrong” complaint is really one or the other, and the difference tells you whether there’s anything you can do about it.

What OCR produces

Not a receipt. A list of text fragments with coordinates, and usually a per-fragment confidence.

Something like: "COFFEE" at these pixels, "3.40" at those, "TOTAL" lower down, "14.90" to its right. Position and text, nothing else. No structure, no meaning, no idea which fragment matters.

OCR on printed receipts is a mature technology. It struggles with genuinely degraded input — faded thermal print, dot-matrix output, heavy creasing — and it does well on anything legible. When OCR fails you can usually see why by looking at the image, because the characters are visibly hard to read.

What extraction has to do

Take that unstructured pile and produce typed fields. This is the hard part, and it’s hard for a reason that has nothing to do with image quality: a receipt has no schema.

Consider finding the total. The obstacles:

• It might be labelled TOTAL, AMOUNT, AMOUNT DUE, BALANCE, BALANCE DUE, TO PAY, or nothing.
• It is not necessarily the largest number on the slip — CASH TENDERED is often larger.
• It shares the page with subtotal, tax, tip, change, discount, and sometimes a loyalty points balance formatted exactly like currency.
• Position is a hint, not a rule: totals are usually low on the slip, but plenty of receipts print a summary block at the top.

So extraction combines weak signals — nearby labels, position, format, and above all internal arithmetic. If the line items sum to the subtotal and subtotal plus tax equals the candidate total, the labelling is probably right. That cross-check is the single most valuable tool extraction has.

The two failure modes look nothing alike

OCR failure: garbled or missing text. TOTAL 1A.9O instead of 14.90. Obvious, usually flagged as low confidence, and diagnosable by looking at the photo. If you can’t read it either, this is an input problem — capture it better or key it manually.

Extraction failure: every character read perfectly, wrong field chosen. The tip was taken as the total. The subtotal was taken as the total. The card’s last four digits were parsed as a date. The text is right, the meaning is wrong, and the result looks completely normal.

The second is far more dangerous, because there is nothing to notice. A garbled amount announces itself. A plausible wrong amount sits in your books until reconciliation.

Why this determines what you should check

If you know the failure modes, you know where to spend your attention.

Don’t spend it re-reading text you can see is clear. OCR on a clean photo of an intact receipt is not where your errors are.

Do spend it on arithmetic. Does subtotal plus tax equal total? Does the total match what you remember paying? Arithmetic checking catches extraction errors that no amount of squinting at characters will.

Do spend it on restaurant receipts specifically. The tip-versus-total problem is the most common extraction error in practice, because restaurant slips print two or three candidate amounts in close proximity, sometimes with the final one handwritten.

Do spend it on anything with a summary block at the top. Receipts that break positional convention are where positional heuristics fail.

What this means when comparing tools

Two products can quote similar accuracy and behave very differently, because they measured different stages.

A vendor reporting “99% character accuracy” is reporting OCR. That’s a real number and it’s nearly irrelevant to your experience, because your experience is dominated by whether the right field was picked.

The question worth asking is about extraction: how does it handle a receipt with a tip line? What does it do when arithmetic doesn’t reconcile — flag it, or silently pick something? Does it tell you which fields it was unsure about, per field, or hand you one overall score?

A tool that flags “total ambiguous, three candidates” is doing better work than one that confidently returns the wrong number, even though the first feels like the worse product.

The handwriting case

Worth separating out, because it’s where both stages fail at once. Handwritten amounts — a tip written on the merchant copy, a total filled in by hand — are hard for OCR (handwriting recognition is a distinct and harder problem than printed text) and hard for extraction (the handwritten number often contradicts the printed one, correctly, and deciding which wins requires knowing which document you’re looking at).

Treat any receipt with handwriting on it as one to check manually. That’s not a limitation of a particular product; it’s the honest state of the problem.

The short version

Character recognition is solved for legible receipts. Field interpretation is not solved and will not be, because receipts are printed by tens of thousands of point-of-sale systems that agree on nothing.

So capture well to make OCR easy, and review arithmetic to catch extraction. Those two habits address the two stages, and between them they account for nearly everything that goes wrong.