Designing a review queue people actually use

A review queue should contain the items where a human decision changes the outcome, and nothing else. In practice that means low-confidence fields, failed arithmetic, first-time vendors, amounts above a threshold you set, and possible duplicates. Everything that passes those tests posts without review.

Get the contents wrong in either direction and the queue fails: too much and it gets rubber-stamped, too little and errors flow through unexamined. Rubber-stamping is the more common failure and the more dangerous one, because it looks like review.

The queue’s job

Not to double-check the software. To place human attention where automation is structurally weak — ambiguity, intent, and novelty — and to keep it away from where automation is strong.

That framing settles most design questions. “Should every receipt be reviewed?” No: reviewing a correct entry costs attention and returns nothing, and doing it a hundred times a week guarantees the review becomes mechanical. “Should nothing be reviewed?” Also no: the errors that matter are exactly the ones that look normal.

What belongs in it

Low confidence on a specific field. Per-field, not per-receipt. “Total uncertain” tells the reviewer where to look; “this receipt scored 0.71” tells them to re-read everything, which they won’t.

Arithmetic that doesn’t reconcile. Line items not summing to subtotal, subtotal plus tax not equalling total. This is the highest-value trigger there is, because it catches confident extraction errors — the ones with no low-confidence flag attached. A wrong field usually breaks the arithmetic even when it reads perfectly.

First occurrence of a vendor. Not because the amount is likely wrong, but because the category is a guess and this is where a pattern gets set. Correcting the first one prevents the next fifty.

Amounts above your own threshold. Set by what would actually hurt if wrong. Reviewing everything over a trivial figure is how a queue becomes noise; reviewing everything over a figure that matters is how it earns its place.

Possible duplicates. Always shown as a pair, with both images, never resolved silently. Two seconds of human looking beats any matching rule.

Anything with handwriting. Handwritten tips and totals are hard for recognition and genuinely ambiguous for extraction — which of the two amounts on the slip is authoritative depends on which copy you’re holding.

Missing required fields. If your books need a business purpose and there isn’t one, only the person who spent the money can supply it.

What should never reach it

Repeat vendors matching an existing rule. The decision was already made. Re-presenting it is asking the same question again.

Small amounts that reconcile cleanly. High confidence, arithmetic agrees, known vendor, trivial value. Reviewing these is pure cost.

Anything a rule could resolve. If reviewers keep making the same correction, that’s not a review item, it’s a missing rule. Every recurring correction should end with someone writing the rule down.

That last point is the difference between a queue that shrinks and one that grows forever.

Make each item decidable in seconds

A queue that is slow per item won’t be worked, regardless of how well-targeted it is.

Show the image next to the fields. Every time. The reviewer’s actual task is comparing extracted values to the photo, and anything that makes them navigate to find the image adds friction to the core action.

Point at the problem. Highlight the uncertain field, or the arithmetic that fails. Don’t make the reviewer hunt for what triggered the flag.

Make the common fix one action. Usually: accept, or correct one field and accept.

Offer “always do this”. When a correction is a rule in disguise, capture it right there. This is where a queue converts one-off work into permanent improvement.

Never make review a form. If confirming a correct receipt requires filling anything in, the reviewer will optimise by not looking.

Watch the queue’s own metrics

Two numbers tell you whether the design is working:

Correction rate. Of the items flagged, how many actually needed changing? Very low means you’re flagging too broadly and training reviewers to accept by reflex. Very high means capture or extraction has a problem worth fixing upstream instead.

Time per item. Rising time means items are getting harder or context is missing. Falling toward near-instant means rubber-stamping.

A well-tuned queue is small, has a meaningful correction rate, and shrinks over time as rules accumulate.

The failure mode to design against

The queue that contains everything. It starts from caution — review it all, be safe — and ends with a reviewer who accepts a hundred items in two minutes without reading them. That is strictly worse than no queue, because it produces a record of human approval that means nothing, and it hides the ten items that genuinely needed a decision among ninety that didn’t.

If you can only enforce one rule: keep the queue small enough that working it carefully is realistic. A reviewer who reads five items properly catches more than one who skims a hundred.