Building an Intelligent Document Processing System

Every greenfield project is defined by a handful of early decisions you’ll either live with or pay to undo. Ours were deliberately boring:

• One codebase, two audiences.
  A single frontend with role-based routing — staff and admins share the same app with different capabilities — instead of maintaining two separate products.
• Own the auth.
  Self-managed authentication instead of an external identity provider. The integration cost wasn't justified for an internal operations tool with a known audience.
• Profile-swappable infrastructure.
  Storage, email, document processing, and external services all swap implementations per environment. Dev, test, and prod share code without sharing config.
• Interface-segregated services.
  Every dependency with a vendor or external concern sits behind an interface. Swapping a provider later is a config change, not a refactor — a decision that paid off within weeks.

Pick abstractions you’ll actually use. Discard the ones you picked because they sound right.

Before a single line of vendor-specific code ships, we run the comparison. Picking the right model for a pipeline like this one is a four-axis problem:

• Accuracy on the distribution you’ll actually see.
  Not on benchmark leaderboards that test a different distribution. Build a fixed eval set from real inputs. Score every candidate against it.
• Cost per call at your projected volume.
  At low volume, any model is cheap. At real volume, per-call cost is a product decision — it determines whether you can afford to retry, re-run, or grade anything.
• Latency from where users actually are.
  A model behind a cross-region network boundary will surprise you with long tails. Measure from the user’s region, not from where you develop.
• Operational fit.
  Billing, compliance, quota, regional data rules, support responsiveness. The stuff that determines whether you can actually run the thing you benchmarked.

A model that wins on one or two axes and loses on the others looks great on paper and loses in production.

The model we chose for this project outperformed more famous alternatives on the specific distribution we cared about, at an order-of-magnitude lower per-call cost, from closer to the deployment region, with an operational fit that simplified billing and quota management. It wasn’t a contrarian pick. It was the result of actually running the comparison most teams skip.

Unbundling the pipeline

Our initial approach put extraction and structured parsing behind a single vendor call. Simpler to wire up. More expensive to iterate on — every prompt change ran the whole pipeline.

A few weeks in, we split it. A dedicated stage for raw text extraction. A separate stage for structured parsing, powered by a different class of model. Each became independently improvable, independently swappable, and dramatically cheaper to iterate on.

“Swap either stage without rewriting the other” is an investment, not a refactor. We make it on day one because we don’t know which stage will need to change first — only that one of them will.

Reading labels accurately is the heart of this product. The technique wasn’t the hardest lesson. The measurement was.

We versioned prompts like code. Every change had a ticket, a rationale, and a score against a fixed eval set. We refused to merge regressions. The scoring harness ran on every change.

That discipline caught the only thing it could catch: regressions against the distribution we had defined. It didn’t catch the subtler failure.

One iteration looked like a breakthrough — close to 98% on the hardest fields across our eval set. Every measurement agreed.

Production held a fraction of that accuracy.

The eval set was too clean. Flat documents, good lighting, recurring names. Real inputs were rotated, creased, glare-hit, and featured names we’d never seen. The harness we’d built so carefully was measuring the wrong distribution.

A test set is a hypothesis, not a verdict.

We did three things with that learning.

First: grade against live-sampled inputs, not curated ones. Every prompt change now scores against a rolling sample pulled from recent production uploads. If the distribution drifts, the score drifts, and we notice.

Second: we moved from negative rules to positive examples. An earlier iteration had leaned on “do not return X, do not confuse Y with Z.” It regressed. The replacement described positive patterns — “extract things that look like these” — with concrete examples from the real distribution. It recovered and surpassed.

That pattern generalizes. Models learn what to do better than they learn what to avoid. Negative rules leave every other bad option open; positive examples show the shape of the right answer.

Third: we stopped treating any single vendor as structurally critical. Once extraction and parsing were unbundled, vendor choice became a config decision. Measurement became the contract.

Processing documents asynchronously is table stakes for a pipeline of this shape. The design choices aren’t.

Act one — the durable-first polling path

Our first version pulled work from a durable store on a short interval. Slower than an event-driven design. Also: survived every failure mode we could think of. Restarts, crashes, network blips, partial writes — the poller resumed on the next tick and picked up what was left.

“Event-driven first” is a popular default. It’s also a popular way to silently lose work when the event layer and the work layer aren’t carefully paired. The poller’s pessimistic contract — “if no one confirmed the work is done, I’m going to keep trying” — is a design principle, not a fallback.

Act two — the event overlay

When we measured real-world latency against user expectations on the mobile intake flow, the tradeoff had gotten expensive. Users expected instant acknowledgement. A multi-second worst case wasn’t shippable.

We added a fast path. New work fires a processing signal immediately. The durable poller stayed in place as a safety net — same contract, same guarantees, now behind a typical fast path.

This is the shape we’ve ended up with repeatedly: a fast path for the happy case, a durable slow path for everything else. The fast path doesn’t need to survive every failure mode because the slow path does. The slow path doesn’t need to be fast because the fast path is.

Act three — atomic work-unit claim

Hybrid introduced a new problem. With two paths converging on the same work, the same unit could be processed twice — once by the fast trigger, once by the poller on its next tick. Our first-generation guard was an in-process check. It worked until it didn’t.

The real fix was smaller and more fundamental. Claim the work atomically at the transaction layer. The work-unit boundary is now a transaction, not an application-level convention. Both paths are safe to overlap.

That’s the pattern for every queue-like subsystem we’ve built since. The shape doesn’t change. The sophistication is in knowing when to add each layer — and in resisting the temptation to skip the durable one because the event one is faster.

The fastest way to onboard a package is to scan it — camera, barcode, done. The textbook answer is to pick a library and wire it up. In practice, every library we tried broke on some device.

The shipped solution isn’t a single library. It’s a platform-aware fallback chain that tries options in order until one works on the user’s specific device, plus a small bag of hacks around undocumented behaviors we got tired of rediscovering. Mobile browser fragmentation is a tax you pay whether you budget for it or not.

The same logic extended elsewhere in the app: image formats that worked on one platform but not another, APIs that worked on HTTPS but silently failed on insecure contexts, capture flows that needed different timing on different hardware. None of this is interesting engineering. All of it is necessary.

The product had been running in production for a few weeks before we put a formal security pass on the calendar. When we did, we found the usual suspects: development shortcuts that had quietly outlived their purpose, hardening that had been deferred in favor of velocity, and tests that relied on behaviors we were now changing.

We tightened the auth surface. We retired development-era endpoints. We hardened the response posture. Rate limits landed where they belonged. The accompanying test migration caught a couple of places where old behavior was still quietly being relied on.

None of this was novel. All of it was necessary.

“The feature works” and “the feature is safe” are different statements. Ship the feature. Then put the audit on the calendar.

Real availability at this scale isn’t about nine nines. It’s about two questions: when something breaks, can we see it; and can we undo it?

We optimized for both without overpaying. Deployments are triggered manually from the pull request that introduces the change, so nothing ships by accident. Every deploy is pinned to an immutable build, and rolling back is pointing at a previous one — a two-minute operation that doesn’t require rebuilding anything. Logs are archived cheaply for later investigation. Client-side errors are captured the moment they happen.

Every layer picks the cheap option that’s still defensible, and the compounding effect is what matters.

Eight weeks from empty repo to a production system covering document understanding, a mobile-first operational workflow, role-based backend services, public-facing security, and deploy-plus-rollback automation. The velocity surprises people. The velocity is the output. The discipline is the input.

What we invested in up front:

• Interface-segregated services — so swapping any dependency later is cheap.
• Measurement harnesses before features — so regressions are visible before they ship.
• Production-shaped staging — so deployment-time bugs surface at the staging boundary, not at go-live.
• Immutable deploy artifacts and fast rollback — so every release is reversible in minutes, not hours.

What we deliberately didn’t invest in up front:

• Abstractions we weren’t going to use.
• Observability tooling that cost more than the problem it solved.
• Auth complexity our audience didn’t need.
• Vendor lock-in our architecture couldn’t absorb.

Each of those “didn’t invest” lines looks like cutting corners. It isn’t. It’s the thing that bought us the time to do the research, run the comparisons, and get each critical subsystem right on the first or second attempt instead of the fifth.