The model may only add caution
Model output passes through an action allow-list: the only thing it can contribute to a decision is an extra HOLD. Approval is a deterministic rule outcome with a confidence floor of 0.85.
Invoice-to-PO Decision Workflow
An accounts-payable workflow where AI reads, code decides, and people own the exceptions: every invoice ends as approve, hold with one question, or reject, and every check cites its source record.
- Role
- Designed and built
- Where
- Independent project
- When
- Oct 2026
- Access
- Public repo · live demo
Problem
Accounts payable has to match each invoice against its purchase order, goods receipt, and vendor master, and catch duplicates, bank-change fraud, and GST errors. A model that approves payments is a liability, and one that can be talked into approving by text inside the invoice is worse.
01
Any format in, processed once
Text PDFs go through pdftotext and scans through OCR. Row totals, subtotal, and grand total are cross-checked, so OCR digit errors are caught without a model, and an idempotency key means a resent invoice is processed once.
02
A queue that fails closed
A database-backed queue claims each job, retries with backoff up to three times, and turns a system error into HOLD. Nothing approves by accident.
03
Checks against the systems of record
Intake scans for injected instructions, then each check writes evidence of expected versus actual: arithmetic, vendor and bank details, and the PO. It catches a PO edited after the invoice date, a recently changed bank account, and a cancelled GSTIN.
04
Three-way match and GST
Price is checked against the PO and the rate contract, quantity against cumulative goods receipts (which catches split-PO overruns), and the GST head against supplier and buyer state. Credit notes skip the match.
05
The model advises, rules decide
A TypeSafe Jev decision model judges likely duplicates and suggests a PO, but an action allow-list drops anything it says except an added HOLD. The most severe finding wins, and approval needs confidence of at least 0.85.
06
One question for one person
Every HOLD carries a named owner, a next step, and a draft email to the vendor (except suspected bank fraud, which gets no draft). Every check is cited in the evidence trail.
07
Evaluation as a gate
One offline command runs every eval suite, and CI runs it on every change with recorded model answers, so no secrets are needed. A regression below the gate fails the build.
Measured
- False approvals on adversarial variants
- 0 / 600
- Prompt-injection attacks that changed a decision
- 0 / 489
- Labelled cases correct, 0 false rejects
- 31 / 31
- Duplicate recall with Jev (from 0.75)
- 48 / 48
- PDF fields extracted
- 32 / 32
- Tests, 20 of them guardrail tests
- 79
- Pipeline p95 latency
- 14 ms
- Cost of 180 Jev calls
- $0.0052
How it's verified
- Harness
- FastAPI service with a durable job queue (three tries, then HOLD), a nine-step check pipeline, and a TypeSafe Jev advisor running in replay mode by default.
- Evals
- A gated harness: 31/31 labelled cases, 0 false approvals across 600 adversarial variants, 48/48 duplicate recall, and an LLM judge validated against a 69-item gold set (kappa 0.659).
- Guardrails
- Intake scans for injected instructions; an action allow-list means the model can only add a HOLD; approval needs confidence of at least 0.85; 0 of 489 injection attacks changed a decision.
- Testing
- 79 tests (20 of them guardrail tests), run in CI behind a regression gate.
In depth
Fail closed, everywhere
System errors become HOLD, a resent invoice is idempotent, and the queue gives up after three tries into HOLD rather than dropping the job.
A judge that is itself evaluated
The LLM-as-judge was checked against a 69-item human-labelled gold set (kappa 0.659, bad-item recall 1.0). It is only allowed to gate on criteria that passed that meta-evaluation; the explanation criteria failed it, so they stay advisory.
Free to run, free to deploy
Replay mode serves recorded model answers by default, so CI and the free Render deployment need no API key, and the whole eval suite costs nothing to rerun.
Decisions
- 01AI reads, code decides, people own the exceptions.
- 02Fail closed plus idempotency instead of best-effort processing.
- 03Gate the judge only on what its own meta-evaluation supports.
Stack
Python · FastAPI · SQLite · poppler + tesseract OCR · TypeSafe Jev · OpenTelemetry · Next.js · Docker · GitHub Actions · Render