How it works, how well, and where it stops

The spec

The brief is dated 12 August 2026. It chose the own-coverage case: if a policy can be analyzed this deeply for the owner’s own coverage, then checking a party’s document against a fixed question set falls out of the same engine. It also said to treat the question set as swappable configuration, not something to invent.

This build follows both sentences. Requirements go in, documents go in, a cited answer comes out. The five questions are one requirement set. A subcontractor’s requirements are another. The engine does not change.

The three versions

• V1 reads. Supply policy documents; get back what each document is, the coverage tower, the questions answered, and a flag for each problem the reading can prove. Every figure cites its page and printed words. V1 is the reading itself, running live. Done when the spec’s evaluation list is demonstrated on the sample set.
• V2 operates. The same engine run by a team: the run stops and asks a person when it cannot decide, every step names its actor and instruction document, and documents belong to a client account, not a browser session. The stops and the provenance run today. Accounts, roles, and persistence are designed. What V2 adds.
• V3 prices. The target is not the gap, it is the dollars: total uncovered exposure, split into retained and gap exposure. Every dollar is arithmetic over figures the tower already holds, citable to a page. Designed. What V3 adds.

Approach

Six steps: ingest, classify each document into a closed type vocabulary, extract with every field carrying a page and a verbatim quote, resolve documents into policies, resolve policies into a program, check and render. Steps one through three interpret and are a model. Steps four through six execute and are code.

No model performs arithmetic on a limit anywhere. The tower, the totals, and every comparison are code over extracted figures, so the same documents produce the same tower every run.

Every value’s citation is checked in code against the text of the page it names. If the quoted words sit on exactly one other page, the page is corrected and the correction reported. Words found nowhere, or in several places, are not repaired, and a verdict resting on that citation is discarded.

Questions are bindings: required fields, a test, three outcomes. Pass, fail, cannot determine. Answerability is checked first, so a missing field is never reported as a failure. When a schedule of underlying insurance names a policy the file does not contain, the run stops and asks a person, showing the line that names it.

Libraries and models

• Next.js 14.2 with TypeScript. One application serves the pages, the reader, and the API. Pinned to the patched 14.2 release; nothing here needs a newer framework.
• Zod for the data contracts: citation, document, policy, program, review, question. The reading is validated in code against these, not by the model’s decoder.
• The Anthropic API for the two interpreting steps. The reader is Claude Sonnet 4.6. Everything vendor-specific is in one file, so the model is configuration.
• pdf-lib and pdfjs-dist: slicing pages to the API’s per-document cap, and extracting page text for the citation checks.
• Vercel Blob for uploads, because a policy package is larger than a serverless request body allows. The server still fetches and reads every byte.
• Node’s built-in test runner. 115 tests, no framework.

Observed accuracy

The evaluation harness scores extraction against labels that record their own provenance. On the corroborated labels, extraction of every tower value scored 100 percent. Classification has no defensible number yet, and the harness says so instead of reporting one.

The tower is reproducible. The same two-document program was read five times in one day: same $6,000,000 total, same two layers, same attachment point, cited to the same page.

Two answers read policy language rather than figures: whether defense costs erode the limits, and whether the umbrella drops down. Runs can differ on them, and the answer degrades itself and says why. The restriction count from the text sweep also varies between runs. The tower does not.

Limitations

• The model API accepts one hundred pages per document as images. The front of a document is read as images; the rest is read as extracted text and swept. No page goes unread, but deep pages are read less richly, and the reading says which was which.
• A document with no text layer is refused before any model is called. A citation that cannot be checked against a page would rest on trust.
• Restrictions are identified, never interpreted. Endorsements are flagged by form number in code; restricting provisions are found in the text, cited, and listed. What a provision does to a specific loss is a legal reading this build does not perform.
• A sub-limit paired to the wrong provision is the least trustworthy output. The figures are printed and cited; which provision owns a figure is the weakest link.
• One reading is one request, with a 300 second ceiling on the deployed platform. The run is budgeted to finish inside it. Production changes this; see below.
• Nothing persists. Reviews are in memory behind a storage interface, and uploads are deleted after reading. Right for a prototype, wrong for a product.

Production integration

The pipeline already runs behind the same API route the page uses. Integration is calling it: documents in, a streamed reading out, a stored result. What changes for production:

• Storage. A database replaces the in-memory implementation behind the existing interface. The pipeline does not change.
• Long runs become queued jobs. One request per step removes the request ceiling and lets a reading survive a deploy.
• Stops become a queue. A halted run waits for a person with the evidence attached, and the decision is recorded with who and when.
• Requirement sets are configuration. A customer’s own question set loads the same way the five questions do.
• The model boundary is one file, and each reading meters its tokens and cost. A full reading measured $1.50 to $3.

Go deeper