A missing number gets noticed. A wrong one does not.
That single asymmetry is why reading documents with a language model has to be treated differently from writing text with one. When a model invents a paragraph, someone usually catches it while reading. When it turns 349,000 into 340,000, there is a plausible number, in the right field, in the right format, and nobody has any reason to go look it up.
Most extraction projects do not fail loudly. They fail quietly, in the third digit, six months after launch.
A Wrong Number Costs More Than a Missing One#
An empty field is an interruption. Someone sees the gap, opens the source, fills it in. The cost is a minute of attention and the system stays trustworthy.
A wrong field is a decision. It gets forwarded, quoted in an offer, added to a total. By the time anyone notices, it has been copied into three other places, and the question is no longer what the right number is but which of the four versions in circulation came from the original.
That inverts the usual instinct. In extraction, coverage is the cheap goal and correctness is the expensive one. A system that leaves eight percent of fields empty and is right about the rest is worth more than one that fills everything and is quietly wrong about two percent.
Numbers Are the Trickiest Case#
Depending on the model and the tokenizer, a multi-digit number is not processed as one indivisible value. It is split into units that do not necessarily match the value a human sees in it. The work on counting ability and tokenization describes exactly that gap between the unit you mean and the unit that gets processed.
There is a second finding, about how numbers are represented. Research shows that language models encode them digit by digit, base 10, and that errors appear digit-wise: an incorrect prediction sits close to the correct answer in string edit distance and far away from it in value.
Those papers study arithmetic and counting, not copying out of a document. So they do not prove that every extraction error arises this way. They are still enough for the caution this article is about: an error in a number does not have to look like an error. 349,000 becomes 340,000. A property of 132 square metres becomes 138. It looks like a typo and behaves like nine thousand euros.
A valid number format is not the same thing as a verbatim copy. So I no longer treat "the model reads the document" as a workable architecture, but as an assumption that has to be backed up.
A Valid Schema Is Not a Statement About Truth#
The common answer to all of this is structured output, and it is a good tool that solves a different problem than people take it for.
OpenAI's own documentation is precise about what it covers: the response adheres to the supplied schema, no missing required key, no invalid enum value. It is equally clear about the limit, namely that structured outputs can still contain mistakes, and that input unrelated to the schema can still produce hallucinations.
So a schema guarantees that a field called price will exist and will contain a number. It says nothing about whether that number is in the document. Format compliance still reads as correctness, because the output looks like a database row, and that is exactly what makes it dangerous. The tidier the shape, the less anyone questions the contents.
The Source Is Fixed Before the Model#
The first of the five layers is a rule about inputs, not about prompts: the model gets no bare address, but the text that was fetched and stored beforehand.
An address is no evidence of which content was actually processed. Pages change, a fetch can fail, and without a stored copy there is no way to say later what stood there at the time of extraction. Fetch the document first, keep it, hand it over as fixed raw text, and exactly one version is in play.
This is not a side detail about security. Everything downstream, every check and every citation, depends on there being one document and on it still being there later.
Numbers First, and Without the Model#
The second layer is the one that gives this article its title. Prices, areas, room counts, dates and identifiers are pulled from the raw text first, without the model, by pattern, as candidates. The model then gets the text and those candidates, and its job is assignment: which figure belongs to which field, what the description says, how the parts relate. It does not get to regenerate the value.
How reliable that first step is depends on the document. With clean HTML or a PDF with embedded text it is very reliable. With a scan the uncertainty starts at text recognition, and tables, different decimal separators, currencies and numbers broken across line breaks do not help. Deterministic here means traceable, not automatically correct.
The split still follows the strengths. A language model helps well with deciding whether a figure means plot size or living area, because that is a question about language. For carrying eleven digits unchanged across a long context it is the weaker choice, because that is a question about copying. The assignment stays subject to review too; it is simply the task where the model actually helps.
The Checker Cannot Be the Author#
The third layer is a check that reads the result back against the raw text, and what matters is that it does not happen inside the same context it was produced in. No "now check that again" at the end of the same conversation. The second pass gets the raw text, the result and fixed criteria, and for critical fields the comparison additionally runs mechanically.
Why that separation is needed is shown by Large Language Models Cannot Self-Correct Reasoning Yet, presented at ICLR 2024. A group from Google DeepMind and the University of Illinois examined intrinsic self-correction, the case where a model revises its own answer without external feedback. Models struggled with it, and in some cases the result got worse after the correction step.
The finding applies to exactly that case, and that is the usable lesson. A model that just produced an answer produced it because it considered it right; asking it for a verdict from inside the same conversation asks it to contradict itself. Give it the source text and a specific claim to compare against, and it is no longer the same case.
What is missing, then, is not necessarily a different model. It is feedback from outside. In extraction that comes free: the source text is right there.
Every Field Names Its Evidence#
The fourth layer is a small requirement with a large effect. Every extracted field carries the passage it came from.
With that, verification stops being a judgment call. Either the quoted passage holds the value or it does not, and that comparison is mechanical and cheap. It follows from the architecture and needs no study as proof; work on citation-grounded extraction goes the same way, because the check then runs against the document instead of against another derived database.
One thing has to be thought through, or the layer only half works. The verbatim comparison holds as long as the stored value appears in the document exactly like that. As soon as anything is normalised, it does not: "349.000 €" becomes 349000, "01.03.2026" becomes 2026-03-01. So every field carries both versions, the raw value as it stands and the stored value, plus what happened in between. Otherwise you check the conversion along with everything else and, when something is off, cannot tell which of the two steps failed.
The indirect benefit is bigger than the direct one. A field that cannot name its source is not merely weaker evidence, it is a different category. It is a guess. Once evidence is mandatory, guesses become visible instead of blending into the same table as everything else, and then you can decide what to do with them.
The Last Layer Comes From the Subject Itself#
The fifth layer has nothing to do with language models any more. It is domain arithmetic: price against area, rooms against area, year built against a plausible range. Anything outside gets flagged, not deleted.
It catches the remainder, and it is the only layer that also catches a value that really is in the document and is wrong there, because the source had a typo. Against a source that errs, no amount of grounding helps. Plausibility does.
Where the Line Runs#
None of this argues against letting models near data. It argues for a division of labour, and the line is real.
The model owns the value when the value is language: a classification, a summary, a yes or no about whether the text mentions a garden. What I would not hand it are values a person would check on a bank statement. If someone could reasonably ask where a figure comes from, the figure needs a traceable path and a citation.
The honest cost of this route is that it takes longer than one instruction with a schema and demonstrates worse. There is no moment where you paste an address and a clean table appears. Instead there is a pipeline with a boring pattern check in the middle. The demonstration comes two months later, when someone asks where a number came from and the answer takes four seconds.
Of all the possible shortcuts, one would be especially dangerous: dropping the independent check. It is the layer that looks most redundant while everything is going well, and the one that catches exactly what nobody thought of.
A system that can show its sources is not a more careful version of a system that cannot. It is a different product, and it is the only one I would let near a customer's data.
