Measured before claimed.
The benchmark is not marketing — it is how the platform is governed. The best measured system ships, weaknesses are hunted on purpose, and a number is published only after it survives pre-registered evaluation.
Honest states, not empty promises.
Most numbers here are still in publication. Rather than advertise internal results, each dimension shows its true state — the same discipline the platform applies to knowledge.
| State | Meaning |
|---|---|
| published | survived pre-registered evaluation; number, n and confusion matrix available |
| measured · internal | measured against internal gold sets; publication pending external validation |
| publication pending | gold set or evaluation in construction |
| UNKNOWN · fail-closed | the capability abstains in production rather than guess |
Parsing, dimension by dimension.
| Dimension | Unit | Status | Notes |
|---|---|---|---|
| Reading order | F1 vs gold | measured · internal | publication after pre-registered eval |
| Tables | structure F1 | measured · internal | gold set expanding — n reported with every number |
| Text recognition accuracy | CER / WER | measured · internal | measured per document family |
| Forms | field F1 | publication pending | gold set in construction |
| Long documents | fidelity at 60+ pages | publication pending | long-document bench in progress |
| Charts | chart-to-table | UNKNOWN · fail-closed | abstains instead of guessing |
Every product is a measured surface.
| Dimension | Unit | Status | Notes |
|---|---|---|---|
| Audio | transcription fidelity | measured · internal | acquisition → evidence, certified |
| Determinism | identical re-runs | measured · internal | byte-level round-trip checks |
| Structure alignment | output vs source structure | publication pending | measured against gold |
| Retrieval | grounded recall | publication pending | measured on governed knowledge |
| Claim extraction | precision / recall | measured · internal | abstention counted, never hidden |
| Reasoning grounding | answers traceable to claims | measured · internal | adversarial sets included |
| Latency | p50 / p95 per unit | publication pending | per product, per deployment |
| Cost | per 1k units | publication pending | published with credit rates |
Designed so we cannot flatter ourselves.
Every vendor wins their own benchmark. Ours is built to find our weaknesses first.
Pre-registered
Metrics, thresholds and datasets are fixed before the run. No metric shopping after the fact.
Scoped numbers
No naked scores: every published number carries its dataset and version, sample size (n), inclusion criteria, the comparators it was measured against, its run date, and known limitations.
Independent gold
Gold sets are held independent of the systems under test — never derived from the same process that produces the results being scored.
Gold is corrigible
Gold sets are scientific artifacts: versioned, adjudicated, corrected through a recorded process — never silently edited.
Abstention is scored
UNKNOWN is a measured outcome. A system that hides its abstentions is optimizing the wrong thing.
Reproducible evaluation
The evaluation itself is reproducible: same datasets, same metrics, same protocol — you can re-run the comparison, not just read the score.
Continuously re-measured
Every capability is re-measured as methods improve. The best measured result ships — and only keeps its place while the numbers hold, including ours.
Detection proven
Evaluation layers are validated by deliberate fault injection — we prove the benchmark can catch failure before trusting its green.
Method, not mechanism
What we publish is the proof, not the machine. Datasets, metrics and results are open; the implementation that achieves them is not part of what you audit — the number is.
A benchmark that never stops.
Every release is re-measured against the benchmark. Regressions are caught by the benchmark, not by customers.
Gold sets grow by adjudicated annotation. Coverage gaps are tracked as debt, visibly.
Published results with datasets and methodology, verifiable end to end.