Modern Gravity is generally availableBlog ↗
Seen, but not chosenNational moving and storage companyGravityBench: how we test Gravity
Ask a business AI a plain question and it gives a plain number. The number looks the same whether it is right or wrong. GravityBench measures how often it is wrong without a warning, with and without Gravity, and this page shows exactly how the test is run so you can check it.
- The same model gets the same questions and the same data, with and without Gravity.
- The Gravity setup also has the company's certified definitions. The model alone does not, so the comparison measures the complete Gravity setup, including those definitions.
- The correct answers were computed from the raw data before any model saw a question.
- Answers are graded with the setup hidden. The key that says which setup wrote which answer is opened only after grading is finished, and the order is logged.
- Every published result carries its date, its size and a label that says whether it has been independently audited.
What GravityBench measures
GravityBench asks one question: when an AI answers from a company's own data, can you trust the number?
Five headline measures
| Measure | In plain words |
|---|---|
| Right answers | The figure matches the correct answer. |
| Confidently wrong | A wrong figure stated as fact with no warning, or a figure given where none should be, such as a number for a period that has not happened. This is the measure we care about most. |
| Honest refusals | The question cannot be answered well as asked (a measure nobody has defined, an unclear period, a month still in progress treated as complete), and the answer says so or asks, instead of guessing. |
| Different definition | A believable number built on a different definition than the one the company certified. Counted on its own, never folded into confidently wrong. |
| Same figure every time | Asked the same question three times, does it give the same figure each time. |
Six tracks
GravityBench is organized into six tracks. Each published run lists the tracks it covers, and a track a run did not cover is not claimed.
- Accuracy. Is the number right, compared with a correct answer worked out independently from the raw data?
- Honest refusal. When a question should not be answered as asked, does the system say so or ask, instead of guessing?
- Catching wrong answers. When we insert a known wrong answer, how many are stopped before they reach a reader?
- Grounding. Does the written answer stay inside the figures it was given, with no invented number?
- Data boundary. Does anything that should stay inside the company's own environment, such as known markers in the test data, show up outside it?
- Drift. After a known change to the company's data or calendar, does the system stop and say a definition needs to be certified again, or does it keep returning a number that is quietly wrong?
The double-blind comparison
The comparison is built so that neither the people grading nor the grading programs know which answer came from which setup until grading is done.
- 01Frozen first.The question set, the correct answers and the grading program are frozen. Their fingerprints are written down with a timestamp before the first answer is produced. Any change afterward starts a new, separately labeled run.
- 02Hypotheses written first.We write down what we expect to find, and how each expectation will be tested, before the run. Only those registered results get a headline. Every other comparison is labeled exploratory.
- 03Same model, same questions, same data, same date.Two setups answer every question. In the first, the model works alone with a read-only login to the database. In the second, the same model works with Gravity installed the way a customer installs it. Neither setup is told about the comparison.
- 04Answer cards.Every final answer is turned into a card. Anything that shows which setup wrote it (headings, labels, product and tool names) is removed or replaced with neutral words. The cards are shuffled and given random ids. The key that links ids to setups is sealed in a file the graders never read.
- 05Two graders and an audit.First, a program compares every figure on every card with the correct answer. Second, a grader model from a different model family than the one that wrote the answers reads each card against the correct answer and judges it by the definitions in Section 4. Then the answer key is recomputed from the raw data and a third grader, from another model family, checks every answer. No person hand-checks a sample, and there is no outside audit.
- 06Unblinding in a logged order.Each grader writes a completion record to an append-only log. The step that opens the key checks that both records exist and match their score files, and writes its own timestamp to the same log, so the log shows that grading finished before the key was opened.
- 07Everything is reported.Every registered result is published, including the questions where the model alone was right and the questions where Gravity was wrong or held back.
Hypotheses for the current comparison run
| ID | What we expected | How it is tested |
|---|---|---|
| H1 | The frontier model with Gravity gives fewer confidently wrong answers than the same model alone. | Paired by question; exact paired test on "confidently wrong on most repeats"; rates with 95% intervals |
| H2 | An open model with Gravity gives fewer confidently wrong answers than the same open model alone. | Same as H1 |
| H3 | The open model with Gravity compared with the frontier model alone. Stated as a comparison, not an assumed win. | Same as H1, two-sided |
| H4 | Each setup with Gravity gives the same figure more consistently across repeats than its counterpart. | Paired by question; exact paired test on "same figure in every repeat" |
The test data
Why the data is synthetic. The test database is built, not borrowed. No customer data is ever used, and no model in any setup sees any. It describes a services firm's workforce: headcount, joins, exits, hiring and offers, over 36 months of history plus the current month, which is only partly done on the day of the run. Rebuilding it gives identical rows.
Real-world data issues. Real business data is messy, and a messy fact has a defensible handling and an easy mistake. The easy mistake produces a number that looks exactly like a right one. We build 19 kinds of these issues in at known places and counts. Each is listed in a file the grader reads and confirmed by a separate scan of the raw rows.
| Code | What is in the data |
|---|---|
| D1 | Some rows have no customer recorded. |
| D2 | Some headcount numbers are blank in the source. |
| D3 | Some exits have no exit type. |
| D4 | Some rows appear twice in the source. |
| D5 | Some rows are dated in the future. |
| D6 | The current month is only partly done. |
| D7 | Exit types are spelled in mixed ways. |
| D8 | An exit was added after its month had closed. |
| D9 | Some people left and rejoined. |
| D10 | A quarter had no exits at all. |
| D11 | One month has no rows in any table. |
| D12 | An exit has no customer. |
| D13 | A movement date falls in a different month from its month column. |
| D14 | The monthly snapshot does not add up to the movements. |
| D15 | Some days-to-fill figures or dates are blank, negative or impossible. |
| D16 | A customer was renamed over time. |
| D17 | Some groups are very small. |
| D18 | Offer statuses are not what the name suggests. |
| D19 | Open positions are only a draft measure. |
The questions. The current comparison run uses 43 questions, and every setup is asked each one three times. Each run's row in the runs table gives its own question count and repeats. They are in plain business wording and include single figures, tables, follow-ups, questions that ask why something changed, and questions that should be refused or clarified. The questions are available on request, exactly as asked.
What synthetic data limits. Synthetic data lets us know the right answer for certain. It cannot contain the problems we did not think of. Known limits says what that means for how far a result travels.
How answers are scored
The correct answers. Before any model saw a question, an independent program worked out the correct answer to every question straight from the raw rows. It shares no code with the program that builds the test data and none with Gravity.
What counts as what.
- Right: the figure matches the correct answer.
- Confidently wrong: a wrong figure stated as fact with no warning, or a figure given where none should be. An answer that is wrong but tells the reader to be careful is counted separately.
- Honest refusal: the answer says it cannot answer as asked, or asks which reading is meant, on a question where that is the right response. A refusal on a question that could be answered is a wrong refusal, and it is counted against the setup that gave it.
- Different definition: a believable number built on a different definition than the certified one, for example counting a blank exit type as voluntary. Counted as its own category.
Questions with more than one fair reading. Some questions have no written rule that picks one reading, such as an undated "how many offers are still pending?". The grader accepts each reasonable reading it was written to recognize, and the results say which reading an answer used. A reading it does not yet recognize is counted against the setup that gave it, and the run's limits list says so.
Small numbers. A count is always shown, however small. A percentage over a small base is shown with a note, for example "(small number: based on 2 people)". The grader follows the rule as it stood on the run date, and the version history records the date the rule changed.
Consistency. For each question asked in every repeat, did the setup give the same figure each time.
Statistics. Rates carry 95% Wilson intervals. Registered comparisons use exact paired tests, paired by question. We always show the counts behind a rate.
Fairness rules
- What each setup was given. The model alone had the model, a read-only login to the database, a database client and the same plain description of each table. Nothing else, and no certified definitions. The model with Gravity had the same model and the same database, with Gravity installed the way a customer installs it, answering from the company's certified definitions. The model alone had a fair chance: it could query the data directly and was free to be careful. This comparison measures the complete Gravity setup, including its certified definitions, against the same model without them.
- Same everything else. Same questions, same wording, same database, same date, same number of repeats.
- Separate folders, proven. Every setup runs in its own clean folder. Before a run, a live test proves a setup cannot read outside its own folder. If a setup is found to have read files it should not have, its answers are kept on disk unchanged, never graded, never used in any claim, and the setup is asked again under isolation. The run record says so.
- Timed-out answers. An answer that times out or breaks is asked once more, the same way in every setup, before blinding. The second result stands, whatever it is.
- Nothing left out. Each answer is graded against the correct answer for the day it was produced, and no answer is removed because it looks bad.
- Held-out questions. About 80% of each question set stays out of public view so later runs of the same benchmark stay blind. The standard is that Gravity is not tuned on a run's held-out questions. Each run states in the runs table whether its question set had been run before and whether Gravity or the grading program was changed because of what that earlier pass showed.
Evidence
Published with each run.
- The headline, per-problem and per-question-type tables.
- Up to five worked examples, chosen by a fixed rule.
- The pre-registration fingerprints and timestamp, so anyone can check that nothing changed after the first answer.
On request. The questions with their correct answers, every answer card and score file, the sealed key and the append-only log that proves the order, the claims register that traces every published number to a source file and its fingerprint, and the methods file. We keep the questions out of public view for now, which also keeps later runs blind.
Never shared. Raw session transcripts, internal paths, credentials, and anything about how Gravity or the test data is built.
Ask for the evidence. Tell us who you are and what you want to check, using the form below. A person on our team replies within one business day.
Ask for the evidence
Tell us who you are and what you want to check.
Runs and results
One row per published run. Every number comes from that run's claims register, never typed by hand. Every row carries its date, its size and a label.
- Every published run stays in the table, including runs that score lower than earlier ones. When a later run with a lower score is published, its numbers replace the earlier headline numbers, and the earlier row is marked as superseded and stays listed.
- A row is added only after the run has been graded, unblinded and signed off by the person who owns the benchmark.
- Until an outside audit signs, every row is labeled internal, with no outside audit.
Counts: accuracy and confidently wrong rates use the scored answers (105 per setup for the October 2026 run, across 35 questions); steps and the same-figure-every-repeat rate use all 43 questions, each asked three times.
Read the confidently wrong rate alongside the different-definition count. A different-definition answer is not counted as confidently wrong, even when the number would mislead someone using another definition. Each run shows that count for both setups beside its headline results.
| Date | Gravity version | Models | Tracks covered | Questions and repeats | Answers graded | Confidently wrong | Different definition | Accuracy (right answer or honest refusal) | Same figure every repeat | Question set run before? | Label | Paper and limits |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2026-10-15 to 2026-10-17 | 0.1.11 | Claude Sonnet 5.5, DeepSeek V4.1 Flash | Accuracy, Honest refusal | 43 x 3 | Alone: 105With Gravity: 105 | Claude Sonnet 5.5 Alone: 43.8% (95% interval 34.7% to 53.4%), 46 of 105With Gravity: 0.0% (95% interval 0.0% to 3.5%), 0 of 105 DeepSeek V4.1 Flash Alone: 46.7% (95% interval 37.4% to 56.2%), 49 of 105With Gravity: 1.0% (95% interval 0.2% to 5.2%), 1 of 105 | Not reported for this run | Claude Sonnet 5.5 Alone: 43.8% (95% interval 34.7% to 53.4%), 46 of 105With Gravity: 98.1% (95% interval 93.3% to 99.5%), 103 of 105 DeepSeek V4.1 Flash Alone: 39.0% (95% interval 30.3% to 48.6%), 41 of 105With Gravity: 94.3% (95% interval 88.1% to 97.4%), 99 of 105 | Claude Sonnet 5.5 Alone: 53.5% (95% interval 38.9% to 67.5%), 23 of 43With Gravity: 88.4% (95% interval 75.5% to 94.9%), 38 of 43 DeepSeek V4.1 Flash Alone: 58.1% (95% interval 43.3% to 71.6%), 25 of 43With Gravity: 79.1% (95% interval 64.8% to 88.6%), 34 of 43 | Yes. This question set was run once before, and what that first pass showed was used to fix Gravity and the grader before this run. | Internal double-blind test, October 2026, 43 questions, synthetic data; grading audited by a third model; no outside audit Mean AI work steps per answer, Claude alone 6.1, with Gravity 4.1 (32% fewer; measured after the run, not pre-registered). Steps are not tokens or cost: with its answer check, Gravity used more tokens per answer in this run. |
Version history
Every change to the benchmark, its questions or its scoring is listed here with its date, so older results stay comparable. A result is only compared with results from the same version.
| Date | Version | What changed |
|---|---|---|
| 2026-10-16 | Comparison method | The double-blind comparison method in Section 2 is the standing procedure for every comparison run. Scoring rule for small numbers: a count is always shown; a percentage over a small base is shown with a note. |
| 2026-09-25 | v3 | The benchmark was rebuilt to test the current Gravity release. Results from earlier versions tested an older release; they are not comparable and are not used here. |
| 2026-07-24 | v2 | Protocol with a public sample and a held-out set, a grader from a different model family, and a person check on a 10% sample. The October 2026 run replaced the person check with a recompute of the answer key and a third grader on every answer. |
Known limits
We list these on the page, not in a footnote.
Standing limits
- Synthetic data. The data is built to contain known problems. Real data has problems we did not think of, and real rates may differ.
- One domain. Workforce headcount, exits, hiring and offers. We make no claim about finance, supply chain or any other area.
- Tool versions. A run uses one version of the assistant tools and one Gravity release. Later releases will differ. Each run's limits name them.
- Writing style can hint at the setup. Blinding removes labels, headings, tool names and code, but a reader could still guess which setup wrote a card from its style.
- Model graders, no person check. The grading program has its own limits. The grader models can be wrong. The answer key is recomputed and a third grader checks every answer, but no person hand-checks a sample and no outside party has audited the grading.
- Questions with more than one fair reading. The grader accepts the readings it was written to recognize. Readings it does not recognize are counted against the setup that gave them.
- The date moves. Some answers depend on the day they are asked, such as how far through the month it is. Each answer is graded against the correct answer for the day it was produced.
- Usage limit stops. Runs on a subscription plan stop cleanly at the plan's usage limit and resume where they stopped. Nothing is graded wrong because of a stop.
- No competitor comparison. The comparison is a model with and without Gravity. It does not compare Gravity with any other product.
- The test is ours. We built Gravity and we run the benchmark. That is why the hypotheses are written first, the grading is blind and the evidence can be requested.
Limits specific to a run
Each runs-table row links to its limits: grader issues found and fixed before grading, grader issues still open and the questions they touch, answers asked again, answers that came from a backup model, and any isolation failures and how they were handled.