Tested in the open
Evaluations
How Critias is tested — scripted conversations over synthetic and public data, exact checks on every figure, and published results including failures.
Critias is tested the way it is used: a conversation with a manager over one dataset. The evaluation suite ships in the agent repository, so every result here can be rerun.
How a case works
A runner plays the manager's turns into the real app, one at a time. The manager never names files or columns. At the end, the runner asks Critias to save the figures it gave. A case passes only when every check passes.
What is checked
- Saved figures match the answer key, with tight tolerances for margins and rates.
- Figures in the reply appear as stated, with the right sign where direction matters.
- Questions: Critias asks when it must, and does not ask when it should not.
- Outliers: required outliers are found and known false positives are not listed.
- Saved context holds only the facts the manager confirmed.
- Reasoning is graded by a judge model from a different model family, which must quote the reply to award a point.
The data
Most cases run on synthetic datasets built by scripts, so every expected answer is computed, not written by hand. Others run on three public datasets:
- Online Retail II, Daqing Chen, UCI Machine Learning Repository, CC BY 4.0
- AdventureWorks, Microsoft, MIT licence
- Olist e-commerce data, Olist, MIT licence
Costs, competitor prices and data faults added to the public data are ours, not the original publishers'.
Results
October 2026 benchmark: 111 cases, three runs each, Critias 0.5.0 with Pricing Plugin 0.4.12. A run passes only when every check in it passes.
| Analyst model | Thinking | Runs passed | Same figures on every repeat | Analyst cost | Mean time per run |
|---|---|---|---|---|---|
| Claude Haiku 5.5 | high | 239 of 333 (72%) | 65 of 84 cases | $19.72 | 67 s |
| DeepSeek V4.1 Flash (1017) | high | 218 of 333 (65%) | 58 of 84 cases | $14.79 | 129 s |
Both were graded by GPT-6.1 Sol (thinking low), a judge from a different model family.
By difficulty
Each case is rated 1 to 5 by the kind of pricing work it asks for. The rating is set by the task, not by how the models scored.
| Level | Kind of work | Cases | Haiku 5.5 | DeepSeek V4.1 Flash |
|---|---|---|---|---|
| 1 | Look up a figure or definition, or route a request to the right method | 10 | 30 of 30 (100%) | 28 of 30 (93%) |
| 2 | One standard calculation with a stated rule or a single data trap | 34 | 86 of 102 (84%) | 80 of 102 (78%) |
| 3 | Multi-step analysis: bridges, outliers under a peer rule, breakeven scenarios, joins | 43 | 98 of 129 (76%) | 88 of 129 (68%) |
| 4 | Diagnosis: why a number moved, messy exports from several systems, unit economics | 11 | 13 of 33 (39%) | 18 of 33 (55%) |
| 5 | Decisions and estimates the data cannot settle: recommendations, policy, willingness to pay | 13 | 12 of 39 (31%) | 4 of 39 (10%) |
What this means in practice:
- Levels 1 to 3: use with spot checks. Check the basis Critias states and the figures your decision rests on.
- Level 4: use as a first pass. Critias finds the numbers but often does not test the cause it names.
- Level 5: treat as a draft. A person should decide.
A frontier model on the hardest cases (indicative)
GPT-6.1 Sol (thinking medium) ran the 24 level-4 and level-5 cases, three runs each. It was graded by Claude Sonnet 5.5 (thinking high), not by the judge above, so read these figures as indicative rather than a like-for-like comparison.
| Level | Cases | GPT-6.1 Sol | Haiku 5.5 | DeepSeek V4.1 Flash |
|---|---|---|---|---|
| 4 | 11 | 20 of 33 (61%) | 13 of 33 (39%) | 18 of 33 (55%) |
| 5 | 13 | 19 of 39 (49%) | 12 of 39 (31%) | 4 of 39 (10%) |
On the level-4 cases Sol scored 16 of 30 on thinking high against 17 of 30 on medium, so more thinking did not help.
GPT-6.1 Terra (thinking xhigh) ran the 13 level-5 cases, three runs each, graded by DeepSeek V4.1 Flash (thinking high). It passed 18 of 39 (46%), level with Sol. Read by hand, one of the judge's fails meets the rubric, which would make it 19 of 39.
Where a person needs to check the answer
| Area | Both small models, runs passed | What goes wrong |
|---|---|---|
| Explaining why a number moved | 4 of 30 | A likely cause is named but not tested against the data. |
| Outlier and peer comparisons | 3 of 30 | Peer medians and matched competitor products differ from run to run. |
| Recommendations and rollout plans | 8 of 48 | Actions are not matched to each account's cause or staged against the risk of losing it. |
| Willingness to pay and volume response | 7 of 24 | A figure is given that the data cannot support, such as a price response from one promotion. |
Six cases fail in every run on both small models, and the five of them that Sol ran fail on Sol too. Terra fails the three of them at level 5 in every run. They test cause-finding, peer comparison and account-level recommendations.
Limits
- Three runs per case catch large differences, not rare ones. A one-run difference is within the spread of repeated runs.
- The judge is one model. In at least one case it failed replies that, read by hand, meet the rubric.
- The answer keys are public, so a model that has seen the repository could score higher for that reason alone.
- Some datasets are small and synthetic; results on your own data may differ.
The full report, with results by area and the fixes made during the benchmark, is in the evaluation suite in the agent repository.
Last updated