# Evaluations

How Critias is tested — scripted conversations over synthetic and public data, exact checks on every figure, and published results including failures.

Critias is tested the way it is used: a conversation with a manager over one dataset. The evaluation suite ships in the [agent repository](https://github.com/critias-ai/critias-pricing-agent), so every result here can be rerun.

## How a case works

A runner plays the manager's turns into the real app, one at a time. The manager never names files or columns. At the end, the runner asks Critias to save the figures it gave. A case passes only when every check passes.

## What is checked

- **Saved figures** match the answer key, with tight tolerances for margins and rates.
- **Figures in the reply** appear as stated, with the right sign where direction matters.
- **Questions:** Critias asks when it must, and does not ask when it should not.
- **Outliers:** required outliers are found and known false positives are not listed.
- **Saved context** holds only the facts the manager confirmed.
- **Reasoning** is graded by a judge model from a different model family, which must quote the reply to award a point.

## The data

Most cases run on synthetic datasets built by scripts, so every expected answer is computed, not written by hand. Others run on three public datasets:

- Online Retail II, Daqing Chen, UCI Machine Learning Repository, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
- AdventureWorks, Microsoft, MIT licence
- Olist e-commerce data, Olist, MIT licence

Costs, competitor prices and data faults added to the public data are ours, not the original publishers'.

## Results

October 2026 benchmark: 111 cases, three runs each, Critias 0.5.0 with Pricing Plugin 0.4.12. A run passes only when every check in it passes.

| Analyst model | Thinking | Runs passed | Same figures on every repeat | Analyst cost | Mean time per run |
| --- | --- | --- | --- | --- | --- |
| Claude Haiku 5.5 | high | 239 of 333 (72%) | 65 of 84 cases | $19.72 | 67 s |
| DeepSeek V4.1 Flash (1017) | high | 218 of 333 (65%) | 58 of 84 cases | $14.79 | 129 s |

Both were graded by GPT-6.1 Sol (thinking low), a judge from a different model family.

### By difficulty

Each case is rated 1 to 5 by the kind of pricing work it asks for. The rating is set by the task, not by how the models scored.

| Level | Kind of work | Cases | Haiku 5.5 | DeepSeek V4.1 Flash |
| --- | --- | --- | --- | --- |
| 1 | Look up a figure or definition, or route a request to the right method | 10 | 30 of 30 (100%) | 28 of 30 (93%) |
| 2 | One standard calculation with a stated rule or a single data trap | 34 | 86 of 102 (84%) | 80 of 102 (78%) |
| 3 | Multi-step analysis: bridges, outliers under a peer rule, breakeven scenarios, joins | 43 | 98 of 129 (76%) | 88 of 129 (68%) |
| 4 | Diagnosis: why a number moved, messy exports from several systems, unit economics | 11 | 13 of 33 (39%) | 18 of 33 (55%) |
| 5 | Decisions and estimates the data cannot settle: recommendations, policy, willingness to pay | 13 | 12 of 39 (31%) | 4 of 39 (10%) |

What this means in practice:

- **Levels 1 to 3: use with spot checks.** Check the basis Critias states and the figures your decision rests on.
- **Level 4: use as a first pass.** Critias finds the numbers but often does not test the cause it names.
- **Level 5: treat as a draft.** A person should decide.

### A frontier model on the hardest cases (indicative)

GPT-6.1 Sol (thinking medium) ran the 24 level-4 and level-5 cases, three runs each. It was graded by Claude Sonnet 5.5 (thinking high), not by the judge above, so read these figures as indicative rather than a like-for-like comparison.

| Level | Cases | GPT-6.1 Sol | Haiku 5.5 | DeepSeek V4.1 Flash |
| --- | --- | --- | --- | --- |
| 4 | 11 | 20 of 33 (61%) | 13 of 33 (39%) | 18 of 33 (55%) |
| 5 | 13 | 19 of 39 (49%) | 12 of 39 (31%) | 4 of 39 (10%) |

On the level-4 cases Sol scored 16 of 30 on thinking high against 17 of 30 on medium, so more thinking did not help.

GPT-6.1 Terra (thinking xhigh) ran the 13 level-5 cases, three runs each, graded by DeepSeek V4.1 Flash (thinking high). It passed 18 of 39 (46%), level with Sol. Read by hand, one of the judge's fails meets the rubric, which would make it 19 of 39.

### Where a person needs to check the answer

| Area | Both small models, runs passed | What goes wrong |
| --- | --- | --- |
| Explaining why a number moved | 4 of 30 | A likely cause is named but not tested against the data. |
| Outlier and peer comparisons | 3 of 30 | Peer medians and matched competitor products differ from run to run. |
| Recommendations and rollout plans | 8 of 48 | Actions are not matched to each account's cause or staged against the risk of losing it. |
| Willingness to pay and volume response | 7 of 24 | A figure is given that the data cannot support, such as a price response from one promotion. |

Six cases fail in every run on both small models, and the five of them that Sol ran fail on Sol too. Terra fails the three of them at level 5 in every run. They test cause-finding, peer comparison and account-level recommendations.

### Limits

- Three runs per case catch large differences, not rare ones. A one-run difference is within the spread of repeated runs.
- The judge is one model. In at least one case it failed replies that, read by hand, meet the rubric.
- The answer keys are public, so a model that has seen the repository could score higher for that reason alone.
- Some datasets are small and synthetic; results on your own data may differ.

The full report, with results by area and the fixes made during the benchmark, is in the evaluation suite in the [agent repository](https://github.com/critias-ai/critias-pricing-agent/tree/main/pricing-harness/tests/evals).