@@ -51,9 +51,9 @@ The headline score uses the numeric answers only. Explanations are retained for
5151auditing, scenario exploration, and qualitative error analysis; they should not
5252be described as faithful reasoning traces.
5353
54- Numeric-only runs are allowed only as ablations for measuring prompt-contract
55- effects. They should be labeled as ablations and should not be mixed into
56- leaderboard claims.
54+ Canonical runs require numeric answers and explanations for each requested
55+ output. If future prompt-contract ablations omit explanations, they should be
56+ labeled as ablations and not mixed into leaderboard claims.
5757
5858CLI default outputs under ` results/local/ ` are scratch artifacts, not canonical
5959leaderboard snapshots.
@@ -98,21 +98,25 @@ that means US tax year 2026 and UK fiscal year 2026-27.
9898
9999The public scenario explorer exposes the household prompts, model outputs, and
100100reference outputs. The public leaderboard should therefore be treated as an
101- open-set benchmark. Future protected leaderboard claims require a separate
102- held-out or rotating evaluation set.
101+ open-set benchmark with possible leakage from released cases into future model
102+ behavior or benchmark-specific prompting. Future protected leaderboard claims
103+ require a separate held-out or rotating evaluation set.
103104
104105## Country data paths
105106
106107### United States
107108
108- The US benchmark uses households derived from PolicyEngine US Enhanced CPS and
109- scores outputs against PolicyEngine US reference outputs.
109+ The US benchmark uses households derived from PolicyEngine US Enhanced Current
110+ Population Survey (CPS) and scores outputs against PolicyEngine US reference
111+ outputs.
110112
111113### United Kingdom
112114
113115The current public UK path uses a calibrated transfer dataset rather than
114116restricted native UK survey microdata. It should be described as a public UK
115- transfer path for benchmarking, not as a replacement for enhanced FRS.
117+ transfer path for benchmarking, not as a replacement for enhanced Family
118+ Resources Survey (FRS) microdata or as a population-representative UK household
119+ sample.
116120
117121## Naming discipline
118122
@@ -121,12 +125,14 @@ Public prose should prefer:
121125- ` reference outputs `
122126- ` frozen snapshot `
123127- ` public calibrated transfer dataset `
128+ - ` equal-country global summary `
124129
125130Public prose should avoid:
126131
127132- unqualified ` truth ` language for reference outputs
128133- ` current best model ` without a dated snapshot
129134- ` first public benchmark `
135+ - treating the global score as a universal model ranking
130136
131137## Minimum reporting standard
132138
@@ -139,6 +145,7 @@ Every public writeup should state:
139145- the output set used
140146- whether the claim refers to the live site or a frozen paper snapshot
141147- whether UK results come from the public transfer dataset or a later artifact
148+ - whether global results are equal-country summaries or use another weighting
142149- sensitivity checks for at least amount-only, binary-only, positive-reference
143150 cases, zero-reference cases, country-only rankings, and household-equal
144151 impact scores when available
0 commit comments