Skip to content

Commit 4cf91ed

Browse files
committed
Fix PolicyBench paper provenance and snapshot artifacts
1 parent ab12e85 commit 4cf91ed

47 files changed

Lines changed: 4707 additions & 250 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

app/public/paper/policybench.pdf

164 KB
Binary file not shown.
110 KB
Loading
106 KB
Loading

app/public/paper/web/index.html

Lines changed: 112 additions & 57 deletions
Large diffs are not rendered by default.

app/public/paper/web/site_libs/quarto-contrib/glightbox/glightbox.min.css

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

app/public/paper/web/site_libs/quarto-contrib/glightbox/glightbox.min.js

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
body:not(.glightbox-mobile) div.gslide div.gslide-description,
2+
body:not(.glightbox-mobile) div.gslide-description .gslide-title,
3+
body:not(.glightbox-mobile) div.gslide-description .gslide-desc {
4+
color: var(--quarto-body-color);
5+
background-color: var(--quarto-body-bg);
6+
}
7+
8+
body:not(.glightbox-mobile) div.gslide-media {
9+
background-color: var(--quarto-body-bg);
10+
}
11+
12+
.goverlay {
13+
background: rgba(0, 0, 0, 0.7);
14+
}
15+
16+
div.gslide-description .gslide-title {
17+
margin-top: 0.25em;
18+
margin-bottom: 0.25em;
19+
font-weight: 500;
20+
font-family: inherit;
21+
}
22+
23+
div.gslide-description .gslide-desc {
24+
padding-bottom: 0.5em;
25+
font-family: inherit;
26+
}

app/src/data.json

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

docs/benchmark_card.md

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -51,9 +51,9 @@ The headline score uses the numeric answers only. Explanations are retained for
5151
auditing, scenario exploration, and qualitative error analysis; they should not
5252
be described as faithful reasoning traces.
5353

54-
Numeric-only runs are allowed only as ablations for measuring prompt-contract
55-
effects. They should be labeled as ablations and should not be mixed into
56-
leaderboard claims.
54+
Canonical runs require numeric answers and explanations for each requested
55+
output. If future prompt-contract ablations omit explanations, they should be
56+
labeled as ablations and not mixed into leaderboard claims.
5757

5858
CLI default outputs under `results/local/` are scratch artifacts, not canonical
5959
leaderboard snapshots.
@@ -98,21 +98,25 @@ that means US tax year 2026 and UK fiscal year 2026-27.
9898

9999
The public scenario explorer exposes the household prompts, model outputs, and
100100
reference outputs. The public leaderboard should therefore be treated as an
101-
open-set benchmark. Future protected leaderboard claims require a separate
102-
held-out or rotating evaluation set.
101+
open-set benchmark with possible leakage from released cases into future model
102+
behavior or benchmark-specific prompting. Future protected leaderboard claims
103+
require a separate held-out or rotating evaluation set.
103104

104105
## Country data paths
105106

106107
### United States
107108

108-
The US benchmark uses households derived from PolicyEngine US Enhanced CPS and
109-
scores outputs against PolicyEngine US reference outputs.
109+
The US benchmark uses households derived from PolicyEngine US Enhanced Current
110+
Population Survey (CPS) and scores outputs against PolicyEngine US reference
111+
outputs.
110112

111113
### United Kingdom
112114

113115
The current public UK path uses a calibrated transfer dataset rather than
114116
restricted native UK survey microdata. It should be described as a public UK
115-
transfer path for benchmarking, not as a replacement for enhanced FRS.
117+
transfer path for benchmarking, not as a replacement for enhanced Family
118+
Resources Survey (FRS) microdata or as a population-representative UK household
119+
sample.
116120

117121
## Naming discipline
118122

@@ -121,12 +125,14 @@ Public prose should prefer:
121125
- `reference outputs`
122126
- `frozen snapshot`
123127
- `public calibrated transfer dataset`
128+
- `equal-country global summary`
124129

125130
Public prose should avoid:
126131

127132
- unqualified `truth` language for reference outputs
128133
- `current best model` without a dated snapshot
129134
- `first public benchmark`
135+
- treating the global score as a universal model ranking
130136

131137
## Minimum reporting standard
132138

@@ -139,6 +145,7 @@ Every public writeup should state:
139145
- the output set used
140146
- whether the claim refers to the live site or a frozen paper snapshot
141147
- whether UK results come from the public transfer dataset or a later artifact
148+
- whether global results are equal-country summaries or use another weighting
142149
- sensitivity checks for at least amount-only, binary-only, positive-reference
143150
cases, zero-reference cases, country-only rankings, and household-equal
144151
impact scores when available

0 commit comments

Comments
 (0)