Skip to content

The golden masters could not tell when the flight happened - #110

Open
thc1006 wants to merge 2 commits into
developfrom
fix/golden-masters-anchor-absolute-time
Open

The golden masters could not tell when the flight happened#110
thc1006 wants to merge 2 commits into
developfrom
fix/golden-masters-anchor-absolute-time

Conversation

@thc1006

@thc1006 thc1006 commented Jul 28, 2026

Copy link
Copy Markdown
Member

Two defects in the golden-master regression tests, both measured against
develop.

The baselines had no absolute-time anchor

Neither baseline stores a time axis or a launch step, and post_launch_positions
re-zeroes the origin at the first finite row, so the rocket trajectory carries no
information about when the flight happened. Displacing the whole flight later
while keeping the total step count passed every test up to 60 steps, 0.60 s, on
scenario 0 and 62 steps on scenario 1. What stopped it there was the downsampled
row count drifting outside its tolerance, not any check on time.

Scenario 0 is partly protected because pop_step is an absolute step index, so
displacing the pops with the flight brings the largest passing shift to zero.
Scenario 1 pops nothing, so it had no absolute anchor for the rocket at all
beyond the two steps STEP_COUNT_ABS_TOL allows. That matters there more than
anywhere: scenario 1 compares two arrays on two different clocks, the rocket
launch-relative and the balloons absolute from step 0, and whether a pop happens
depends on their relative phase.

Both baselines now record the launch step and both tests assert it before
comparing positions. The largest displacement that passes drops from 60 and 62
to 2, which is STEP_COUNT_ABS_TOL itself.

A test oracle that recomputed the code it was checking

test_scenario1_regression.py derived the release spacing as
int(release_interval / time_step) and its docstring called that "the
independent half, derived from the scenario file, not read back from the
environment". It is the identical float division production runs, so the test
could not detect the error class it advertised. With the truncation made visible:
an interval of 0.3 s at a 0.1 s step gives 2 in production and 2 in the oracle
where 3 is correct, and the test passes. Same at 0.7 and at 2.3.

Production now rounds, through a named _release_spacing_in_steps, and the
oracle compares in seconds so it never repeats that division. The two shipped
scenarios are unaffected: 1 / 0.01 and 0.5 / 0.01 are exact, and every committed
baseline number is byte for byte what it was.

On the baselines

Running the regenerators produced a diff on every trajectory digit, up to
0.694 mm. That is not from this change. Dumping all eight arrays with the fixed
production code and again with balloon_world.py checked out from HEAD gives
bit-identical results, so the rounding switch moves nothing here; the committed
baselines were simply taken on a different build, 700x inside the 0.5 m
tolerance. The regenerated files were discarded and the new field inserted at
the position the regenerator puts it, so the baseline diff is exactly one added
line per file with no existing byte touched.

Verification

The release-spacing test was landed against the original truncating body first
and failed there, 22 != 23. Six mutations each fail a named test, including
both baseline values and the rounding itself. Two controls survive, one of them
the rounding change measured against the shipped-values test, which proves the
claim that it is latent for the shipped scenarios rather than asserting it.

No tolerance was weakened. ruff check and ruff format --check pass on the
paths CI uses, uv lock --check is unchanged, pytest tests/ -q gives 301
passed and BPC_RUN_SLOW_TESTS=1 gives 321 passed.

Noted, not changed

_release_spacing_in_steps rounds half to even for an interval that is exactly
half a step, such as 0.25 s at 0.1 s. A scenario like that is arguably a
configuration error worth rejecting outright, which is a rules decision rather
than a test fix.

…lease oracle recomputing production

Two things the scenario 0 and 1 regression tests could not see.

The baselines carried no time axis and no launch step, and both runners
slice their positions from the first finite row, so the trajectory that
gets compared is launch-relative. Measured against the committed
baselines: moving the whole rocket flight later while keeping the episode
length passed every test up to 60 steps on scenario 0 and 62 on scenario
1, and 61 and 63 only failed on the downsampled row count, not on any
clock. Scenario 1 is the one that matters, since it pops nothing and so
has no absolute anchor for the rocket at all, while the balloons it is
scored against run from step 0. Both baselines now record the launch
step and both files assert it inside STEP_COUNT_ABS_TOL, which drops the
largest displacement that passes from 60 and 62 to 2.

The new field is the only change to either baseline. Regenerating them
here also moved the trajectory digits, up to 0.7 mm of 3D displacement,
but that is this machine against the machine they were taken on: HEAD's
own production code reproduces the same drift, and the rounding change
below is bit identical on both scenarios. So the field was inserted
rather than the baselines rewritten.

The release schedule oracle claimed to be derived from the scenario file
and was not: it evaluated int(release_interval / time_step), the same
float division the environment used, so an error there appeared on both
sides and cancelled. 0.3 / 0.1 is 2.9999999999999996, which truncates to
a spacing of 2 against a correct 3, and the comparison was 2 against 2.
The environment now rounds that quotient, the oracle compares in seconds
instead, and the conversion has its own test on intervals the shipped
scenarios do not use. 1 / 0.01 and 0.5 / 0.01 are exactly integral, so
neither shipped scenario moves; that is pinned too.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 28, 2026 21:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Both found by adversarial review of this branch, and both are the shape the
second half of this PR is about: a check that cannot fail for the reason it
claims.

The displacement test asserted a bare `assertRaises(AssertionError)`, and
`launch_step` raises AssertionError for its own input guards too. So the test
passes when the helper rejects its arguments rather than when the clock is
wrong. Measured: dropping the tail trim in `displace_flight_in_time`, which
breaks the same-step-count contract its docstring promises, left the test
passing on both scenarios while the real failure was "the clock has 6012 entries
and the trajectory has 6022, so a row cannot be dated". Matched on the message
now.

The new baseline field had no semantic pin, so a regeneration could bless a real
delay. Measured: move the agent's launch_time from 1 s to 3 s, regenerate, and
the baseline records launch_step 302 with every test passing. This file already
guards popped_count against exactly that, on the stated grounds that a
regression must not be blessed by regenerating, and the field needs the same.
The new check derives the expected step from the agent's own configuration and
the scenario clock, so it is an independent statement rather than the baseline
restating itself, with the two-step pipeline lag named rather than folded into
the tolerance.

Both mutations were silent before and each now fails a named test; a control
survives.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006 thc1006 mentioned this pull request Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants