The golden masters could not tell when the flight happened - #110
Open
thc1006 wants to merge 2 commits into
Open
Conversation
…lease oracle recomputing production Two things the scenario 0 and 1 regression tests could not see. The baselines carried no time axis and no launch step, and both runners slice their positions from the first finite row, so the trajectory that gets compared is launch-relative. Measured against the committed baselines: moving the whole rocket flight later while keeping the episode length passed every test up to 60 steps on scenario 0 and 62 on scenario 1, and 61 and 63 only failed on the downsampled row count, not on any clock. Scenario 1 is the one that matters, since it pops nothing and so has no absolute anchor for the rocket at all, while the balloons it is scored against run from step 0. Both baselines now record the launch step and both files assert it inside STEP_COUNT_ABS_TOL, which drops the largest displacement that passes from 60 and 62 to 2. The new field is the only change to either baseline. Regenerating them here also moved the trajectory digits, up to 0.7 mm of 3D displacement, but that is this machine against the machine they were taken on: HEAD's own production code reproduces the same drift, and the rounding change below is bit identical on both scenarios. So the field was inserted rather than the baselines rewritten. The release schedule oracle claimed to be derived from the scenario file and was not: it evaluated int(release_interval / time_step), the same float division the environment used, so an error there appeared on both sides and cancelled. 0.3 / 0.1 is 2.9999999999999996, which truncates to a spacing of 2 against a correct 3, and the comparison was 2 against 2. The environment now rounds that quotient, the oracle compares in seconds instead, and the conversion has its own test on intervals the shipped scenarios do not use. 1 / 0.01 and 0.5 / 0.01 are exactly integral, so neither shipped scenario moves; that is pinned too. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Both found by adversarial review of this branch, and both are the shape the second half of this PR is about: a check that cannot fail for the reason it claims. The displacement test asserted a bare `assertRaises(AssertionError)`, and `launch_step` raises AssertionError for its own input guards too. So the test passes when the helper rejects its arguments rather than when the clock is wrong. Measured: dropping the tail trim in `displace_flight_in_time`, which breaks the same-step-count contract its docstring promises, left the test passing on both scenarios while the real failure was "the clock has 6012 entries and the trajectory has 6022, so a row cannot be dated". Matched on the message now. The new baseline field had no semantic pin, so a regeneration could bless a real delay. Measured: move the agent's launch_time from 1 s to 3 s, regenerate, and the baseline records launch_step 302 with every test passing. This file already guards popped_count against exactly that, on the stated grounds that a regression must not be blessed by regenerating, and the field needs the same. The new check derives the expected step from the agent's own configuration and the scenario clock, so it is an independent statement rather than the baseline restating itself, with the two-step pipeline lag named rather than folded into the tolerance. Both mutations were silent before and each now fails a named test; a control survives. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects in the golden-master regression tests, both measured against
develop.
The baselines had no absolute-time anchor
Neither baseline stores a time axis or a launch step, and
post_launch_positionsre-zeroes the origin at the first finite row, so the rocket trajectory carries no
information about when the flight happened. Displacing the whole flight later
while keeping the total step count passed every test up to 60 steps, 0.60 s, on
scenario 0 and 62 steps on scenario 1. What stopped it there was the downsampled
row count drifting outside its tolerance, not any check on time.
Scenario 0 is partly protected because
pop_stepis an absolute step index, sodisplacing the pops with the flight brings the largest passing shift to zero.
Scenario 1 pops nothing, so it had no absolute anchor for the rocket at all
beyond the two steps
STEP_COUNT_ABS_TOLallows. That matters there more thananywhere: scenario 1 compares two arrays on two different clocks, the rocket
launch-relative and the balloons absolute from step 0, and whether a pop happens
depends on their relative phase.
Both baselines now record the launch step and both tests assert it before
comparing positions. The largest displacement that passes drops from 60 and 62
to 2, which is
STEP_COUNT_ABS_TOLitself.A test oracle that recomputed the code it was checking
test_scenario1_regression.pyderived the release spacing asint(release_interval / time_step)and its docstring called that "theindependent half, derived from the scenario file, not read back from the
environment". It is the identical float division production runs, so the test
could not detect the error class it advertised. With the truncation made visible:
an interval of 0.3 s at a 0.1 s step gives 2 in production and 2 in the oracle
where 3 is correct, and the test passes. Same at 0.7 and at 2.3.
Production now rounds, through a named
_release_spacing_in_steps, and theoracle compares in seconds so it never repeats that division. The two shipped
scenarios are unaffected: 1 / 0.01 and 0.5 / 0.01 are exact, and every committed
baseline number is byte for byte what it was.
On the baselines
Running the regenerators produced a diff on every trajectory digit, up to
0.694 mm. That is not from this change. Dumping all eight arrays with the fixed
production code and again with
balloon_world.pychecked out from HEAD givesbit-identical results, so the rounding switch moves nothing here; the committed
baselines were simply taken on a different build, 700x inside the 0.5 m
tolerance. The regenerated files were discarded and the new field inserted at
the position the regenerator puts it, so the baseline diff is exactly one added
line per file with no existing byte touched.
Verification
The release-spacing test was landed against the original truncating body first
and failed there,
22 != 23. Six mutations each fail a named test, includingboth baseline values and the rounding itself. Two controls survive, one of them
the rounding change measured against the shipped-values test, which proves the
claim that it is latent for the shipped scenarios rather than asserting it.
No tolerance was weakened.
ruff checkandruff format --checkpass on thepaths CI uses,
uv lock --checkis unchanged,pytest tests/ -qgives 301passed and
BPC_RUN_SLOW_TESTS=1gives 321 passed.Noted, not changed
_release_spacing_in_stepsrounds half to even for an interval that is exactlyhalf a step, such as 0.25 s at 0.1 s. A scenario like that is arguably a
configuration error worth rejecting outright, which is a rules decision rather
than a test fix.