Fix percentile/probability message key collisions for PDT 9 and 10 - #131
Conversation
e98dd97 to
c992e16
Compare
|
I had to make some editorial decisions on how to structure the xarray dataset. Here are the conventions I chose, let me know if you would like to modify anything. Percentile (PDT 10)Messages with the same variable but different percentile values are joined along a ds.tmp # (time, percentile, latitude, longitude)
ds.percentile # [1, 50, 99]
ds.tmp.sel(percentile=50) # (time, latitude, longitude)Probability — split by type (PDT 9)Messages with different ds.tmp_prob_abnorm # P(above normal) → (time, latitude, longitude)
ds.tmp_prob_bnorm # P(below normal) → (time, latitude, longitude)
ds.tmp_prob_nnorm # P(near normal) → (time, latitude, longitude)
ds.tmp_prob_below # P(X < limit) → (time, latitude, longitude)
ds.tmp_prob_above # P(X > limit) → (time, latitude, longitude)
ds.tmp_prob_between # P(lo < X < hi) → (time, latitude, longitude)
ds.tmp_prob_between_inc # P(lo ≤ X ≤ hi) → (time, latitude, longitude)
# Each carries a probability_type attribute:
ds.tmp_prob_abnorm.attrs['probability_type']
# → "Probability of event in above normal category"Probability — threshold dimension (PDT 9, same type, varying limits)When multiple messages share the same variable and probability type but differ by threshold limits, they are joined along a ds.tmp_prob_between_inc # (time, threshold, latitude, longitude)
ds.threshold # [1.0, 3.0, 10.0]
ds.tmp_prob_between_inc.sel(threshold=3.0) # (time, latitude, longitude) |
0f2c8ea to
aad426d
Compare
|
Another bug fix for probability thresholds. It's a little ugly, but avoids tuples in coordinates, which always causes issues. # Each (lower, upper) pair is a separate variable
ds.tmp_prob_between_inc_1_3 # (time, lat, lon) — lower tercile
ds.tmp_prob_between_inc_1_5 # (time, lat, lon) — lower quintile
ds.tmp_prob_between_inc_1_10 # (time, lat, lon) — lower decile
ds.tmp_prob_between_inc_3_3 # (time, lat, lon)
ds.tmp_prob_between_inc_5_5 # (time, lat, lon)
ds.tmp_prob_between_inc_10_10 # (time, lat, lon)Single-limit types still use the threshold dimension: # P(X < threshold) with varying thresholds → threshold dim
ds.tmp_prob_below # (time, threshold, lat, lon) with threshold=[273, 283, 293] |
|
Just some conflicts since i merged the others first |
…ions
S2S statistical GRIB2 files contain probability (PDT 9) and percentile
(PDT 10) messages that share the same variable/level/time but differ by
probability type, limits, or percentile value. Without this info in the
message key, scan_message_metadata silently drops duplicates — reading
only 203 of 275 messages.
Changes:
- Add ProbabilityType variants 5-10 (S2S uses types 6 and 10)
- Add percentile_value, probability_type, forecast_probability_number,
probability_lower_limit, probability_upper_limit to ProductTemplate trait
- Implement these for PDT 5, 9, and 10 templates
- Include :pctl{N} and :probt{type}_{lower}_{upper} in message key
- Add new fields to MessageMetadata
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add test fixture (1.6KB, 8 messages) with constant values covering PDT 9 (probability), PDT 10 (percentile), and PDT 12 (derived ensemble) on a 0.5° global grid. Verifies: - All 8 messages parse successfully - Correct product template IDs - Percentile values (1, 50, 99) on PDT 10 messages - Probability type and limits on PDT 9 messages - All messages produce unique keys (no collisions) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 2 PDT 9 messages with varying probability limits (lower=3,upper=3 and lower=10,upper=10) to the S2S test fixture. Together with the existing message (lower=1,upper=10), these exercise the threshold coordinate dimension with 3 distinct values [1, 3, 10]. Update Rust test to verify exact probability limit values and the new messages. Update Python test to assert threshold coordinate exists with expected values. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…llisions Between-type probabilities (BetweenLimits, BetweenLimitsInclusive) with different (lower, upper) pairs were colliding on the threshold dimension because the threshold key only used lower_limit. Messages like (1,3), (1,5), (1,10) all mapped to threshold=1. Fix: include both limits in the variable hash for between-type probabilities, splitting each unique (lower, upper) pair into a separate variable. The threshold dimension is now only used for single-limit types (e.g., P(X < threshold)) where the threshold space is genuinely 1D. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5062b36 to
aae5a4d
Compare
|
Thanks! Rebased |
This contains a few edge cases related to obscure GRIB2 templates including probabilities and percentiles from ensemble forecasts.