Skip to content

Fix percentile/probability message key collisions for PDT 9 and 10 - #131

Merged
mpiannucci merged 14 commits into
mpiannucci:mainfrom
jbeezley:feature/fix-probability-percentile-keys
Mar 20, 2026
Merged

Fix percentile/probability message key collisions for PDT 9 and 10#131
mpiannucci merged 14 commits into
mpiannucci:mainfrom
jbeezley:feature/fix-probability-percentile-keys

Conversation

@jbeezley

Copy link
Copy Markdown
Contributor

This contains a few edge cases related to obscure GRIB2 templates including probabilities and percentiles from ensemble forecasts.

@jbeezley
jbeezley force-pushed the feature/fix-probability-percentile-keys branch from e98dd97 to c992e16 Compare March 19, 2026 18:03
@jbeezley

Copy link
Copy Markdown
Contributor Author

I had to make some editorial decisions on how to structure the xarray dataset. Here are the conventions I chose, let me know if you would like to modify anything.

Percentile (PDT 10)

Messages with the same variable but different percentile values are joined along a percentile coordinate dimension.

ds.tmp                    # (time, percentile, latitude, longitude)
ds.percentile             # [1, 50, 99]

ds.tmp.sel(percentile=50) # (time, latitude, longitude)

Probability — split by type (PDT 9)

Messages with different ProbabilityType values become separate variables with the type abbreviation appended.

ds.tmp_prob_abnorm        # P(above normal)   → (time, latitude, longitude)
ds.tmp_prob_bnorm         # P(below normal)   → (time, latitude, longitude)
ds.tmp_prob_nnorm         # P(near normal)    → (time, latitude, longitude)
ds.tmp_prob_below         # P(X < limit)      → (time, latitude, longitude)
ds.tmp_prob_above         # P(X > limit)      → (time, latitude, longitude)
ds.tmp_prob_between       # P(lo < X < hi)    → (time, latitude, longitude)
ds.tmp_prob_between_inc   # P(lo ≤ X ≤ hi)    → (time, latitude, longitude)

# Each carries a probability_type attribute:
ds.tmp_prob_abnorm.attrs['probability_type']
# → "Probability of event in above normal category"

Probability — threshold dimension (PDT 9, same type, varying limits)

When multiple messages share the same variable and probability type but differ by threshold limits, they are joined along a threshold coordinate.

ds.tmp_prob_between_inc   # (time, threshold, latitude, longitude)
ds.threshold              # [1.0, 3.0, 10.0]

ds.tmp_prob_between_inc.sel(threshold=3.0)  # (time, latitude, longitude)

@jbeezley
jbeezley force-pushed the feature/fix-probability-percentile-keys branch 3 times, most recently from 0f2c8ea to aad426d Compare March 19, 2026 18:18
@jbeezley

Copy link
Copy Markdown
Contributor Author

Another bug fix for probability thresholds. It's a little ugly, but avoids tuples in coordinates, which always causes issues.

 # Each (lower, upper) pair is a separate variable
 ds.tmp_prob_between_inc_1_3   # (time, lat, lon) — lower tercile
 ds.tmp_prob_between_inc_1_5   # (time, lat, lon) — lower quintile
 ds.tmp_prob_between_inc_1_10  # (time, lat, lon) — lower decile
 ds.tmp_prob_between_inc_3_3   # (time, lat, lon)
 ds.tmp_prob_between_inc_5_5   # (time, lat, lon)
 ds.tmp_prob_between_inc_10_10 # (time, lat, lon)

Single-limit types still use the threshold dimension:

# P(X < threshold) with varying thresholds → threshold dim
ds.tmp_prob_below  # (time, threshold, lat, lon) with threshold=[273, 283, 293]

@mpiannucci

Copy link
Copy Markdown
Owner

Just some conflicts since i merged the others first

jbeezley and others added 14 commits March 20, 2026 08:28
…ions

S2S statistical GRIB2 files contain probability (PDT 9) and percentile
(PDT 10) messages that share the same variable/level/time but differ by
probability type, limits, or percentile value. Without this info in the
message key, scan_message_metadata silently drops duplicates — reading
only 203 of 275 messages.

Changes:
- Add ProbabilityType variants 5-10 (S2S uses types 6 and 10)
- Add percentile_value, probability_type, forecast_probability_number,
  probability_lower_limit, probability_upper_limit to ProductTemplate trait
- Implement these for PDT 5, 9, and 10 templates
- Include :pctl{N} and :probt{type}_{lower}_{upper} in message key
- Add new fields to MessageMetadata

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add test fixture (1.6KB, 8 messages) with constant values covering
PDT 9 (probability), PDT 10 (percentile), and PDT 12 (derived
ensemble) on a 0.5° global grid. Verifies:
- All 8 messages parse successfully
- Correct product template IDs
- Percentile values (1, 50, 99) on PDT 10 messages
- Probability type and limits on PDT 9 messages
- All messages produce unique keys (no collisions)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 2 PDT 9 messages with varying probability limits (lower=3,upper=3
and lower=10,upper=10) to the S2S test fixture. Together with the
existing message (lower=1,upper=10), these exercise the threshold
coordinate dimension with 3 distinct values [1, 3, 10].

Update Rust test to verify exact probability limit values and the new
messages. Update Python test to assert threshold coordinate exists with
expected values.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…llisions

Between-type probabilities (BetweenLimits, BetweenLimitsInclusive) with
different (lower, upper) pairs were colliding on the threshold dimension
because the threshold key only used lower_limit. Messages like (1,3),
(1,5), (1,10) all mapped to threshold=1.

Fix: include both limits in the variable hash for between-type
probabilities, splitting each unique (lower, upper) pair into a
separate variable. The threshold dimension is now only used for
single-limit types (e.g., P(X < threshold)) where the threshold
space is genuinely 1D.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jbeezley
jbeezley force-pushed the feature/fix-probability-percentile-keys branch from 5062b36 to aae5a4d Compare March 20, 2026 14:30
@jbeezley

Copy link
Copy Markdown
Contributor Author

Thanks! Rebased

@mpiannucci
mpiannucci merged commit 1629092 into mpiannucci:main Mar 20, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants