Skip to content

Merged forecast step fixes and SSL fixes #1690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
clessig wants to merge 95 commits into
base: clessig/sophiex/dev/include-reg-tokens-in-query-agg-engine
Choose a base branch
from
Draft

Merged forecast step fixes and SSL fixes #1690

clessig wants to merge 95 commits into from

Conversation

@clessig
Copy link
Collaborator

@clessig clessig commented Jan 25, 2026

Description

Draft branch that contains all current fixes for forecasting and SSL training

Issue Number

See #1610 and #1656

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

kctezcan and others added 30 commits January 13, 2026 08:31
@kctezcan
WIP added a predictor class
bc19177
@kctezcan
using the transformer predictor for jepa
f3d81b8
@kctezcan
lint
d3fc692
@kctezcan
Merge branch 'develop' into ktezcan/dev/iss1587_predictor_jepa
0d23560
@kctezcan
added pred_ params in the test config
37145b3
@kctezcan
renamed params to sslpred_
3e9372b
@kctezcan
merged develop
1dcd781
@kctezcan
lint
c845509
@kctezcan
Merge branch 'develop' into ktezcan/dev/iss1587_predictor_jepa
3a00e40
@kctezcan
added the only jepa config
2048d16
@sophie-xhonneux
Merge branch 'develop' into sophiex/kerem/pr/transformer-head
bca9097
@sophie-xhonneux
Clean-up config and model create
399a97e
@sophie-xhonneux
Current status
aa2e34a
@sophie-xhonneux
Now it works
f0dac8c
@Jubeku
first-pass fixing fsteps
815ae67
@clessig
Allows for writing normalized samples; fixed config to keep it well-s…
b263857 
…tructured (#1653)
@s6sebusc @iluise
Skipping missing scores in JSONreader (#1655)
2abc7df 
* split WeatherGenReader functionality to allow reading only JSON

adding weathergen JSON reader to develop

* informative error when metrics are not there

* restore JSONreader after rebase

* JSONreader mostly restored

* MLFlow logging independent of JSON/zarr

* linting, properly cheking fsteps, ens, samples in JSONreader

* tiny change to restore the MergeReader

* lint

* enabling JSONreader to skip plots and missing scores gracefully

* required reformatting

* move skipping of metrics to the reader class

* slighly more explicit formulations

---------

Co-authored-by: Sebastian Buschow <[email protected]>
Co-authored-by: Sebastian Buschow <[email protected]>
Co-authored-by: iluise <[email protected]>
Co-authored-by: Ilaria Luise <[email protected]>
@Jubeku
correct forecast loop condition
605b91c
@Jubeku
update ModelOutput definition
a3860ec
@Jubeku
rm breakpoint
075ccc1
@kctezcan @sophie-xhonneux
WIP added a predictor class
08ace79
@kctezcan
pred_blocks -> blocks
590b1c6
@kctezcan
added assert False for unknown prediction head
b3fadd4
@kctezcan
using dist.get_node_local_rank()
5d38d57
@kctezcan
'-1'-> -1
7dd64ae
@kctezcan
rename class_token -> use_class_token
925875d
@kctezcan
updated LatentPredictionHeadMLP
d6688ba
@kctezcan
removed print() statements
7df3a28
@sophie-xhonneux
using the transformer predictor for jepa
699027d
@kctezcan @sophie-xhonneux
lint
6145f35
Jubeku and others added 23 commits January 22, 2026 16:13
@Jubeku
correct loop in dataloader and lint
efe8ead
@Jubeku
Merge branch 'develop' into jk/develop/1654_fix_fsteps
1717a7e
@sophie-xhonneux
Address comments
82fe54c
@sophie-xhonneux
Add ddp_find_unused_parameters to physical/jepa cfg
42d59ca
@clessig
More interface cleanup
0e43227
@clessig
Merge branch 'sophiex/dev/include-reg-tokens-in-query-agg-engine' of …
0bab6cf 
…github.com:ecmwf/WeatherGenerator into sophiex/dev/include-reg-tokens-in-query-agg-engine
@sophie-xhonneux
Fix small bugs for DINOv2
3caab0e 
I think it still hangs in multi-GPU mode even with just DDP :/
@clessig
Re-enabled target_and_aux_calculators_val
fc30b2b
@clessig
Minor fixes to integration test, in particular making it self-contain…
75db40e 
…ed and removing dependence on default_config.yml
@clessig
Merge branch 'sophiex/dev/include-reg-tokens-in-query-agg-engine' of …
c41afcd 
…github.com:ecmwf/WeatherGenerator into sophiex/dev/include-reg-tokens-in-query-agg-engine
@clessig
Fixing linting
2160dc9
@clessig
Merge branch 'develop' into sophiex/dev/include-reg-tokens-in-query-a…
ae09442 
…gg-engine
@Jubeku
update forecast policy and step validation
726bab5
@Jubeku
add mode to forecast config validation output
d06c850
@clessig
Fixed forecast config testing
b5e1a49
@clessig
Simplified and clarified forecast loop. Switched from forecast* to ou…
4ac50b8 
…tput* to be more general. Cleaned up various details.
@clessig
Merge branch 'sophiex/dev/include-reg-tokens-in-query-agg-engine' of …
f7e06bf 
…github.com:ecmwf/WeatherGenerator into clessig/jk/develop/1654_fix_fsteps
@clessig
Fixes to merge
c4a42b6
@clessig
Re-enabled support for missing or incomplete forecast config. Improve…
bd8625c 
…d some variable naming
@clessig
Fixed bug in definition of class token indices
068d3d3
@clessig
Fixed bug with aliased variable naming
4649d1a
@clessig
Fixed outdated model forward call
221059f
@clessig
Merge branch 'clessig/jk/develop/1654_fix_fsteps' of github.com:ecmwf…
0d8d36e 
…/WeatherGenerator into clessig/jk/develop/1654_1608_fix_fsteps_ssl
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

WeatherGen-dev
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@clessig @kctezcan @sophie-xhonneux @Jubeku @s6sebusc