Open
Conversation
…py, save.py, and related tests
…anced functionality in save.py and tests
…e clarity and maintainability
…ved mocking in tests
…ward compatibility; remove outdated test files
…mprove forecast writing logic
…o enable ME value retrieval
…_forecast functionality
Comment out the load_dotenv function call to prevent loading environment variables from .env.local.
Removed unused dotenv import statement.
| metadata = Struct(fields={"legacy_uuid": Value(string_value=legacy_uuid)}) | ||
|
|
||
| req = dp.CreateLocationRequest( | ||
| location_name=f"synced_site_{legacy_uuid[:8]}", |
Contributor
There was a problem hiding this comment.
get name from legacy_sites
Contributor
There was a problem hiding this comment.
this should work if all names are unique
| try: | ||
| await save_forecast_to_dataplatform( | ||
| forecast_df=forecast_values_df, | ||
| location_uuid=UUID(str(forecast_meta["location_uuid"])), |
Contributor
There was a problem hiding this comment.
check if we need str here
| torchvision = [ | ||
| { index = "pytorch-cpu" }, | ||
| ] | ||
| dp-sdk = { url = "https://github.com/openclimatefix/data-platform/releases/download/v0.16.0/dp_sdk-0.16.0-py3-none-any.whl" } |
…extract adjustment logic into a helper, and add a timestamp utility function.
…eat/dataplatform-save
…t, removing capacity-based limiting, and streamlining timezone conversion.
…ing forecasts to the platform
Comment on lines
+207
to
+217
| # DP validates version with a restricted charset; normalize local package versions | ||
| # (which may contain e.g. '+' build metadata) to a compatible value. | ||
| app_version = "".join( | ||
| ch if (ch.isalnum() or ch in "._-") else "." | ||
| for ch in raw_version.lower() | ||
| ) | ||
| while ".." in app_version: | ||
| app_version = app_version.replace("..", ".") | ||
| app_version = app_version.strip("._-") | ||
| if len(app_version) < 2: | ||
| app_version = "0.0" |
Contributor
There was a problem hiding this comment.
Im not sure we need this, we just know the version needs to be X.Y.Z
…ing forecasts to the platform.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces the implementation for saving generated forecasts to the Data Platform.
The forecast saving logic has been added and integrated into the application flow to ensure forecasts are persisted after generation. The changes improve the overall pipeline by enabling reliable downstream consumption of forecast data.
Key changes include:
Implemented forecast saving to the Data Platform
Integrated save logic into the main forecast execution flow
Refactored related code for better structure and readability
Added necessary dependency updates
Added unit and integration tests to validate save behaviour
Improved error handling around the save process
Fixes #163
How Has This Been Tested?
Added unit tests covering the forecast save logic
Added integration tests to verify forecasts are correctly constructed and sent to the Data Platform
Ran the application locally to confirm:
Forecast generation works as expected
Save calls are triggered successfully
No runtime or validation errors occur
Performed a basic sanity check to confirm forecast values are correctly passed through the save pipeline
Checklist: