Feature/add nighttime zeros issue #104 by Sagarpillai · Pull Request #110 · openclimatefix/solar-consumer

Sagarpillai · 2025-11-03T13:55:07Z

Pull Request

Changes made:

Added import for make_night_time_zeros from pvliveconsumer.nightime
Added function call in the PVLive data processing loop to fill nighttime gaps
Added pvliveconsumer dependency to pyproject.toml
Fixes #104

Sagarpillai · 2025-11-03T17:23:17Z

Hi @peterdudfield
quick note: I originally tried to call the function directly from pvlive-consumer, but CI fails because that package isn’t on PyPI. So I’ve just added a small local helper file for the nighttime-zero logic (same function, just vendored here).

If you’d prefer this to live in a different folder or a different name, let me know and I’ll move it.

peterdudfield · 2025-11-03T17:30:00Z

solar_consumer/data/nighttime.py

@@ -0,0 +1,43 @@
+# solar_consumer/pvlive/nighttime.py


if possible, could you put this under the data folder

peterdudfield · 2025-11-03T17:30:43Z

solar_consumer/data/nighttime.py

+    at_night = (df["target_time_utc"] < df["sunrise_utc"]) | (df["target_time_utc"] >= df["sunset_utc"])
+    df.loc[at_night, "generation_mw"] = 0.0
+
+    return df


could you also copy any tests across from PVLive consumer?

peterdudfield · 2025-11-03T17:35:34Z

Thank you btw for doing this

Sagarpillai · 2025-11-03T19:04:11Z

@peterdudfield, sure, I've made the changes, please let me know if everything's okay

peterdudfield · 2025-11-05T09:41:24Z

solar_consumer/data/nighttime.py

+
+    # Build mask: night is before sunrise OR at/after sunset
+    at_night = (df[ts_col] < df[sunrise_col]) | (df[ts_col] >= df[sunset_col])
+    df.loc[at_night, mw_col] = 0.0


I dont remeber these sunrise and sunset columns? How was this done in PVLive-Consumer?

…hookup

Sagarpillai · 2025-11-05T11:23:49Z

@peterdudfield ,
I switched to pvlib solar elevation threshold (<5°) which removes the sunrise/sunset columns entirely and keeps the logic simpler. Test added. All feedback addressed, and CI is green. Ready for re-review.

peterdudfield · 2025-11-05T11:55:39Z

solar_consumer/data/fetch_gb_data.py

+# Prefer loading locations from the canonical datamodel; fall back to empty frame
+try:
+    from nowcasting_datamodel.read.gsp import get_all_locations  # type: ignore
+    _GSP_LOCATIONS = get_all_locations().set_index("gsp_id")


sorry, could we not use nowcasting_datamodel, i think copying in a csv of the centeroid of the gsps would be fine

sure I'll change that

peterdudfield · 2025-11-05T11:56:13Z

solar_consumer/data/nighttime.py

+        Copy of df with nighttime rows set to zero in `mw_col`.
+    """
+    try:
+        import pvlib  # lazy import


why have this try and except? I would just import it at the opt

I'll change this as well, I'll update the PR soon, is there an issue with anything else?

…/sunset

Sagarpillai · 2025-11-05T17:23:40Z

@peterdudfield yep, made the changes. Hope everything's good now, please let me know if im missing anything, thank you.

solar_consumer/data/fetch_gb_data.py

Sagarpillai · 2025-11-05T18:54:29Z

@peterdudfield, For the CSV with gsp_id + lat/lon which repo/source would you like me to take the centroid values from?

Once I know the preferred source, I’ll copy those values into solar_consumer/data/uk_gsp_locations_20250109.csv and push it, for now ive just added

gsp_id,latitude,longitude
0,52.355518,-1.17432

peterdudfield · 2025-11-05T18:59:18Z

@peterdudfield, For the CSV with gsp_id + lat/lon which repo/source would you like me to take the centroid values from?

Once I know the preferred source, I’ll copy those values into solar_consumer/data/uk_gsp_locations_20250109.csv and push it, for now ive just added

gsp_id,latitude,longitude 0,52.355518,-1.17432

where does it get them from in PVLive-consumer?

Sagarpillai · 2025-11-05T19:11:17Z

@peterdudfield, For the CSV with gsp_id + lat/lon which repo/source would you like me to take the centroid values from?
Once I know the preferred source, I’ll copy those values into solar_consumer/data/uk_gsp_locations_20250109.csv and push it, for now ive just added
gsp_id,latitude,longitude 0,52.355518,-1.17432

where does it get them from in PVLive-consumer?

@peterdudfield yep, got it was confused on copying .csv files, hope this is the one.

peterdudfield · 2025-11-06T07:19:31Z

solar_consumer/data/fetch_gb_data.py

+# Load GSP lat/lon for night-time zeroing from CSV (no datamodel dependency)
+DIR = os.path.dirname(__file__)
+
+def _load_gsp_locations() -> pd.DataFrame | None:


could this be moved to nighttime, as a specific to that?

peterdudfield

This looks great!

I think with one more test, we would be good to go

Sagarpillai · 2025-11-11T07:57:36Z

This looks great!

I think with one more test, we would be good to go

sure, I've added the test to check whether it loads data

peterdudfield · 2025-11-11T08:23:56Z

tests/data/test_nighttime.py

+def test_load_gsp_locations():
+    """Verify the bundled GSP locations CSV loads and contains expected columns."""
+    if GSP_LOCATIONS is None:
+        pytest.skip("GSP locations CSV not present in this environment")


Can you remove this please

We want to make sure the test run

peterdudfield · 2025-11-11T08:24:26Z

tests/data/test_nighttime.py

+def test_make_night_time_zeros_uses_gsp_id_fallback():
+    """Ensure make_night_time_zeros can look up coords using gsp_id and run."""
+    if GSP_LOCATIONS is None:
+        pytest.skip("GSP locations CSV not present in this environment")


remove this too

Sagarpillai · 2025-11-11T15:16:19Z

@peterdudfield Ive removed the skipping tests part, please review it when you get a moment

Sagarpillai · 2025-11-17T12:09:28Z

@peterdudfield any update on this?

…builds start/end time index and includes capacity columns (fix reviewer request)

…e gives empty df (reviewer request)

Sagarpillai · 2025-11-18T12:35:39Z

@peterdudfield
Just pushed the requested changes, nighttime zeroing now only triggers when PVLive returns no rows, and the helper builds the correct time index + capacity columns.
Let me know if you want any more tweaks!

peterdudfield · 2025-11-18T12:49:31Z

solar_consumer/data/fetch_gb_data.py

                      - `Datetime_GMT`: Combined date and time in UTC.
                      - `solar_forecast_kw`: Estimated solar forecast in kW.
    """
-    meta_url = "https://api.neso.energy/api/3/action/datapackage_show?id=embedded-wind-and-solar-forecasts"


why has this changed?

i dont really understand why any of this function has been changed?
(aprt from adding nighttime in)

Sagarpillai · 2025-11-19T13:22:07Z

@peterdudfield , sorry about the mess, I've reverted back to the original code. Please check if everything's okay, thank you

Sagarpillai · 2025-11-25T13:07:17Z

Hey @peterdudfield any updates on this?

peterdudfield · 2025-11-25T14:22:17Z

Sorry for my slow reply. I think im getting a bit confused. Looking back at the issue, #104, this function should add night time zeros, if there is no data provded by PVLive. However looking at this PR, it looks like it changes results. I think we might need to change this.

Sorry i should have said this when you first submiited it

Sagarpillai · 2025-11-26T14:17:35Z

Sorry for my slow reply. I think im getting a bit confused. Looking back at the issue, #104, this function should add night time zeros, if there is no data provded by PVLive. However looking at this PR, it looks like it changes results. I think we might need to change this.

Sorry i should have said this when you first submiited it

Apologies for the misunderstanding, I'll redo it and put up a PR soon.

Sagarpillai · 2025-12-03T16:55:57Z

Hi, I've been travelling and I'm a little sick, I'll make the changes as soon as I can :)

Sagarpillai added 2 commits November 3, 2025 18:43

Add nighttime zeros to PVLive data - fixes openclimatefix#104

5514218

Fix pvliveconsumer dependency - install from git repo

adc90d7

Sagarpillai marked this pull request as draft November 3, 2025 13:56

apply review fixes and vendor nighttime zeros

52e0f38

Sagarpillai marked this pull request as ready for review November 3, 2025 17:18

peterdudfield reviewed Nov 3, 2025

View reviewed changes

Sagarpillai added 2 commits November 4, 2025 00:15

Add nighttime zeroing test (openclimatefix#104)

cd6c422

fix nighttime zeros call + fix test

751bf51

peterdudfield reviewed Nov 5, 2025

View reviewed changes

Sagarpillai added 4 commits November 5, 2025 16:19

feat: use pvlib solar elevation for night-time zeroing + unit test + …

436d6ef

…hookup

style: move imports to top to satisfy Ruff E402

ff0ae9c

feat: pvlib elevation night-time zeroing + test + datamodel gsp lookup

d3357ad

fix: include saved pvlib night-time zeroing + test + deps

ab5b4a3

peterdudfield reviewed Nov 5, 2025

View reviewed changes

Sagarpillai added 2 commits November 5, 2025 22:39

feat: pvlib elevation night-time zeroing (<5°) + test; remove sunrise…

c1cf5a9

…/sunset

fix: robust CSV path lookup for gsp locations

a76a601

peterdudfield reviewed Nov 5, 2025

View reviewed changes

solar_consumer/data/fetch_gb_data.py Outdated Show resolved Hide resolved

chore: add small centroid csv for gsp lookup

e739c0f

added values from pvlive-consumer)

101b845

peterdudfield reviewed Nov 6, 2025

View reviewed changes

peterdudfield reviewed Nov 11, 2025

View reviewed changes

Add tests for GSP location loading and gsp_id fallback in nighttime

574d8b9

Sagarpillai requested a review from peterdudfield November 11, 2025 07:57

peterdudfield reviewed Nov 11, 2025

View reviewed changes

Finalize tests: remove pytest.skip

1ccd28b

Sagarpillai requested a review from peterdudfield November 11, 2025 15:16

Sagarpillai added 2 commits November 18, 2025 17:06

Only synthesize night-time zeros when PVLive returns no rows; helper …

38e7d2d

…builds start/end time index and includes capacity columns (fix reviewer request)

Fix PVLive missing-GSP case: only generate nighttime zeros when PVLiv…

f1b8df8

…e gives empty df (reviewer request)

peterdudfield reviewed Nov 18, 2025

View reviewed changes

Sagarpillai added 8 commits November 18, 2025 18:38

Fix: restore meta_url placement and remove unintended change

a042958

Use local CSV first, fallback to API; fix CI 500 error

618ce62

Fixing CI issues

e066e2e

fixes

a209018

Reverting Neso API call

aaa1734

Merge origin/main; resolve conflict by removing uv.lock

3c35c9e

Ignore uv.lock

87c9b76

Reverting to intial code

6a6939f

Sagarpillai requested a review from peterdudfield November 19, 2025 13:22

This was referenced Feb 12, 2026

PVLive: add nighttime zeros #104

Open

Feat/add pvlive nighttime zeros #160

Open

Uh oh!

Conversation

Sagarpillai commented Nov 3, 2025

Uh oh!

Sagarpillai commented Nov 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdudfield commented Nov 3, 2025

Uh oh!

Sagarpillai commented Nov 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sagarpillai commented Nov 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sagarpillai commented Nov 5, 2025

Uh oh!

Uh oh!

Sagarpillai commented Nov 5, 2025

Uh oh!

peterdudfield commented Nov 5, 2025

Uh oh!

Sagarpillai commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdudfield left a comment

Choose a reason for hiding this comment

Uh oh!

Sagarpillai commented Nov 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sagarpillai commented Nov 11, 2025

Uh oh!

Sagarpillai commented Nov 17, 2025

Uh oh!

Sagarpillai commented Nov 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sagarpillai commented Nov 19, 2025

Uh oh!

Sagarpillai commented Nov 25, 2025

Uh oh!

peterdudfield commented Nov 25, 2025

Uh oh!

Sagarpillai commented Nov 26, 2025

Uh oh!

Sagarpillai commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sagarpillai commented Nov 5, 2025 •

edited

Loading