fix: detect spec changes in ScheduledSparkApplication controller

[puwun]

Fixes #2555

Purpose of this PR

The ScheduledSparkApplication controller ignores spec changes (e.g. schedule, timeZone) once in ScheduleStateScheduled, and provides no recovery from ScheduleStateFailedValidation.

Proposed changes:

• Add status.observedGeneration to detect spec changes via the standard Kubernetes generation pattern
• Recalculate status.nextRun when a spec change is detected in ScheduleStateScheduled
• Reset FailedValidation to New on spec change, allowing re-validation
• Add two regression tests covering both scenarios

Change Category

• Bugfix (non-breaking change which fixes an issue)

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chenyi015 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

This PR addresses issue #2555 by adding spec-change detection to the ScheduledSparkApplication controller using the standard Kubernetes generation / status.observedGeneration pattern, so schedule-related spec updates are re-applied after the resource is already scheduled and after validation failures.

Changes:

• Add status.observedGeneration to ScheduledSparkApplicationStatus and CRD schemas.
• Update the controller reconcile loop to detect spec changes and (a) recalculate status.nextRun while scheduled, and (b) reset FailedValidation back to New on spec updates.
• Add regression tests for schedule-change recalculation and recovery from FailedValidation after a spec fix.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/controller/scheduledsparkapplication/controller.go	Implements observedGeneration-based spec change detection and nextRun recalculation / FailedValidation recovery.
internal/controller/scheduledsparkapplication/controller_test.go	Adds regression tests covering schedule change and FailedValidation recovery scenarios.
api/v1beta2/scheduledsparkapplication_types.go	Adds ObservedGeneration to the status type.
config/crd/bases/sparkoperator.k8s.io_scheduledsparkapplications.yaml	Updates CRD schema to include status.observedGeneration.
charts/spark-operator-chart/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml	Mirrors the CRD schema update for Helm chart distribution.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.