GH Action schedule failed to create promotion ticket

[achave11-ucsc]

On Tuesday May 19 the schedule job that creates the ticket did not run.

The promotion issue becomes eligible, and only during a one hour window, to be created on Tuesdays at 9 AM. Because the runs for 9 AM were dropped, the ticket wasn't created.

Using a script created by Claude to analyze the full hours missed since the end of December last year, it appears that this started happening at the end of March this year. After which we started missing a single full hour run almost daily between 21:00 & 22:00 PT. However, since the beginning of May missing full hour runs has become more exacerbated, with up to four runs (05/18), two/three commonly, being dropped.

❯ python .github/workflows/analyze_schedule_misses.py --limit 7500
Fetching up to 7500 runs for schedule.yaml …
Fetched 7500 runs.
Range:          2025-12-27T04:00Z → 2026-05-21T16:00Z
Total runs fetched:       7500
Complete hours analysed:  3492
  All 3/3 runs fired:     1606  (46.0%)
  Partially missed (1-2):  1830  (52.4%)
  Fully missed (0 runs):   56  (1.6%)

Fully missed hours (56):
  2026-03-30T04:00Z
  2026-04-01T04:00Z
  2026-04-06T04:00Z
  2026-04-10T04:00Z
  2026-04-12T04:00Z
  2026-04-13T04:00Z
  2026-04-14T04:00Z
  2026-04-15T04:00Z
  2026-04-16T04:00Z
  2026-04-17T04:00Z
  2026-04-19T04:00Z
  2026-04-20T04:00Z
  2026-04-21T04:00Z
  2026-04-22T04:00Z
  2026-04-23T04:00Z
  2026-04-24T04:00Z
  2026-04-26T04:00Z
  2026-04-27T04:00Z
  2026-04-28T05:00Z
  2026-04-29T05:00Z
  2026-04-30T05:00Z
  2026-05-01T00:00Z
  2026-05-01T05:00Z
  2026-05-02T00:00Z
  2026-05-02T04:00Z
  2026-05-03T00:00Z
  2026-05-03T03:00Z
  2026-05-04T00:00Z
  2026-05-04T05:00Z
  2026-05-05T04:00Z
  2026-05-06T04:00Z
  2026-05-07T00:00Z
  2026-05-07T05:00Z
  2026-05-10T00:00Z
  2026-05-10T05:00Z
  2026-05-11T04:00Z
  2026-05-11T09:00Z
  2026-05-12T05:00Z
  2026-05-13T05:00Z
  2026-05-14T05:00Z
  2026-05-15T05:00Z
  2026-05-16T05:00Z
  2026-05-17T05:00Z
  2026-05-18T04:00Z
  2026-05-18T07:00Z
  2026-05-18T10:00Z
  2026-05-18T14:00Z
  2026-05-19T04:00Z
  2026-05-19T08:00Z
  2026-05-19T16:00Z
  2026-05-20T04:00Z
  2026-05-20T08:00Z
  2026-05-20T16:00Z
  2026-05-21T04:00Z
  2026-05-21T07:00Z
  2026-05-21T12:00Z

[achave11-ucsc]

@hannes-ucsc: "Spike to provide symptoms and estimate how many times in the past we missed a full hours of schedule runs."

[achave11-ucsc]

As proposed by @hannes-ucsc, a straight forward solution is to simply run the schedule more aggressively, every five minutes.

diff --git a/.github/workflows/schedule.yaml b/.github/workflows/schedule.yaml
index 9c6076cfd..891bedd4e 100644
--- a/.github/workflows/schedule.yaml
+++ b/.github/workflows/schedule.yaml
@@ -9,12 +9,12 @@ on:
     #
     # Additionally, GitHub documents that scheduled events can be delayed during
     # periods of high load, or even skipped. To reduce the chances of these
-    # restrictions impacting our schedule, we run the script more than once per
-    # hour.
+    # restrictions impacting our schedule, we run the script every 5 minutes
+    # (12 times per hour).
     #
     # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule
     #
-    - cron: '3,23,43 * * * *'
+    - cron: '*/5 * * * *'
 
 jobs:
   schedule:

[achave11-ucsc]

As a more involved but reliable alternative, we may migrate the creation of tickets on a schedule to run in GitLab.
The self-hosted GitLab instance fires pipeline schedules reliably at the hour (eliminating the need for the multi-run-per-hour workaround currently in place).

The migration would require the following changes:

• Add a new lightweight job to .gitlab-ci.yml (with only Python and the gh CLI), the .base one is over equipped for this task.
• A new GitLab pipeline schedule (0 * * * *) with a new $azul_gitlab_schedule value, similar to fedramp_inventory and sell_unused_slots.
• A new CI variable to GitLab: GITHUB_TOKEN (the same token already employed by GitHub Actions), we already have a mechanism to determine GITHUB_REPO (e.g. DataBiosphere/azul).
• Retire .github/workflows/schedule.yaml from the repository, since the GitHub Actions scheduler would otherwise continue to run in parallel.

[dsotirho-ucsc]

@hannes-ucsc: "Assignee to implement the easy solution (increasing the schedule frequency to 12 times an hour). For demo, rerun the script one week after this was merged."