Builds on github reporting http 429

[dweiss]

Description

There have been failures on certain builds on github recently:
https://github.com/apache/lucene/actions/workflows/run-nightly-smoketester.yml

They mostly fail because of request ratio limits when resolving references. This must be something new (?). I am also a bit puzzled at the address these requests get resolved from: repo.maven.apache.org (?).

Something to look into.

* What went wrong:
Execution failed for task ':build-infra:compileJava'.
> Could not resolve all files for configuration ':build-infra:compileClasspath'.
   > Could not download org.eclipse.jgit-7.5.0.202512021534-r.jar (org.eclipse.jgit:org.eclipse.jgit:7.5.0.202512021534-r)
      > Could not get resource 'https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/7.5.0.202512021534-r/org.eclipse.jgit-7.5.0.202512021534-r.jar'.
         > Could not GET 'https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/7.5.0.202512021534-r/org.eclipse.jgit-7.5.0.202512021534-r.jar'. Received status code 429 from server: Too Many Requests
   > Could not download randomizedtesting-runner-2.8.3.jar (com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.3)
      > Could not get resource 'https://repo.maven.apache.org/maven2/com/carrotsearch/randomizedtesting/randomizedtesting-runner/2.8.3/randomizedtesting-runner-2.8.3.jar'.
         > Could not GET 'https://repo.maven.apache.org/maven2/com/carrotsearch/randomizedtesting/randomizedtesting-runner/2.8.3/randomizedtesting-runner-2.8.3.jar'. Received status code 429 from server: Too Many Requests
   > Could not download zstd-jni-1.5.7-6.jar (com.github.luben:zstd-jni:1.5.7-6)
      > Could not get resource 'https://repo.maven.apache.org/maven2/com/github/luben/zstd-jni/1.5.7-6/zstd-jni-1.5.7-6.jar'.
         > Could not GET 'https://repo.maven.apache.org/maven2/com/github/luben/zstd-jni/1.5.7-6/zstd-jni-1.5.7-6.jar'. Received status code 429 from server: Too Many Requests
   > Could not download software-and-algorithms-1.0.jar (com.github.kevinstern:software-and-algorithms:1.0)
      > Could not get resource 'https://repo.maven.apache.org/maven2/com/github/kevinstern/software-and-algorithms/1.0/software-and-algorithms-1.0.jar'.
         > Could not GET 'https://repo.maven.apache.org/maven2/com/github/kevinstern/software-and-algorithms/1.0/software-and-algorithms-1.0.jar'. Received status code 429 from server: Too Many Requests

[dweiss]

ok, both repo1.maven.org and repo.maven.apache.org point to the same content at cloudflare. Those limits must be something new at cloudflare then (or maybe they started throttling github runners?).

[dweiss]

The only thing that comes to my mind is using an alternative mirror like https://maven-central.storage-download.googleapis.com/maven2/. My guess is github runners have a small set of outbound IPs and those 429 errors will only happen more and more frequently (we do have artifact caching but whenever deps change, it's potentially a problem).

[prudhvigodithi]

Along with mavenCentral() we can declare the mirror as if I'm not wrong Gradle resolves dependencies by trying repositories in the order they're declared. Example https://github.com/apache/spark-kubernetes-operator/blob/main/build.gradle#L64-L72. To begin with we can update the following files:

build-tools/build-infra-shadow/build.gradle
build-tools/build-infra/build.gradle
settings.gradle

[prudhvigodithi]

I have created a draft PR #15542, Let me know if ok to try with this alternate mirror and observe the nightlies.

[dweiss]

Ok, https://github.com/apache/lucene/actions/runs/20619145757/job/59217574234 passed. I'll keep an eye on that.

[rmuir]

@dweiss I'll try to find some time to look into the caching too.

often times it is configured wrong, gets trashed by PRs, or invalidates entirely if a single dep changes.

I use this pattern for other programming languages (example here: python/uv) at work:

    # restore exact or inexact cache for pull requests and merge queue. do not save.
    - name: Read-only package cache
      if: ${{ contains(fromJSON('["pull_request", "merge_group"]'), github.event_name) }}
      uses: actions/cache/restore@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1
      with:
        path: ~/.cache/uv
        key: pypi-${{ github.job }}-${{ inputs.subdirectory }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.cache-key.outputs.hash }}
        restore-keys: pypi-${{ github.job }}-${{ inputs.subdirectory }}-${{ runner.os }}-${{ runner.arch }}-

    # save exact cache on the default branch
    - name: Read-write package cache
      if: ${{ !contains(fromJSON('["pull_request", "merge_group"]'), github.event_name) }}
      uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1
      with:
        path: ~/.cache/uv
        key: pypi-${{ github.job }}-${{ inputs.subdirectory }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.cache-key.outputs.hash }}

[dweiss]

I believe we already have fine-grained caching in place; this should be handled here:

https://github.com/apache/lucene/blob/main/.github/actions/prepare-for-build/action.yml#L40-L47

The problems I've seen all apply to the "nightly" smoke tester check where we deliberately do not use any caches (to ensure all artifacts and resources are indeed available, have the same checksums, etc.).

https://github.com/apache/lucene/blob/main/.github/workflows/run-nightly-smoketester.yml

This has been working fine for quite some time now (and caught a few regressions). It is only recently that it started failing on dependency resolution.

[dweiss]

Nope, still an issue -
https://github.com/apache/lucene/actions/runs/20686612348

smoke-tester-logs-jdk-25.zip

Could be that gradle fetches part of the descriptor (pom file, for example), then the repo for that artifact becomes "sticky" but an attempt to download the rest of it fails. Don't know how to make it more verbose in terms of where it attempts to download artifacts from.

[prudhvigodithi]

Nope, still an issue

Looks like this is same issue but from different module, coming from #15541 (comment) we should extend this across all the code. This time I see the error is for lucene:spatial-extras https://github.com/apache/lucene/tree/main/lucene/spatial-extras.

Seeing the Gradle code updating the https://github.com/apache/lucene/blob/main/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/misc/LuceneRootConfigurationPlugin.java#L117 should apply to all projects.

Let me take a look at applyCommonConfiguration() to see if we can add a alternate maven for across the projects.