[Enhancement]: 3rd party mirror (maven / eclipse) throttling issues with jenkins pipelines

[gaiksaya]

Describe the bug

We have been seeing increase in 3rd party repository throttling during 3.7.0 and 3.8.0 releases on jenkins distribution jobs as well as integration testing and gradle checks.
As the release gains momentum during the release windows the issues surface more rather than in nightly builds.

To reproduce

See below workflows:
https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-build-opensearch/detail/distribution-build-opensearch/12112/pipeline
https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-build-opensearch/detail/distribution-build-opensearch/12109/pipeline/103

Expected behavior

All repos to use maven mirrors and jobs are not hindered by this issue

Acceptance Criteria

• Add our own eclipse mirror to all repos to enable eclipse dependencies caching.
• Regression testing to verify maven caching works by triggering multiple builds and test workflows at a time.
• Verify and validate that all repos have updated list of maven mirrors.

[github-actions]

Possible Duplicate

Found 1 possible duplicate issue(s):

• Use Nexus (sonatype.org) to cache the download repo and cache on the local repo #3956

If this is not a duplicate:

• Add a comment on this issue, or
• 👎 this comment to prevent auto-closure

Otherwise, this issue will be automatically closed in 7 days.

[peterzhuamazon]

Maven mirror already fully fixed and eclipse mirror has nothing to do with maven.
I would say maybe close this issue and I can create one specifically for eclipse.

Thanks.

[gaiksaya]

Please see the acceptance criteria. We need to do regression testing so created issue for that also need to validate all repos are on-boarded as pointed out. There were few misses that we need to avoid in future.

[peterzhuamazon]

Previously Attempts (Maven)

A lot of the changes has already been done to try resolve the maven issues, and eventually all resolved in 3.8.0, need further testing to confirm they are all covered:

• Add maven cache before mavenCentral to reduce calls made to maven central from jenkins runners OpenSearch#21682
• [RELEASE] Release version 3.7.0 #6062 (comment)
• Add pluginManagement block to avoid Maven Central throttling OpenSearch#21853
• [RELEASE] Release version 2.19.6 #6050 (comment)
• [RELEASE] Release version 3.8.0 #6278 (comment)
• [RELEASE] Release version 3.8.0 #6278 (comment)
• [RELEASE] Release version 3.8.0 #6278 (comment)
• [RELEASE] Release version 3.8.0 #6278 (comment)

---

Previous Attemps (Eclipse)

We tried to change to umd eclipse mirror, but it was not stable most of the time so we changed back.

• OpenSearch
  • added: Fix failing CI's with Failed to load eclipse jdt formatter. OpenSearch#17172
  • removed: Attempt to fix the jdt formatter error OpenSearch#19199
• k-NN
  • added: Add mirror repository for spotless k-NN#2453
  • removed: Update step SHA versions, remove maven mirrors k-NN#3390
• neural-search
  • added: Fix github CI by adding eclipse dependency in formatting.gradle neural-search#1079
  • removed: Removed mirror for jdt failures neural-search#1543
• ml-commons
  • added: Fixes spotless on Java 11 ml-commons#3449
  • removed: Fix jdt formatter error ml-commons#4151
• query-insights
  • added: Fix github CI by adding eclipse dependency in build.gradle query-insights#181
  • removed: Fix 2.19 CI, security IT, eclipse query-insights#565
• opensearch-learning-to-rank-base
  • added via non-PR commit
  • removed: Fix Windows CI build failure caused by Spotless P2 mirror timeout opensearch-learning-to-rank-base#303

Right now we can update to a new mirror by ourselves.