fix(tests): create MinIO buckets from test to avoid flaky operator bu…#882
Merged
fix(tests): create MinIO buckets from test to avoid flaky operator bu…#882
Conversation
…cket creation The MinIO operator creates buckets over HTTPS. With custom TLS (tls-ssl-minio signed by the test CA), the operator often fails with x509 or connection refused, making test_install_minio flaky. Create buckets from the test via boto3 using the test CA so the test no longer depends on operator timing. Log whether each bucket was created via boto3 or already existed (MinIO operator). Made-with: Cursor
MCK 1.7.1 Release NotesOther Changes
|
nammn
added
the
skip-changelog
lucian-tosa
reviewed
Mar 10, 2026
MaciejKaras
approved these changes
Mar 11, 2026
MaciejKaras
reviewed
Mar 11, 2026
| try: | ||
| s3.create_bucket(Bucket=bucket) | ||
| ready.add(bucket) | ||
| except ClientError as ce: |
Collaborator
There was a problem hiding this comment.
nit: this check is unnecessary, we will find about existing bucket in s3.head_bucket(Bucket=bucket) call
Collaborator
Author
There was a problem hiding this comment.
lucian-tosa
approved these changes
Mar 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
e2e_om_ops_manager_backup_restore_miniotest continued to experience intermittent failures even after PR #852 fixed the MinIO operator's CA trust issue. The remaining flakiness was caused by a race condition between the MinIO operator's bucket creation attempts and MinIO's readiness to accept HTTPS connections.Why the MinIO operator still fails to create buckets (even with CA trust fixed):
Even though PR #852 resolved the x509 certificate trust issue by providing the test CA to the MinIO operator, the operator still occasionally fails with "connection refused" errors when attempting to create buckets. This happens because:
The test then times out waiting for buckets that were never created.
The fix:
Instead of relying on the MinIO operator's timing-dependent bucket creation, we now create the buckets directly from the test code using boto3:
_create_minio_buckets()which uses boto3's S3 clientThis approach completely bypasses the operator's unreliable bucket creation and makes the test deterministic.
Proof of Work
5 independent test patches were run to verify the fix eliminates flakiness:
69b02324c32e6b00075f770b69b0232e8a5d860007735fc969b02339c03a4d00075eac0c69b02343f4bdb00007c3f6cd69b0234cc1f1690007bb9f21Result: 20/20 tests passed (100% success rate) all were created by boto3
Checklist
skip-changeloglabel if not needed