fix(tests): create MinIO buckets from test to avoid flaky operator bu… (#882)

# Summary

The `e2e_om_ops_manager_backup_restore_minio` test continued to
experience intermittent failures even after PR #852 fixed the MinIO
operator's CA trust issue. The remaining flakiness was caused by a race
condition between the MinIO operator's bucket creation attempts and
MinIO's readiness to accept HTTPS connections.

**Why the MinIO operator still fails to create buckets (even with CA
trust fixed):**

Even though PR #852 resolved the x509 certificate trust issue by
providing the test CA to the MinIO operator, the operator still
occasionally fails with "connection refused" errors when attempting to
create buckets. This happens because:

1. The MinIO operator's reconcile loop triggers immediately after the
MinIO tenant pods become "Ready"
2. Pod readiness does not guarantee that MinIO is fully ready to accept
HTTPS connections
3. The operator attempts to create buckets via HTTPS API calls before
MinIO's TLS listener is fully initialized
4. This results in intermittent "connection refused" errors, causing
bucket creation to fail silently

The test then times out waiting for buckets that were never created.

**The fix:**

Instead of relying on the MinIO operator's timing-dependent bucket
creation, we now create the buckets directly from the test code using
boto3:

1. After the MinIO tenant pods are ready, call `_create_minio_buckets()`
which uses boto3's S3 client
2. Configure boto3 to use the test CA certificate for TLS verification
3. Implement retry logic (120s timeout, 5s intervals) to handle MinIO
startup timing
4. Check if buckets already exist (created by operator) or create them
via boto3
5. Log which method created each bucket for observability

This approach completely bypasses the operator's unreliable bucket
creation and makes the test deterministic.

## Proof of Work

5 independent test patches were run to verify the fix eliminates
flakiness:

| Patch # | Patch ID | Status | Build URL |
|---------|----------|--------|-----------|
| 1 | `69b02324c32e6b00075f770b` | ✅ Success (4/4) |
https://evergreen.mongodb.com/version/69b02324c32e6b00075f770b |
| 2 | `69b0232e8a5d860007735fc9` | ✅ Success (4/4) |
https://evergreen.mongodb.com/version/69b0232e8a5d860007735fc9 |
| 3 | `69b02339c03a4d00075eac0c` | ✅ Success (4/4) |
https://evergreen.mongodb.com/version/69b02339c03a4d00075eac0c |
| 4 | `69b02343f4bdb00007c3f6cd` | ✅ Success (4/4) |
https://evergreen.mongodb.com/version/69b02343f4bdb00007c3f6cd |
| 5 | `69b0234cc1f1690007bb9f21` | ✅ Success (4/4) |
https://evergreen.mongodb.com/version/69b0234cc1f1690007bb9f21 |

**Result: 20/20 tests passed (100% success rate)** all were created by
boto3


## Checklist

- [ ] Have you linked a jira ticket and/or is the ticket in the title?
- [ ] Have you checked whether your jira ticket required DOCSP changes?
- [x] Have you added changelog file?
    - use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in CONTRIBUTING.md for more details

Author: nammn

docker/mongodb-kubernetes-tests/tests/opsmanager/conftest.py

@@ -3,7 +3,7 @@
 import os
 import time
 from pathlib import Path
-from typing import Dict, List, Optional
+from typing import Dict, List, Optional, Set
 
 import boto3
 from botocore.exceptions import ClientError
@@ -117,47 +117,49 @@ def mino_operator_install(
     )
 
 
-def _wait_for_minio_buckets(
+def _create_minio_buckets(
     endpoint: str,
     bucket_names: List[str],
     access_key: str = "minio",
     secret_key: str = "minio123",
-    timeout: int = 500,
-    interval: int = 10,
+    timeout: int = 120,
+    interval: int = 5,
     issuer_ca_filepath: Optional[str] = os.getenv("MINIO_ISSUER_CA_FILEPATH", None),
-):
-    """Poll S3/MinIO until all buckets are accessible or timeout is reached.
-
-    Pod readiness does not guarantee bucket provisioning is complete. This
-    function bridges the gap by probing headBucket() with retry/backoff,
-    mirroring the exact check OpsManager performs when saving S3 store config.
-    """
+) -> None:
+    """Ensure MinIO buckets exist via S3 API (create if missing). Uses test CA when tenant has custom TLS."""
     s3 = boto3.client(
         "s3",
         endpoint_url=f"https://{endpoint}",
         aws_access_key_id=access_key,
         aws_secret_access_key=secret_key,
         verify=issuer_ca_filepath,
     )
-
+    target: Set[str] = set(bucket_names)
+    ready: Set[str] = set()
     deadline = time.time() + timeout
-    pending = set(bucket_names)
 
     while time.time() < deadline:
-        for bucket in list(pending):
+        for bucket in bucket_names:
+            if bucket in ready:
+                continue
             try:
                 s3.head_bucket(Bucket=bucket)
-                print(f"MinIO bucket '{bucket}' is accessible")
-                pending.discard(bucket)
+                ready.add(bucket)
             except ClientError as e:
-                code = e.response["Error"]["Code"]
-                print(f"MinIO bucket '{bucket}' not ready (HTTP {code}), retrying in {interval}s...")
-        if not pending:
-            print(f"All MinIO buckets accessible: {bucket_names}")
+                if e.response["Error"].get("Code") in ("404", "NoSuchBucket"):
+                    try:
+                        s3.create_bucket(Bucket=bucket)
+                        ready.add(bucket)
+                    except ClientError as ce:
+                        if ce.response["Error"].get("Code") == "BucketAlreadyOwnedByYou":
+                            ready.add(bucket)
+            except Exception:
+                pass  # MinIO not ready (connection/SSL), retry
+        if ready >= target:
             return
         time.sleep(interval)
 
-    raise TimeoutError(f"MinIO buckets still inaccessible after {timeout}s: {pending}")
+    raise TimeoutError(f"Could not create MinIO buckets within {timeout}s: missing {target - ready}")
 
 
 def mino_tenant_install(
@@ -214,10 +216,9 @@ def mino_tenant_install(
         print(f"Minio tenant already installed, skipping helm installation!")
 
     get_pod_when_ready(namespace, f"app=minio", api_client=cluster_client)
-    # Wait for MinIO bucket provisioning (pod ready ≠ buckets ready)
-    # MinIO creates the buckets async via a kubernetes Job after the tenant pod is running,
-    # so we need to wait for the buckets to be accessible before proceeding with tests that depend on them.
-    _wait_for_minio_buckets(
+    # Ensure buckets exist from the test so we don't rely on the operator (custom TLS often
+    # breaks operator bucket creation). Retries until all buckets are created/accessible.
+    _create_minio_buckets(
         endpoint=f"minio.{namespace}.svc.cluster.local",
         bucket_names=["s3-store-bucket", "oplog-s3-bucket"],
         issuer_ca_filepath=issuer_ca_filepath,

docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/minio/values-tenant.yaml

@@ -109,14 +109,6 @@ tenant:
   features:
     bucketDNS: false
     domains: { }
-  ## List of bucket definitions to create during tenant provisioning.
-  ## Example:
-  #   - name: my-minio-bucket
-  #     objectLock: false        # optional
-  #     region: us-east-1        # optional
-  buckets:
-     - name: oplog-s3-bucket
-     - name: s3-store-bucket
 
   ## List of secret names to use for generating MinIO users during tenant provisioning
   users: [ ]