Skip to content

chore: add new test structure for pluginDivisionMode: schema#4288

Open
Fortune-Ndlovu wants to merge 47 commits intoredhat-developer:mainfrom
Fortune-Ndlovu:RHIDP-12355-add-test-plugin-division-mode-schema
Open

chore: add new test structure for pluginDivisionMode: schema#4288
Fortune-Ndlovu wants to merge 47 commits intoredhat-developer:mainfrom
Fortune-Ndlovu:RHIDP-12355-add-test-plugin-division-mode-schema

Conversation

@Fortune-Ndlovu
Copy link
Copy Markdown
Member

@Fortune-Ndlovu Fortune-Ndlovu commented Feb 24, 2026

Description

This PR adds E2E coverage for pluginDivisionMode: schema on both Helm and Operator deployments. The tests check that RHDH runs with a restricted DB user (single database, no CREATEDB), that plugin schemas are created in one database instead of separate plugin databases, and that the app stays reachable. Shared setup (DB creation, user hardening, revoking CONNECT on other DBs, grants) lives in schema-mode-db.ts; Helm- and Operator-specific config (secrets, ConfigMaps, CR patching) stay in the respective specs.

The schema-mode specs are included in the e2e suite via SHOWCASE_RUNTIME_DB when SCHEMA_MODE_DB_HOST is set, and each spec skips when the job is for the other install method (JOB_NAME contains "operator" or "helm") so Helm nightly runs only the Helm spec and Operator nightly only the Operator spec. Tests remain opt-in when SCHEMA_MODE_* env vars are not set (skip with a clear message), and are ignored in SHOWCASE / SHOWCASE_K8S / SHOWCASE_OPERATOR so the default suite is unchanged. Manual runs use --project=any-test as documented in the README.

Which issue(s) does this PR fix

PR acceptance criteria

Please make sure that the following steps are complete:

  • GitHub Actions are completed and successful
  • Unit Tests are updated and passing
  • E2E Tests are updated and passing
  • Documentation is updated if necessary (requirement for new features)
  • Add a screenshot if the change is UX/UI related

How to test changes / Special notes to the reviewer

@rhdh-qodo-merge
Copy link
Copy Markdown

rhdh-qodo-merge bot commented Feb 24, 2026

PR Reviewer Guide 🔍

(Review updated until commit 4a2e099)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

RHIDP-12355 - Partially compliant

Compliant requirements:

  • Add an E2E test that verifies pluginDivisionMode: schema behavior/configuration.

Non-compliant requirements:

  • Ensure the dedicated DB user only has permissions on that single database (principle of least privilege).

Requires further human verification:

  • Provision a single database and a dedicated DB user for the test setup.
  • End-to-end validation that the tests pass in the intended CI/OpenShift environments (Operator and Helm).
⏱️ Estimated effort to review: 5 🔵🔵🔵🔵🔵
🔒 Security concerns

SQL injection / unsafe string interpolation:
In multiple places SQL statements interpolate dbName, dbUser, and dbPassword directly into queries (e.g., CREATE USER ... WITH PASSWORD '...', CREATE DATABASE ..., GRANT ...). While this is test code, it still executes against a real DB and uses env-provided values; use parameterized queries where possible and properly quote identifiers (or validate/whitelist allowed characters for DB/user names).

Command injection risk: execSync is used to run oc patch ... -p='${patchJson}' with interpolated content. Even if currently derived from code, it’s brittle and unsafe if any part ever becomes user-controlled. Prefer non-shell execution or API-only patching.

⚡ Recommended focus areas for review

Privilege Scope

The DB user created for the test is granted very broad permissions (including GRANT ALL PRIVILEGES ON DATABASE ... and ownership changes) which conflicts with the ticket’s requirement that the user should only have permissions on the single DB and ideally only the minimum privileges needed for schema-mode (create/use schemas). Consider tightening grants to only what’s required (CONNECT + CREATE on the target DB, and USAGE/CREATE on specific schemas), and avoid altering public ownership unless strictly needed.

await adminClient
  .query(`CREATE USER ${dbUser} WITH PASSWORD '${dbPassword}'`)
  .catch(async (err) => {
    if (err.message.includes("already exists")) {
      await adminClient.query(
        `ALTER USER ${dbUser} WITH PASSWORD '${dbPassword}'`,
      );
    } else {
      throw err;
    }
  });

await adminClient.query(`GRANT CONNECT ON DATABASE ${dbName} TO ${dbUser}`);
await adminClient.end();

const dbClient = new Client({
  host: dbHost,
  port: 5432,
  user: dbAdminUser,
  password: dbAdminPassword,
  database: dbName,
  connectionTimeoutMillis: 30000,
});
await dbClient.connect();
await dbClient.query(`GRANT CREATE ON DATABASE ${dbName} TO ${dbUser}`);
await dbClient.query(`GRANT USAGE ON SCHEMA public TO ${dbUser}`);
await dbClient.query(`GRANT CREATE ON SCHEMA public TO ${dbUser}`);
await dbClient.query(
  `GRANT ALL PRIVILEGES ON DATABASE ${dbName} TO ${dbUser}`,
);
await dbClient.query(`ALTER SCHEMA public OWNER TO ${dbUser}`);
await dbClient.end();
📚 Focus areas based on broader codebase context

Security

The test falls back to running execSync("oc patch ... -p='${patchJson}'") with interpolated values, which is vulnerable to shell/quoting breakage and potential command injection if any interpolated value contains quotes or shell metacharacters. This also makes the test environment-dependent (requires oc CLI + logged-in context) and can introduce flakiness vs using the Kubernetes client API consistently. (Ref 8)

try {
  const patchJson = JSON.stringify(patch);
  execSync(
    `oc patch deployment ${deploymentName} -n ${namespace} --type='json' -p='${patchJson}'`,
    { stdio: "pipe", encoding: "utf-8" },
  );
  console.log(
    "✓ Added POSTGRES_DB environment variable to deployment via oc command",
  );

  // Verify the patch was applied
  await new Promise((resolve) => setTimeout(resolve, 2000)); // Wait 2 seconds for API to sync
  const verifyDeployment =
    await kubeClient.appsApi.readNamespacedDeployment(
      deploymentName,
      namespace,
    );
  const verifyContainer =
    verifyDeployment.body.spec?.template?.spec?.containers?.find(
      (c) => c.name === "backstage-backend",
    );
  const verifyEnv = verifyContainer?.env || [];
  const verifyPostgresDb = verifyEnv.find(
    (e) => e.name === "POSTGRES_DB",
  );

  if (!verifyPostgresDb) {
    throw new Error(
      "Patch command succeeded but POSTGRES_DB was not found in deployment after patching",
    );
  }
  console.log(
    "✓ Verified POSTGRES_DB environment variable was added to deployment",
  );
  needsRestart = true;
} catch (ocError) {
  const ocErrorMsg =
    ocError instanceof Error ? ocError.message : String(ocError);
  console.error(
    `[ERROR]  Failed to patch deployment via oc command: ${ocErrorMsg}`,
  );
  console.error(
    `   Please manually run this command to add POSTGRES_DB:`,
  );
  console.error(
    `   oc patch deployment ${deploymentName} -n ${namespace} --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"POSTGRES_DB","valueFrom":{"secretKeyRef":{"name":"${secretName}","key":"POSTGRES_DB"}}}}]'`,
  );
  throw new Error(
    `Failed to add POSTGRES_DB environment variable to deployment. Both API and oc command failed.`,
  );
}

Reference reasoning: Existing E2E code that shells out (execSync) includes explicit input validation to prevent command injection (e.g., validating namespace format before constructing a command). The safer pattern in the repo is either to avoid shelling out entirely by using API clients, or to validate/sanitize all interpolated values and avoid embedding raw JSON inside shell quotes.

📄 References
  1. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugins/scaffolder-relation-processor/scaffolder-relation-processor.spec.ts [10-167]
  2. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugins/topology/topology-rbac.spec.ts [1-84]
  3. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugins/orchestrator/orchestrator-rbac.spec.ts [12-1553]
  4. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugins/orchestrator/token-propagation-workflow.spec.ts [37-239]
  5. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugins/rbac/rbac.spec.ts [22-965]
  6. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugins/adoption-insights/adoption-insights.spec.ts [10-267]
  7. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugins/scorecard/scorecard.spec.ts [25-190]
  8. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugins/tekton/tekton.spec.ts [14-56]

@rhdh-qodo-merge
Copy link
Copy Markdown

rhdh-qodo-merge bot commented Feb 24, 2026

Review Summary by Qodo

Add E2E tests for pluginDivisionMode schema mode verification with enhanced Kubernetes diagnostics

🧪 Tests ✨ Enhancement 📝 Documentation

Grey Divider

Walkthroughs

Description
• Adds comprehensive E2E test suites for verifying pluginDivisionMode: schema configuration on
  both OpenShift Operator and Helm chart deployments
• Tests cover database setup, PostgreSQL connectivity, schema creation, and RHDH accessibility with
  detailed error handling and diagnostics
• Enhances PostgreSQL credentials configuration with database parameter support for setting
  POSTGRES_DB environment variable
• Improves Kubernetes client with advanced diagnostics capabilities including pod command execution,
  container logs, pod events, and replica set status logging
• Adds comprehensive documentation for schema mode E2E test setup and execution
• Registers new test files in Playwright configuration

Grey Divider

File Changes

1. e2e-tests/playwright/e2e/plugin-division-mode-schema/verify-schema-mode-operator.spec.ts 🧪 Tests +1487/-0

Add E2E test suite for Operator schema mode verification

• Adds comprehensive E2E test suite for verifying pluginDivisionMode: schema configuration on
 OpenShift Operator deployments
• Tests database setup, PostgreSQL connectivity, schema creation, and RHDH accessibility
• Includes detailed error handling, diagnostics, and troubleshooting guidance for schema mode
 verification
• Configures Kubernetes resources (secrets, ConfigMaps, Backstage CR) to enable schema mode with
 environment variable injection

e2e-tests/playwright/e2e/plugin-division-mode-schema/verify-schema-mode-operator.spec.ts


2. e2e-tests/playwright/e2e/plugin-division-mode-schema/verify-schema-mode-helm.spec.ts 🧪 Tests +1300/-0

Add E2E test suite for Helm chart schema mode verification

• Adds comprehensive E2E test suite for verifying pluginDivisionMode: schema configuration on Helm
 chart deployments
• Tests database setup, PostgreSQL connectivity, schema creation, and RHDH accessibility
• Includes detailed error handling, diagnostics, and troubleshooting guidance for schema mode
 verification
• Configures Kubernetes resources (secrets, deployments, ConfigMaps) to enable schema mode with
 environment variable injection

e2e-tests/playwright/e2e/plugin-division-mode-schema/verify-schema-mode-helm.spec.ts


3. e2e-tests/playwright/utils/postgres-config.ts ✨ Enhancement +13/-4

Enhance PostgreSQL credentials configuration with database support

• Adds database parameter to credentials interface for setting POSTGRES_DB environment variable
• Conditionally sets NODE_EXTRA_CA_CERTS only when SSL mode is not disabled
• Encodes POSTGRES_DB in the secret when database parameter is provided

e2e-tests/playwright/utils/postgres-config.ts


View more (3)
4. e2e-tests/playwright.config.ts ⚙️ Configuration changes +2/-0

Register schema mode test files in Playwright configuration

• Registers two new test files for schema mode verification in the test configuration
• Adds verify-schema-mode-operator.spec.ts and verify-schema-mode-helm.spec.ts to the test suite

e2e-tests/playwright.config.ts


5. e2e-tests/playwright/e2e/plugin-division-mode-schema/README.md 📝 Documentation +79/-0

Add documentation for schema mode E2E test setup and execution

• Provides comprehensive setup and execution guide for schema mode E2E tests
• Documents prerequisites, environment variable configuration for both Operator and Helm deployments
• Includes instructions for running tests and viewing test reports

e2e-tests/playwright/e2e/plugin-division-mode-schema/README.md


6. e2e-tests/playwright/utils/kube-client.ts ✨ Enhancement +525/-13

Enhanced Kubernetes client diagnostics and pod execution capabilities

• Added stream import and customObjectsApi property to support advanced Kubernetes operations
• Enhanced pod failure detection with comprehensive condition checking for scheduling, readiness,
 and container states
• Improved deployment readiness waiting logic with dynamic label selector determination from
 deployment metadata
• Added multiple diagnostic logging methods (logPodContainerLogs, logPodEvents,
 logReplicaSetStatus) for better troubleshooting
• Implemented execPodCommand method to execute commands inside pods with timeout support

e2e-tests/playwright/utils/kube-client.ts


Grey Divider

Qodo Logo

@rhdh-qodo-merge
Copy link
Copy Markdown

rhdh-qodo-merge bot commented Feb 24, 2026

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
Possible issue
Make catch callback async
Suggestion Impact:The commit removed the entire `schema-mode-config.ts` content, thereby eliminating the problematic `.catch()` block that contained an `await` (and the surrounding setup logic), instead of converting it to a `try/catch`.

code diff:

@@ -1,250 +1 @@
-/**
- * Utilities for testing pluginDivisionMode: schema feature.
- * Provides functions to set up a database with a limited-permissions user
- * and verify schema mode is working correctly.
- */
 
-import { Client } from "pg";
-
-/**
- * Set up a database and user with limited permissions for schema mode testing.
- * The user will only have permissions on a single database (no CREATEDB privilege).
- *
- * @param config - Database setup configuration
- * @param config.host - PostgreSQL host
- * @param config.adminUser - Admin user (must have CREATEDB and CREATEROLE)
- * @param config.adminPassword - Admin user password
- * @param config.databaseName - Name of the database to create
- * @param config.userName - Name of the limited-permissions user to create
- * @param config.userPassword - Password for the limited-permissions user
- */
-export async function setupSchemaModeDatabase(config: {
-  host: string;
-  port?: string;
-  adminUser: string;
-  adminPassword: string;
-  databaseName: string;
-  userName: string;
-  userPassword: string;
-}): Promise<void> {
-  console.log(
-    `Setting up database ${config.databaseName} with user ${config.userName}...`,
-  );
-
-  // Connect as admin to set up database and user
-  const adminClient = new Client({
-    host: config.host,
-    port: parseInt(config.port || "5432"),
-    user: config.adminUser,
-    password: config.adminPassword,
-    database: "postgres",
-    connectionTimeoutMillis: 30 * 1000,
-  });
-
-  try {
-    await adminClient.connect();
-
-    // Create database if it doesn't exist
-    await adminClient.query(
-      `CREATE DATABASE ${config.databaseName}`,
-    ).catch((err) => {
-      if (err.message.includes("already exists")) {
-        console.log(`Database ${config.databaseName} already exists`);
-      } else {
-        throw err;
-      }
-    });
-
-    // Create user if it doesn't exist
-    await adminClient.query(
-      `CREATE USER ${config.userName} WITH PASSWORD '${config.userPassword}'`,
-    ).catch((err) => {
-      if (err.message.includes("already exists")) {
-        console.log(`User ${config.userName} already exists, updating password...`);
-        await adminClient.query(
-          `ALTER USER ${config.userName} WITH PASSWORD '${config.userPassword}'`,
-        );
-      } else {
-        throw err;
-      }
-    });
-
-    // Grant CONNECT privilege on the database
-    await adminClient.query(
-      `GRANT CONNECT ON DATABASE ${config.databaseName} TO ${config.userName}`,
-    );
-
-    // Connect to the target database to grant schema privileges
-    await adminClient.end();
-    const dbClient = new Client({
-      host: config.host,
-      port: parseInt(config.port || "5432"),
-      user: config.adminUser,
-      password: config.adminPassword,
-      database: config.databaseName,
-      connectionTimeoutMillis: 30 * 1000,
-    });
-
-    await dbClient.connect();
-
-    // Grant CREATE SCHEMA privilege (required for schema mode)
-    await dbClient.query(
-      `GRANT CREATE ON DATABASE ${config.databaseName} TO ${config.userName}`,
-    );
-
-    // Grant usage on public schema (for initial connection)
-    await dbClient.query(
-      `GRANT USAGE ON SCHEMA public TO ${config.userName}`,
-    );
-
-    // Set default privileges so user can create objects in their schemas
-    await dbClient.query(
-      `ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO ${config.userName}`,
-    );
-
-    await dbClient.end();
-
-    console.log(
-      `Database ${config.databaseName} and user ${config.userName} set up successfully`,
-    );
-  } catch (error) {
-    await adminClient.end().catch(() => {});
-    const errorMsg = error instanceof Error ? error.message : String(error);
-    throw new Error(`Failed to set up database: ${errorMsg}`);
-  }
-}

Fix a syntax error by replacing the .catch() block with a try/catch block, which
allows the use of await when updating an existing user's password.

e2e-tests/playwright/utils/schema-mode-config.ts [59-70]

-await adminClient.query(
-  `CREATE USER ${config.userName} WITH PASSWORD '${config.userPassword}'`,
-).catch((err) => {
-  if (err.message.includes("already exists")) {
+try {
+  await adminClient.query(
+    `CREATE USER ${config.userName} WITH PASSWORD '${config.userPassword}'`,
+  );
+} catch (err) {
+  if ((err as Error).message.includes("already exists")) {
     console.log(`User ${config.userName} already exists, updating password...`);
     await adminClient.query(
       `ALTER USER ${config.userName} WITH PASSWORD '${config.userPassword}'`,
     );
   } else {
     throw err;
   }
-});
+}

[Suggestion processed]

Suggestion importance[1-10]: 9

__

Why: The suggestion correctly identifies a critical syntax error (await inside a non-async .catch() callback) that will cause the test setup to fail and provides a valid fix using a try/catch block.

High
Preserve resourceVersion to prevent race conditions
Suggestion Impact:The lines deleting metadata.creationTimestamp and metadata.resourceVersion (and the surrounding ConfigMap update logic) were removed as part of deleting the entire spec file, thereby eliminating the problematic deletion behavior indirectly rather than implementing the suggested adjustment in-place.

code diff:

-  // Remove metadata fields that shouldn't be in update
-  delete configMap.metadata?.creationTimestamp;
-  delete configMap.metadata?.resourceVersion;
-
-  // Use replaceNamespacedConfigMap to update
-  await kubeClient.coreV1Api.replaceNamespacedConfigMap(
-    configMapName,
-    namespace,
-    configMap,
-  );

Preserve the resourceVersion field when updating the Kubernetes ConfigMap to
prevent race conditions. Remove the lines that delete metadata.creationTimestamp
and metadata.resourceVersion.

e2e-tests/playwright/e2e/plugin-division-mode-schema/verify-schema-mode.spec.ts [174-183]

-// Remove metadata fields that shouldn't be in update
-delete configMap.metadata?.creationTimestamp;
-delete configMap.metadata?.resourceVersion;
-
 // Use replaceNamespacedConfigMap to update
 await kubeClient.coreV1Api.replaceNamespacedConfigMap(
   configMapName,
   namespace,
   configMap,
 );

[Suggestion processed]

Suggestion importance[1-10]: 8

__

Why: The suggestion correctly identifies that deleting resourceVersion before an update disables optimistic locking, which can lead to race conditions and overwritten data, improving test robustness.

Medium
Correctly set default user privileges
Suggestion Impact:The commit removed the entire schema-mode database utility file that contained the incorrect ALTER DEFAULT PRIVILEGES statement, eliminating the misconfigured logic rather than updating it as suggested.

code diff:

@@ -1,250 +1 @@
-/**
- * Utilities for testing pluginDivisionMode: schema feature.
- * Provides functions to set up a database with a limited-permissions user
- * and verify schema mode is working correctly.
- */
 
-import { Client } from "pg";
-
-/**
- * Set up a database and user with limited permissions for schema mode testing.
- * The user will only have permissions on a single database (no CREATEDB privilege).
- *
- * @param config - Database setup configuration
- * @param config.host - PostgreSQL host
- * @param config.adminUser - Admin user (must have CREATEDB and CREATEROLE)
- * @param config.adminPassword - Admin user password
- * @param config.databaseName - Name of the database to create
- * @param config.userName - Name of the limited-permissions user to create
- * @param config.userPassword - Password for the limited-permissions user
- */
-export async function setupSchemaModeDatabase(config: {
-  host: string;
-  port?: string;
-  adminUser: string;
-  adminPassword: string;
-  databaseName: string;
-  userName: string;
-  userPassword: string;
-}): Promise<void> {
-  console.log(
-    `Setting up database ${config.databaseName} with user ${config.userName}...`,
-  );
-
-  // Connect as admin to set up database and user
-  const adminClient = new Client({
-    host: config.host,
-    port: parseInt(config.port || "5432"),
-    user: config.adminUser,
-    password: config.adminPassword,
-    database: "postgres",
-    connectionTimeoutMillis: 30 * 1000,
-  });
-
-  try {
-    await adminClient.connect();
-
-    // Create database if it doesn't exist
-    await adminClient.query(
-      `CREATE DATABASE ${config.databaseName}`,
-    ).catch((err) => {
-      if (err.message.includes("already exists")) {
-        console.log(`Database ${config.databaseName} already exists`);
-      } else {
-        throw err;
-      }
-    });
-
-    // Create user if it doesn't exist
-    await adminClient.query(
-      `CREATE USER ${config.userName} WITH PASSWORD '${config.userPassword}'`,
-    ).catch((err) => {
-      if (err.message.includes("already exists")) {
-        console.log(`User ${config.userName} already exists, updating password...`);
-        await adminClient.query(
-          `ALTER USER ${config.userName} WITH PASSWORD '${config.userPassword}'`,
-        );
-      } else {
-        throw err;
-      }
-    });
-
-    // Grant CONNECT privilege on the database
-    await adminClient.query(
-      `GRANT CONNECT ON DATABASE ${config.databaseName} TO ${config.userName}`,
-    );
-
-    // Connect to the target database to grant schema privileges
-    await adminClient.end();
-    const dbClient = new Client({
-      host: config.host,
-      port: parseInt(config.port || "5432"),
-      user: config.adminUser,
-      password: config.adminPassword,
-      database: config.databaseName,
-      connectionTimeoutMillis: 30 * 1000,
-    });
-
-    await dbClient.connect();
-
-    // Grant CREATE SCHEMA privilege (required for schema mode)
-    await dbClient.query(
-      `GRANT CREATE ON DATABASE ${config.databaseName} TO ${config.userName}`,
-    );
-
-    // Grant usage on public schema (for initial connection)
-    await dbClient.query(
-      `GRANT USAGE ON SCHEMA public TO ${config.userName}`,
-    );
-
-    // Set default privileges so user can create objects in their schemas
-    await dbClient.query(
-      `ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO ${config.userName}`,
-    );
-
-    await dbClient.end();
-
-    console.log(
-      `Database ${config.databaseName} and user ${config.userName} set up successfully`,
-    );
-  } catch (error) {
-    await adminClient.end().catch(() => {});
-    const errorMsg = error instanceof Error ? error.message : String(error);
-    throw new Error(`Failed to set up database: ${errorMsg}`);
-  }
-}
-
-/**
- * Verify that plugin schemas were created in the database.
- *
- * @param credentials - Database connection credentials
- */
-export async function verifySchemasExist(credentials: {
-  host: string;
-  port?: string;
-  user: string;
-  password: string;
-  database: string;
-}): Promise<void> {
-  console.log("Verifying schemas were created...");
-
-  const client = new Client({
-    host: credentials.host,
-    port: parseInt(credentials.port || "5432"),
-    user: credentials.user,
-    password: credentials.password,
-    database: credentials.database,
-    connectionTimeoutMillis: 30 * 1000,
-  });
-
-  try {
-    await client.connect();
-
-    // Query for plugin schemas (excluding system schemas)
-    const result = await client.query<{ schema_name: string }>(`
-      SELECT schema_name 
-      FROM information_schema.schemata 
-      WHERE schema_name NOT IN ('pg_catalog', 'information_schema', 'pg_toast', 'pg_temp_1', 'pg_toast_temp_1')
-        AND schema_name NOT LIKE 'pg_%'
-      ORDER BY schema_name
-    `);
-
-    const schemas = result.rows.map((row) => row.schema_name);
-
-    // Expected plugin schemas
-    const expectedSchemas = [
-      "catalog",
-      "scaffolder",
-      "auth",
-      "permission",
-      "search",
-      "techdocs",
-    ];
-
-    const foundSchemas = expectedSchemas.filter((schema) =>
-      schemas.includes(schema),
-    );
-
-    if (foundSchemas.length === 0) {
-      throw new Error(
-        `No plugin schemas found. Found schemas: ${schemas.join(", ")}`,
-      );
-    }
-
-    console.log(
-      `Found ${foundSchemas.length} plugin schemas: ${foundSchemas.join(", ")}`,
-    );
-    console.log(`All schemas: ${schemas.join(", ")}`);
-
-    // Verify we have at least a few key schemas
-    if (foundSchemas.length < 3) {
-      throw new Error(
-        `Expected at least 3 plugin schemas, found only ${foundSchemas.length}`,
-      );
-    }
-  } catch (error) {
-    const errorMsg = error instanceof Error ? error.message : String(error);
-    throw new Error(`Failed to verify schemas: ${errorMsg}`);
-  } finally {
-    await client.end();
-  }
-}
-
-/**
- * Verify that no separate plugin databases were created.
- * Should only see the test database and system databases.
- *
- * @param credentials - Database connection credentials (using postgres database)
- */
-export async function verifyNoPluginDatabases(credentials: {
-  host: string;
-  port?: string;
-  user: string;
-  password: string;
-}): Promise<void> {
-  console.log("Verifying no separate plugin databases were created...");
-
-  const client = new Client({
-    host: credentials.host,
-    port: parseInt(credentials.port || "5432"),
-    user: credentials.user,
-    password: credentials.password,
-    database: "postgres",
-    connectionTimeoutMillis: 30 * 1000,
-  });
-
-  try {
-    await client.connect();
-
-    // Query for databases
-    const result = await client.query<{ datname: string }>(`
-      SELECT datname 
-      FROM pg_database 
-      WHERE datistemplate = false
-      ORDER BY datname
-    `);
-
-    const databases = result.rows.map((row) => row.datname);
-
-    // Check for plugin databases (should not exist in schema mode)
-    const pluginDatabases = databases.filter((db) =>
-      db.startsWith("backstage_plugin_"),
-    );
-
-    if (pluginDatabases.length > 0) {
-      throw new Error(
-        `Found plugin databases that should not exist in schema mode: ${pluginDatabases.join(", ")}`,
-      );
-    }
-
-    console.log(
-      `No plugin databases found. All databases: ${databases.join(", ")}`,
-    );
-  } catch (error) {
-    const errorMsg = error instanceof Error ? error.message : String(error);
-    throw new Error(`Failed to verify databases: ${errorMsg}`);
-  } finally {
-    await client.end();
-  }
-}
-

Correct the ALTER DEFAULT PRIVILEGES command to apply to the newly created user
(config.userName) rather than the admin user, ensuring plugins can manage
objects in their own schemas.

e2e-tests/playwright/utils/schema-mode-config.ts [100-103]

-// Set default privileges so user can create objects in their schemas
+// Set default privileges for the new user, so they can manage objects
+// in schemas they create.
 await dbClient.query(
-  `ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO ${config.userName}`,
+  `ALTER DEFAULT PRIVILEGES FOR ROLE "${config.userName}" GRANT ALL ON TABLES TO "${config.userName}"`,
+);
+await dbClient.query(
+  `ALTER DEFAULT PRIVILEGES FOR ROLE "${config.userName}" GRANT ALL ON SEQUENCES TO "${config.userName}"`,
+);
+await dbClient.query(
+  `ALTER DEFAULT PRIVILEGES FOR ROLE "${config.userName}" GRANT ALL ON FUNCTIONS TO "${config.userName}"`,
 );

[Suggestion processed]

Suggestion importance[1-10]: 8

__

Why: The suggestion correctly identifies that the ALTER DEFAULT PRIVILEGES command is misconfigured for the wrong user and schema, and provides a fix that correctly sets privileges for the new user.

Medium
High-level
Reconfigure application via Helm instead
Suggestion Impact:The committed change removed the test code that directly modified the app-config ConfigMap (including the updateAppConfigForSchemaMode helper). However, it did not replace it with a Helm-upgrade-based reconfiguration approach.

code diff:

-    // Update app-config to enable schema mode
-    await updateAppConfigForSchemaMode(kubeClient, namespace, {
-      host: dbHost,
-      database: dbName,
-      user: dbUser,
-      password: dbPassword,
-    });
-
-    // Restart deployment to pick up new configuration
-    console.log("Restarting deployment...");
-    await kubeClient.restartDeployment(deploymentName, namespace);
-  });
-
-  test("Verify schemas were created", async () => {
-    test.setTimeout(180000); // 3 minutes for RHDH to start and create schemas
-
-    // Wait for RHDH to be ready
-    const kubeClient = new KubeClient();
-    await kubeClient.waitForDeploymentReady(deploymentName, namespace, 120000);
-
-    // Verify schemas exist
-    await verifySchemasExist({
-      host: dbHost,
-      user: dbUser,
-      password: dbPassword,
-      database: dbName,
-    });
-  });
-
-  test("Verify no separate plugin databases were created", async () => {
-    await verifyNoPluginDatabases({
-      host: dbHost,
-      user: dbUser,
-      password: dbPassword,
-    });
-  });
-
-  test("Verify RHDH is accessible", async ({ page }) => {
-    const common = new Common(page);
-    await common.loginAsGuest();
-  });
-});
-
-/**
- * Update app-config ConfigMap to enable pluginDivisionMode: schema
- */
-async function updateAppConfigForSchemaMode(
-  kubeClient: KubeClient,
-  namespace: string,
-  dbConfig: {
-    host: string;
-    database: string;
-    user: string;
-    password: string;
-  },
-): Promise<void> {
-  const configMapName = process.env.RELEASE_NAME
-    ? `backstage-appconfig-${process.env.RELEASE_NAME}`
-    : "backstage-appconfig-developer-hub";
-
-  const configMap = await kubeClient.getConfigMap(configMapName, namespace);
-  if (!configMap || !configMap.data) {
-    throw new Error(`ConfigMap ${configMapName} not found`);
-  }
-
-  // Find the app-config key (could be app-config.yaml or app-config.local.yaml)
-  const configKey = Object.keys(configMap.data).find((key) =>
-    key.includes("app-config"),
-  );
-  if (!configKey) {
-    throw new Error(`No app-config key found in ConfigMap ${configMapName}`);
-  }
-
-  // Parse YAML
-  const appConfig = yaml.load(configMap.data[configKey]) as any;
-
-  // Update database configuration
-  if (!appConfig.backend) {
-    appConfig.backend = {};
-  }
-  if (!appConfig.backend.database) {
-    appConfig.backend.database = {};
-  }
-
-  appConfig.backend.database = {
-    client: "pg",
-    pluginDivisionMode: "schema",
-    connection: {
-      host: `\${POSTGRES_HOST}`,
-      port: `\${POSTGRES_PORT}`,
-      user: `\${POSTGRES_USER}`,
-      password: `\${POSTGRES_PASSWORD}`,
-      database: dbConfig.database,
-    },
-  };
-
-  // Update ConfigMap data
-  configMap.data[configKey] = yaml.dump(appConfig);
-
-  // Remove metadata fields that shouldn't be in update
-  delete configMap.metadata?.creationTimestamp;
-  delete configMap.metadata?.resourceVersion;
-
-  // Use replaceNamespacedConfigMap to update
-  await kubeClient.coreV1Api.replaceNamespacedConfigMap(
-    configMapName,
-    namespace,
-    configMap,
-  );
-
-  console.log(`ConfigMap ${configMapName} updated with schema mode configuration`);
-}

Instead of directly modifying the app-config Kubernetes ConfigMap, reconfigure
the application by triggering a helm upgrade. This approach is more robust and
aligns with standard deployment practices.

Examples:

e2e-tests/playwright/e2e/plugin-division-mode-schema/verify-schema-mode.spec.ts [121-186]
async function updateAppConfigForSchemaMode(
  kubeClient: KubeClient,
  namespace: string,
  dbConfig: {
    host: string;
    database: string;
    user: string;
    password: string;
  },
): Promise<void> {

 ... (clipped 56 lines)

Solution Walkthrough:

Before:

async function updateAppConfigForSchemaMode(kubeClient, namespace, dbConfig) {
  const configMap = await kubeClient.getConfigMap(...);
  const appConfig = yaml.load(configMap.data['app-config.yaml']);

  // Overwrite the entire database configuration
  appConfig.backend.database = {
    client: "pg",
    pluginDivisionMode: "schema",
    connection: { ... }
  };

  configMap.data['app-config.yaml'] = yaml.dump(appConfig);
  await kubeClient.coreV1Api.replaceNamespacedConfigMap(...);
  await kubeClient.restartDeployment(...);
}

After:

async function upgradeHelmChartWithSchemaMode(releaseName, namespace, dbConfig) {
  const values = {
    "rhdh": {
      "database": {
        "pluginDivisionMode": "schema",
        "connection": {
          "database": dbConfig.database,
          // other values...
        }
      }
    }
  };
  // Execute 'helm upgrade' command with the new values
  await runHelmUpgrade(releaseName, namespace, values);
  // Helm upgrade triggers a new rollout, wait for it to complete
  await kubeClient.waitForDeploymentReady(...);
}
Suggestion importance[1-10]: 8

__

Why: The suggestion correctly identifies that directly manipulating the app-config ConfigMap is a brittle approach and proposes a more robust solution using helm upgrade, which aligns better with standard deployment practices and improves test reliability.

Medium
  • Update

@github-actions
Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 4, 2026

This PR is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 21 days.

@github-actions github-actions bot added the Stale label Mar 4, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 4, 2026

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

…ere created], [Verify pluginDivisionMode: schema › Verify no separate plugin databases], and [Verify pluginDivisionMode: schema › Verify RHDH is accessible]

Signed-off-by: Fortune-Ndlovu <[email protected]>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 4, 2026

The container image build workflow finished with status: cancelled.

Signed-off-by: Fortune-Ndlovu <[email protected]>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 4, 2026

Image was built and published successfully. It is available at:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 4, 2026

Image was built and published successfully. It is available at:

@github-actions github-actions bot removed the Stale label Mar 5, 2026
Signed-off-by: Fortune-Ndlovu <[email protected]>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 5, 2026

Image was built and published successfully. It is available at:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

The container image build workflow finished with status: cancelled.

Signed-off-by: Fortune-Ndlovu <[email protected]>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

The container image build workflow finished with status: cancelled.

Signed-off-by: Fortune-Ndlovu <[email protected]>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 9, 2026

The container image build workflow finished with status: cancelled.

@Fortune-Ndlovu
Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 1, 2026

Image was built and published successfully. It is available at:

@Fortune-Ndlovu Fortune-Ndlovu requested review from rm3l and zdrapela April 1, 2026 11:47
Copy link
Copy Markdown
Member

@zdrapela zdrapela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, added a few comments.

# shellcheck source=.ci/pipelines/playwright-projects.sh
source "$DIR"/playwright-projects.sh

configure_schema_mode_runtime_env() {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that you already moved this logic to the test files. Is this function needed then?
The Helm nightly job has ⚠ Schema-mode nightly: PostgreSQL service not found in showcase-runtime; schema tests remain opt-in. in the logs.
I guess the function exits before exporting the env vars, which is making the schema mode skipped.

# shellcheck source=.ci/pipelines/playwright-projects.sh
source "$DIR"/playwright-projects.sh

configure_schema_mode_runtime_env() {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This basically duplicates the same configure_schema_mode_runtime_env function for Helm. Can this be moved to a separate file/library, which is then sourced for both?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#4288 (comment)
The runtime change tests aren't executed on Operator now.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the tests are incorporated into nightly job, this doc should be updated with the changes.

testMatch: [
"**/playwright/e2e/external-database/verify-tls-config-with-external-rds.spec.ts",
"**/playwright/e2e/external-database/verify-tls-config-with-external-azure-db.spec.ts",
...(process.env.SCHEMA_MODE_DB_HOST
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still valid

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 1, 2026

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 1, 2026

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

Signed-off-by: Fortune-Ndlovu <[email protected]>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 1, 2026

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@Fortune-Ndlovu
Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly

@Fortune-Ndlovu
Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

Signed-off-by: Fortune-Ndlovu <[email protected]>
…+ non-SSL rejection); pending rerun confirmation.

Signed-off-by: Fortune-Ndlovu <[email protected]>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 2, 2026

Image was built and published successfully. It is available at:

@Fortune-Ndlovu
Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 2, 2026

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@Fortune-Ndlovu
Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 2, 2026

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

Signed-off-by: Fortune-Ndlovu <[email protected]>
@Fortune-Ndlovu
Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly

@Fortune-Ndlovu
Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

@Fortune-Ndlovu
Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly

@Fortune-Ndlovu
Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

The showcase-runtime deployment was failing with "Secret 'postgres-crt' not found"
because the secret was never created in the NAME_SPACE_RUNTIME namespace.

Changes:
- utils.sh: Create postgres-crt secret in configure_external_postgres_db()
- ocp-nightly.sh: Call configure_external_postgres_db before initiate_runtime_deployment
- ocp-operator.sh: Call configure_external_postgres_db before deploy_rhdh_operator

This ensures both postgres-crt and postgres-cred secrets exist before deploying
showcase-runtime, which is required by:
- values-showcase-postgres.yaml (Helm deployment)
- rhdh-start-runtime.yaml (Operator deployment)

Fixes the deployment failure that prevented schema-mode tests from running.

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Apr 2, 2026

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 2, 2026

Image was built and published successfully. It is available at:

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Apr 2, 2026

@Fortune-Ndlovu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-ocp-helm-nightly 5d1bcf9 link false /test e2e-ocp-helm-nightly
ci/prow/e2e-ocp-helm 5d1bcf9 link true /test e2e-ocp-helm
ci/prow/e2e-ocp-operator-nightly 5d1bcf9 link false /test e2e-ocp-operator-nightly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants