Vault-managed Azure OpenAI credentials for mit-learn, learn-ai, and edxapp by blarghmatey · Pull Request #5361 · mitodl/ol-infrastructure

blarghmatey · 2026-08-10T19:16:08Z

What are the relevant tickets?

N/A. The design and its rationale live in docs/plans/vault-azure-openai-credentials-spec.md, which this PR updates as part of the change.

Description (What does it do?)

Gives mit-learn, learn-ai and edxapp/mitxonline access to Azure OpenAI using short-lived credentials that Vault mints per lease, instead of a static API key. This is purely additive — every app's existing OPENAI_API_KEY wiring is untouched, and both providers end up available side by side.

Five pieces:

OLVaultAzureSecretsEngine component (src/ol_infrastructure/components/services/vault.py) — mirrors OLVaultAWSSecretsEngine next door, but two details differ and are easy to get wrong by copying it: role TTLs are Go duration strings ("24h") rather than integer seconds, and the role name is passed as role= rather than name=. The second fails silently, creating a role under a generated name.
pulumi-azure-native + pulumi-azuread dependencies. Worth noting for image size: pulumi_azure_native is 155 MB on disk, against the 116 MB pulumi_aws we already carry — same order of magnitude as an existing dependency, so no new class of problem, but the venv grows ~160 MB.
infrastructure/azure/openai (new Pulumi project) — per environment, one resource group holding one Cognitive Services account per app consumer, three model deployments on each (gpt-4o, gpt-4o-mini, gpt-5.2), and the Azure AD application Vault uses as its own root credential, granted User Access Administrator scoped to the resource group rather than the subscription.
substructure/vault/azure (new Pulumi project) — mounts the Azure secrets engine at azure-openai with one role per consumer. Each role is scoped to its own consumer's account, which is what stops one app's dynamic credentials from reaching another app's endpoint.
Per-app wiring — Vault policy grants, a dynamic secret, and the non-secret endpoint/tenant env vars for each of the three apps.

Plus the delivery wiring: project-registry constants, a new pulumi-azure Concourse pipeline, and "azure" added to the Vault substructure chain.

Why one account per consumer rather than one shared account

Two Azure behaviours drove this, and both cut against the intuitive design:

Azure Monitor's Cognitive Services metrics are dimensioned by model deployment name, model name and status code — there is no stable caller dimension. Since Vault mints a fresh service principal every lease, the calling identity's object ID churns every 24h, so attributing usage per app by caller cannot work. Attribution has to come from the resource.
Azure OpenAI TPM quota is allocated per subscription, per region, per model and is shared across accounts. More accounts do not buy more throughput; they divide the same pool. Throughput isolation comes from per-deployment capacity allocation, which works in any layout.

One Vault root service principal per environment (scoped to that environment's resource group) keeps this at three admin-consent grants rather than nine.

Two findings that changed the spec

Answering the open question about edxapp setting names meant reading the plugin source, which turned up two things the spec had wrong. Both are now corrected in docs/plans/vault-azure-openai-credentials-spec.md:

The spec claimed config-source slot 17- was free. It is already 17-webhook-tokens-secrets.yaml, so the Azure file is 18-.
More significantly: delivering the settings does not make edxapp able to use them. The translations plugin reads provider credentials from exactly one place — settings.TRANSLATIONS_PROVIDERS[provider]["api_key"] (course_translations.py#L64-L76) — and dispatches to three hardcoded classes. There is no flat OPENAI_* read path and no Azure provider. Making this work needs a follow-up in mitodl/edx-extensions: an AzureOpenAIProvider on LiteLLM's azure/{deployment} form, and — the real work — exchanging the client_id/client_secret for a bearer token via ClientSecretCredential, since Vault issues a credential pair rather than a token.

That is why the Azure credentials are delivered as flat top-level settings in their own file rather than as another TRANSLATIONS_PROVIDERS block. The edxapp init container concatenates its config sources with cat rather than deep-merging them, so a second file emitting that key would produce a duplicate and silently clobber the deepl/openai/gemini/mistral providers — last one wins, no error anywhere. The plugin folds the flat settings into TRANSLATIONS_PROVIDERS in Python, where merging actually works.

How can this be tested?

Nothing in this PR takes effect on merge. All three apps' Azure wiring is gated on an azure_openai_tenant_id config value that is unset in every stack, so the only change any existing environment sees is an additive Vault policy grant. The gate exists because a VaultDynamicSecret aimed at a mount Vault does not have yet fails rather than degrading, so the cutover has to be per-environment and after the mount is deployed there.

Verified with real pulumi preview runs. In each case the only diff attributable to this change is the Vault policy:

Stack	Result
`ol-application-mit-learn` / `CI`	`~ vault:index:Policy ol-mitlearn-vault-policy-ci` only
`ol-application-learn-ai` / `CI`	`~ vault:index/policy:Policy learn-ai-vault-policy-ci` only
`ol-application-edxapp` / `mitxonline.CI`	`~ vault:index/policy:Policy edxapp-vault-policy` only

No resource replaced, no Azure resource created, OPENAI_API_KEY untouched, and TRANSLATIONS_PROVIDERS / 14-translations-providers-secrets.yaml unchanged. (Remaining diffs in those previews came from the placeholder Docker tag/digest env vars needed to run a preview locally, plus pre-existing Fastly gzip list-ordering noise.)

Also verified:

pre-commit run --all-files passes. The one failing hook, hadolint, fails on pre-existing Dockerfiles that this PR does not touch.
Both new Concourse pipeline modules generate their definition.json without error.
tests/ol_concourse/test_secrets_map.py — all 172 tests pass. This one matters: that suite re-derives the secrets map by static analysis and fails on drift, so the new secrets_map.py entries are required rather than optional.
Every pulumi-azure-native / pulumi-azuread API shape used here was checked against the installed packages rather than assumed (Account.properties.endpoint, ApplicationPassword.value, ServicePrincipal.object_id, DeploymentModelVersionUpgradeOption.NO_AUTO_UPGRADE).

What has not been tested: the two new Pulumi projects have never been previewed against a real subscription, because no Azure subscription has been chosen yet and the SOPS azure.{ci,qa,production}.yaml credential files therefore do not exist.

Additional Context

This is reviewable now but not deployable yet. One open question gates the whole Azure side — which subscription and tenant, whether Microsoft.CognitiveServices is registered and approved for Azure OpenAI, and how much regional TPM quota exists. That last one needs a number rather than a yes: this layout puts 27 deployments (9 accounts × 3 models) on one shared regional pool. Current capacity settings are 50k TPM for Production and 5k elsewhere; if the pool is small, that split needs revisiting before the first deploy.

A few decisions worth a reviewer's attention:

Model versions are deliberately not pinned in code. Which versions exist is a property of the subscription and region, not of this repo, and a wrong version string fails at Deployment create time rather than at preview. Left unset, Azure deploys its current default and upgrades it over time. The project README documents confirming them with az cognitiveservices account list-models and then pinning via azure_openai:model_versions in stack config — which also flips that deployment to NoAutoUpgrade, so a new default version cannot change model behaviour with no deploy and no diff.
The apps derive their endpoint rather than reading it from a stack reference. The infrastructure project sets each account's custom_sub_domain_name to the account's own name, so the URL follows from the environment. A StackReference would have coupled every mit-learn / learn-ai / edxapp deploy to a stack that does not exist yet — an unacceptable cost for an additive change.
The tenant ID doubles as the feature switch because it is the one value that cannot be derived.
disable_local_auth is left false on the Cognitive Services accounts. Flipping it is the final step of the migration, once nothing depends on key auth.
The policies extend sys/leases/renew and revoke allowed_parameters. Without that the vault-secrets-operator can read a credential once but cannot manage its lease afterwards. (edxapp's policy needed no such change — it grants sys/leases/renew without allowed_parameters.)
meta.py and secrets_map.py registration were not in the original plan but are both required. Without the meta.py entry the new pipeline file would exist and never be set; without the secrets_map.py entries the drift test fails.

Checklist:

Choose the Azure subscription/tenant and confirm Microsoft.CognitiveServices registration, Azure OpenAI approval, and regional TPM quota
Create the Pulumi deploy service principal out of band and add SOPS src/bridge/secrets/pulumi/azure.{ci,qa,production}.yaml
Deploy infrastructure/azure/openai before substructure/vault/azure for each environment
Run az ad app permission admin-consent once per environment — every resource applies cleanly without it and the failure only surfaces later, when Vault first tries to mint credentials
Set azure_openai_tenant_id per app, per environment, to switch the app wiring on

Copilot

Pull request overview

Adds Vault-minted, short-lived Azure OpenAI credentials for mit-learn, learn-ai, and mitxonline alongside existing API-key authentication.

Changes:

Provisions per-application Azure OpenAI accounts, deployments, and Vault root identity.
Adds the Vault Azure secrets engine, scoped roles, policies, and gated application wiring.
Registers deployment pipelines and updates dependencies and design documentation.

Reviewed changes

Copilot reviewed 27 out of 32 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`uv.lock`	Locks Azure Pulumi providers.
`src/ol_infrastructure/substructure/vault/azure/Pulumi.yaml`	Defines the Vault Azure project.
`src/ol_infrastructure/substructure/vault/azure/Pulumi.operations.QA.yaml`	Adds QA Vault configuration.
`src/ol_infrastructure/substructure/vault/azure/Pulumi.operations.Production.yaml`	Adds Production Vault configuration.
`src/ol_infrastructure/substructure/vault/azure/Pulumi.operations.CI.yaml`	Adds CI Vault configuration.
`src/ol_infrastructure/substructure/vault/azure/__main__.py`	Configures the Azure secrets engine and roles.
`src/ol_infrastructure/substructure/vault/azure/__init__.py`	Initializes the project package.
`src/ol_infrastructure/lib/pulumi_projects.py`	Registers the new Pulumi projects.
`src/ol_infrastructure/infrastructure/azure/openai/README.md`	Documents Azure resources and deployment prerequisites.
`src/ol_infrastructure/infrastructure/azure/openai/Pulumi.yaml`	Defines the Azure OpenAI project.
`src/ol_infrastructure/infrastructure/azure/openai/Pulumi.QA.yaml`	Configures the QA Azure region.
`src/ol_infrastructure/infrastructure/azure/openai/Pulumi.Production.yaml`	Configures the Production Azure region.
`src/ol_infrastructure/infrastructure/azure/openai/Pulumi.CI.yaml`	Configures the CI Azure region.
`src/ol_infrastructure/infrastructure/azure/openai/__main__.py`	Provisions accounts, models, identity, and authorization.
`src/ol_infrastructure/infrastructure/azure/openai/__init__.py`	Initializes the OpenAI project package.
`src/ol_infrastructure/infrastructure/azure/__init__.py`	Initializes the Azure infrastructure package.
`src/ol_infrastructure/components/services/vault.py`	Adds the reusable Vault Azure engine component.
`src/ol_infrastructure/applications/mit_learn/mitlearn_policy.hcl`	Grants mit-learn Azure credential access.
`src/ol_infrastructure/applications/mit_learn/k8s_secrets.py`	Creates mit-learn dynamic Azure credentials.
`src/ol_infrastructure/applications/mit_learn/__main__.py`	Adds gated Azure environment settings.
`src/ol_infrastructure/applications/learn_ai/learn_ai_policy.hcl`	Grants learn-ai Azure credential access.
`src/ol_infrastructure/applications/learn_ai/__main__.py`	Adds learn-ai credentials and configuration.
`src/ol_infrastructure/applications/edxapp/k8s_secrets.py`	Adds mitxonline Azure settings and credentials.
`src/ol_infrastructure/applications/edxapp/k8s_resources.py`	Mounts Azure configuration into LMS and CMS.
`src/ol_infrastructure/applications/edxapp/edxapp_mitxonline_policy.hcl`	Grants mitxonline Azure credential access.
`src/ol_concourse/pipelines/secrets_map.py`	Registers project secret dependencies.
`src/ol_concourse/pipelines/infrastructure/vault/pipeline.py`	Adds the Azure Vault substructure jobs.
`src/ol_concourse/pipelines/infrastructure/meta.py`	Registers the Azure pipeline.
`src/ol_concourse/pipelines/infrastructure/azure/pipeline.py`	Defines Azure deployment jobs.
`src/ol_concourse/pipelines/infrastructure/azure/__init__.py`	Initializes the pipeline package.
`pyproject.toml`	Adds Azure Pulumi dependencies.
`docs/plans/vault-azure-openai-credentials-spec.md`	Updates the design and implementation findings.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

blarghmatey · 2026-08-12T12:58:25Z

Addressed all six review threads from Copilot — four fixed in 15a82bd, two answered with a deliberate decision. All resolved; check bar is green.

Fixed: the three rollout-restart findings were correct and worth catching. Vault revokes the service principal at the role's max TTL, but pods read the client id and secret into their environment once at start — so without restart targets Azure auth would have started failing roughly 48h after a deploy rather than at deploy time, which is about the worst shape a failure can take. The reviewer was also right that the scope was wider than the webapp: env_from_secret_names reaches the celery workers and beat too, so learn-ai's single learn-ai-app target would have left three workloads on a revoked credential. Deployment names were taken from the live mitlearn and learn-ai namespaces rather than derived, since a target that doesn't match an existing Deployment fails silently. mit-learn's _create_dynamic_secret had no way to express this at all and gained an optional parameter.

Also fixed: added the Pulumi-mock tests components/ requires (10 tests), asserting the two API details this component exists to protect — role= rather than name=, and duration strings rather than integer seconds. I confirmed they actually bite: reintroducing the role=/name= trap fails 6 of the 10.

Decided, not fixed: the two findings about the new pipelines creating guaranteed-failing CI jobs are factually correct — I verified pulumi_jobs_chain sets trigger=True for non-production stacks, and this PR touches both watched paths. Keeping both registrations anyway, as a deliberate call: missing delivery wiring is the exact gap this PR exists to close (spec §6), and deferring it trades a visible red job for an invisible missing one. The azure substructure chain is a separate fragment, so it doesn't block the other Vault chains. Prerequisite ordering is on the pre-merge checklist in the description.

Still gated on the one open question — which Azure subscription and tenant, and how much regional TPM quota. Nothing here can deploy until that lands.

Ardiea · 2026-08-12T14:32:53Z

+            # per worker (underscores become dashes), "{app}-celery-beat".
+            restart_targets=[
+                OLVaultRestartTarget(kind="Deployment", name=name)
+                for name in (


Vault can mint short-lived Azure AD service principals scoped to a single Azure role, which is what we want for Azure OpenAI access instead of the static API keys each app carries today. This adds the component so the substructure project can mount the engine once and hand each app its own role. Two details differ from the AWS engine next door and are easy to get wrong by copying it: the role TTLs are Go duration strings rather than integer seconds, and the role name is passed as `role=` rather than `name=` -- the latter fails silently by creating a role under a generated name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BUoJUvFDsVUVqfE1gp6v29

Five of the six open questions are now decided; only the subscription and tenant remain. Q2 in particular is no longer a risk -- consent can be granted in-house, which is what made the per-consumer account layout affordable enough to choose. Reading the translations plugin to answer Q3 turned up two things the spec had wrong. The 17- config-source slot is already taken by webhook tokens, so the Azure file becomes 18-. More importantly, the plugin reads provider credentials only out of TRANSLATIONS_PROVIDERS and has no Azure provider at all, so shipping the settings from here does not by itself make edxapp able to use them -- the token exchange has to happen in the plugin, which is a separate repo. Written down so nobody discovers it mid-implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BUoJUvFDsVUVqfE1gp6v29

Two new Pulumi projects and the CI wiring to deploy them. The infrastructure project creates, per environment, one resource group holding one Cognitive Services account per app consumer; the substructure project mounts Vault's Azure secrets engine against those accounts with one role per consumer. The account boundary follows the consumer rather than being one shared account, because usage attribution has to come from the resource. Azure Monitor's Cognitive Services metrics carry no stable caller dimension, and Vault mints a fresh service principal on every lease, so grouping usage by calling identity cannot work here. Each role is scoped to its own consumer's account, which is also what stops one app's credentials reaching another's endpoint. Model versions are left unpinned on purpose. Which versions exist is a property of the subscription and region rather than of this code, and a wrong version string fails at deployment-create time rather than at preview, so the README documents how to confirm them and pin via stack config once known. Neither project can be previewed yet -- the subscription is still undecided and the SOPS credential files it implies do not exist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BUoJUvFDsVUVqfE1gp6v29

Each app gets a dynamic secret from the azure-openai mount holding a service principal scoped to its own Cognitive Services account, plus the non-secret endpoint and tenant it needs to exchange those for a token. The existing OPENAI_API_KEY wiring is untouched, so both providers are available at once. Both halves are gated on the app's azure_openai_tenant_id config rather than landing switched on. A VaultDynamicSecret pointed at a mount Vault does not have yet fails rather than degrading, so the cutover has to be per environment and after the mount exists -- and the tenant id is the natural switch because it is the one value that cannot be derived. The endpoint is derived from the environment, since the infrastructure project sets each account's custom subdomain to the account's own name. The policies also extend the sys/leases renew and revoke allowed_parameters: without those the vault-secrets-operator can read a credential once but cannot manage its lease afterwards. Previewed against CI for both stacks: the only attributable change is the policy update, no existing resource is replaced, and no Azure resources appear while the config is unset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BUoJUvFDsVUVqfE1gp6v29

Adds an 18-azure-openai-secrets.yaml config source, mitxonline-only, carrying Vault-minted credentials scoped to mitxonline's own Cognitive Services account alongside the endpoint and deployment the plugin needs to reach it. These are flat top-level settings rather than another TRANSLATIONS_PROVIDERS block on purpose. The init container concatenates the config sources with cat rather than deep-merging them, so a second file emitting that key would silently clobber the deepl/openai/gemini/mistral providers -- last one wins. The plugin folds these into TRANSLATIONS_PROVIDERS in Python instead, where merging actually works. Note this only delivers the settings. The translations plugin currently reads provider credentials solely from TRANSLATIONS_PROVIDERS and has no Azure provider, so making edxapp use these needs a matching change in mitodl/edx-extensions -- including exchanging the client id and secret for a bearer token, which Vault does not issue. Previewed against mitxonline.CI: the only attributable change is the policy update, and no config source is added while the config gate is unset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BUoJUvFDsVUVqfE1gp6v29

… component Vault replaces these credentials at the role's max TTL and revokes the service principal behind them, but pods read the client id and secret into their environment once at start. Without rollout targets every consumer keeps presenting a revoked credential until some unrelated deploy happens, so Azure auth starts failing ~48h after a deploy rather than at deploy time -- the worst shape for this kind of bug. The targets cover every workload that receives the secret, not just the webapp: env_from_secret_names reaches the celery workers and beat too. Names were checked against the deployments actually running in the mitlearn and learn-ai namespaces rather than derived from the naming helper. edxapp reuses the target list already assembled for its database credentials, and mit-learn's dynamic-secret helper grows an optional parameter, since it had no way to express this at all. Also adds the Pulumi-mock tests that components/ requires. They assert the two things this component exists to get right -- roles named via `role=` rather than `name=`, and duration strings rather than integer seconds -- and were confirmed to fail when either trap is reintroduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BUoJUvFDsVUVqfE1gp6v29

sentry · 2026-08-28T17:36:25Z

            redis_creds_secret_name,
            static_secrets_name,
            opik_keycloak_secret_name,
+            *([azure_openai_secret_name] if azure_openai_tenant_id else []),
        ],
        application_security_group_id=learn_ai_application_security_group.id,
        # Use the fixed name used in the SecurityGroupPolicy spec


Bug: The learn_ai_app_k8s resource is missing an explicit depends_on for the conditionally created azure_openai_secret, which can cause deployment race conditions.
_{Severity: HIGH}

Suggested Fix

Conditionally append the azure_openai_secret resource to the depends_on list for the learn_ai_app_k8s resource. This ensures Pulumi creates the secret before attempting to deploy the application that uses it.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: src/ol_infrastructure/applications/learn_ai/__main__.py#L846-L852 Potential issue: The `azure_openai_secret` is created when `azure_openai_tenant_id` is set, and its name is added to the `env_from_secret_names` list for the Kubernetes application. However, the secret resource itself is not added to the `depends_on` list of the `learn_ai_app_k8s` Pulumi resource. This creates a race condition. The application deployment may be applied before the secret exists in the cluster, causing pods to fail to start with a `CreateContainerConfigError`. This issue would also manifest during secret rotation, as new pods would fail to start with the new, not-yet-available secret.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

Copilot AI balanced review requested due to automatic review settings August 10, 2026 19:16

Copilot started reviewing on behalf of blarghmatey August 10, 2026 19:16 View session

Copilot AI reviewed Aug 10, 2026

View reviewed changes

Ardiea approved these changes Aug 12, 2026

View reviewed changes

blarghmatey and others added 6 commits August 28, 2026 13:32

blarghmatey force-pushed the vault-azure-openai-credentials branch from 15a82bd to baa96f4 Compare August 28, 2026 17:32

sentry Bot reviewed Aug 28, 2026

View reviewed changes

Navigation Menu

Uh oh!

Vault-managed Azure OpenAI credentials for mit-learn, learn-ai, and edxapp - #5361

Vault-managed Azure OpenAI credentials for mit-learn, learn-ai, and edxapp#5361
blarghmatey wants to merge 6 commits into
mainfrom
vault-azure-openai-credentials

blarghmatey commented Aug 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blarghmatey commented Aug 12, 2026

Uh oh!

Ardiea Aug 12, 2026

Uh oh!

sentry Bot Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

blarghmatey commented Aug 10, 2026

What are the relevant tickets?

Description (What does it do?)

Why one account per consumer rather than one shared account

Two findings that changed the spec

How can this be tested?

Additional Context

Checklist:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blarghmatey commented Aug 12, 2026

Uh oh!

Ardiea Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

sentry Bot Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants