Add secure scoped usage reporting

config.example.yaml

@@ -59,6 +59,16 @@ general_settings:
   redis_url: os.environ/REDIS_URL
   db_pool_size: 20
   db_pool_timeout: 30
+  # Reserve database capacity for gateway traffic by bounding expensive UI reports locally
+  # and across every API worker connected to the same PostgreSQL database.
+  spend_reporting_max_concurrency: 2
+  spend_reporting_global_max_concurrency: 2
+  spend_reporting_queue_timeout_seconds: 10
+  spend_reporting_execution_timeout_seconds: 60
+  spend_reporting_redis_timeout_seconds: 0.5
+  # Enable only after the reporting-v2 database check passes and every gateway
+  # and batch worker runs the reporting-v2 writer.
+  spend_reporting_v2_enabled: false
 
   # Upstream provider HTTP pool tuning. These are startup-time settings;
   # restart the process or roll the Kubernetes deployment after changing them.

docs/admin-ui/usage.md

@@ -12,6 +12,23 @@ Usage & Spend is the analytics surface for requests, tokens, and cost.
 - Breakdown by model, key, and team
 - Request-log view for operational inspection
 
+## Access scope
+
+The server filters every summary, trend, and breakdown before it reaches the browser:
+
+- Organization owners, billing users, and auditors see usage for their authorized organizations.
+- Team administrators see usage for teams they administer.
+- Regular organization and team members see usage generated by API keys owned by their account.
+- Platform administrators see platform-wide usage.
+
+Users with more than one reporting role get a compact **View** selector. Organization, team, and personal reports stay separate so totals and rankings always describe one clear scope. Self-scoped users get a simplified **Your usage** view: they can partition only their own usage by organization or team, then inspect the selected row's model mix. Detailed request logs, user/API-key dimensions, and tenant-wide rankings remain limited to elevated reporting roles.
+
+Self attribution uses the API key's trusted platform-account owner at request time, not caller-provided `user` or `end_user` values. It is also constrained to the account's current organization or team memberships. Shared-key traffic is therefore attributed to the recorded key owner without allowing former tenant members to retain visibility.
+
+Organization and team breakdowns in **Your usage** retain this same owner-account predicate. Selecting a row narrows the personal report; it does not expose usage generated by other accounts in that organization or team.
+
+Scoped team and personal reporting is protected by `general_settings.spend_reporting_v2_enabled`, which defaults to `false`. For a rolling deployment, first deploy the schema, ownership writers, and reporting API to every gateway and batch worker with the flag disabled. Verify the migration and fleet version, then enable the flag in a separate rollout. Organization and platform reporting remain available while the flag is disabled.
+
 ## Two main modes
 
 - **Overview**: charts and spend breakdown tables
@@ -20,3 +37,9 @@ Usage & Spend is the analytics surface for requests, tokens, and cost.
 ## When to use it
 
 Use this page for cost review, tenant reporting, and identifying which models or keys are driving traffic.
+
+## Legacy attribution
+
+New spend events and managed batch jobs snapshot their key owner's account automatically. Legacy rows without an owner snapshot remain unattributed and are intentionally excluded from self-scoped reports. DeltaLLM does not infer historical ownership from the key's current owner because keys may have been rotated, transferred, or deleted.
+
+Unattributed organizations, teams, users, and models are displayed explicitly as unassigned or unspecified. This state is represented separately from the entity identifier, so a legitimate identifier or model named `__unassigned__` remains distinct and can be inspected normally.

docs/api/admin.md

@@ -433,12 +433,17 @@ Asset-access writes emit the matching scope audit action, such as `ADMIN_ORGANIZ
 
 Supported report parameters include:
 
-- `group_by=model|api_key|team|user`
+- `group_by=model|provider|day|api_key|organization|team|user`
+- `view=organization|team|self` for accounts with multiple reporting views
 - `start_date`
 - `end_date`
-- `include_logs`
-- `page`
-- `page_size`
+- `interval=day|week|month` for time series
+- `scope_type=organization|team|user` with exactly one of `scope_id=<id>` or `scope_unassigned=true` for a model drill-down
+- `limit` and `offset` for grouped reports
+
+Grouped rows return a nullable `group_key` and an explicit `is_unassigned` boolean. Clients must use `is_unassigned` rather than reserving a string identifier; a real model or tenant identifier such as `__unassigned__` remains an ordinary assigned value.
+
+For `view=self`, the report accepts only `group_by=organization|team` in addition to the common day, model, and provider groupings. A model drill-down may use only `scope_type=organization|team`. Both the grouping and drill-down retain the authenticated account's owner predicate and active membership boundary, so they partition personal usage without exposing another account's traffic. User/API-key groupings and request logs remain unavailable to self-only accounts.
 
 ### Batches
 

docs/configuration/general.md

@@ -39,6 +39,12 @@ general_settings:
   database_url: os.environ/DATABASE_URL
   db_pool_size: 20
   db_pool_timeout: 30
+  spend_reporting_max_concurrency: 2
+  spend_reporting_global_max_concurrency: 2
+  spend_reporting_queue_timeout_seconds: 10
+  spend_reporting_execution_timeout_seconds: 60
+  spend_reporting_redis_timeout_seconds: 0.5
+  spend_reporting_v2_enabled: false
   upstream_http_connect_timeout_seconds: 10
   upstream_http_read_timeout_seconds: 300
   upstream_http_write_timeout_seconds: 30
@@ -195,9 +201,19 @@ Recommended steady state:
 | `database_url` | — | PostgreSQL connection string |
 | `db_pool_size` | `20` | Maximum database connection pool size |
 | `db_pool_timeout` | `30` | Connection pool timeout in seconds |
+| `spend_reporting_max_concurrency` | `2` | Maximum cache-miss spend reports executing concurrently per API worker |
+| `spend_reporting_global_max_concurrency` | `2` | Maximum reporting transactions executing across all workers connected to the same PostgreSQL database |
+| `spend_reporting_queue_timeout_seconds` | `10` | Maximum time a spend report waits for a reporting query slot |
+| `spend_reporting_execution_timeout_seconds` | `60` | Maximum execution time after a reporting slot is acquired, including database connection acquisition and query execution |
+| `spend_reporting_redis_timeout_seconds` | `0.5` | Per-operation Redis deadline for reporting cache coordination before falling back to a guarded database load |
+| `spend_reporting_v2_enabled` | `false` | Enables gated team and personal usage views after the reporting-v2 database and fleet rollout is complete |
 
 Pool settings are applied by appending Prisma's `connection_limit` and `pool_timeout` query parameters to the effective database URL at startup.
 
+Keep both reporting concurrency limits comfortably below the database pool size so gateway authentication, spend writes, and control-plane operations retain database capacity. PostgreSQL advisory-lock slots enforce the global limit without blocking, while the per-worker limit bounds local queues. Cache hits do not consume reporting query slots. Reporting concurrency and timeout settings are applied to subsequent report loads when dynamic configuration changes; active loads retain the immutable limits with which they started.
+
+Leave `spend_reporting_v2_enabled` disabled during the initial rolling deployment. Follow the [scoped usage reporting rollout](../deployment/usage-reporting-v2.md) before enabling it.
+
 Environment overrides:
 - `DELTALLM_DATABASE_URL`
 - `DELTALLM_DB_POOL_SIZE`

docs/deployment/index.md

@@ -12,6 +12,7 @@ Use this section when you are moving from local evaluation to a repeatable envir
 | Batch production setup | Async embedding/chat workloads with dedicated workers and shared storage | [Batch API & Production Setup](../features/batching.md#recommended-production-setup) |
 | Batch webhook rollout | Optional terminal callbacks, split workers, alerts, rollout, and rollback | [Batch Webhook Rollout](batch-webhook-rollout.md) |
 | Upstream HTTP tuning | Production provider concurrency, streaming, and egress capacity planning | [Upstream HTTP Tuning](upstream-http.md) |
+| Scoped usage rollout | Safe activation of team and personal usage reporting | [Scoped Usage Reporting](usage-reporting-v2.md) |
 
 ## Quick Path to Success
 
@@ -60,5 +61,6 @@ The application runs Prisma schema setup automatically during container startup.
 - [Railway deployment guide](railway.md)
 - [Kubernetes deployment guide](kubernetes.md)
 - [Upstream HTTP tuning](upstream-http.md)
+- [Scoped usage reporting rollout](usage-reporting-v2.md)
 - [Batch webhook rollout](batch-webhook-rollout.md)
 - [Observability](../features/observability.md)

docs/deployment/usage-reporting-v2.md

@@ -0,0 +1,61 @@
+# Scoped Usage Reporting Rollout
+
+Team and personal usage views depend on immutable API-key owner snapshots. They are disabled by default so schema changes and every ownership writer can be deployed safely before users rely on them.
+
+## Rollout
+
+1. Keep `general_settings.spend_reporting_v2_enabled: false`.
+2. On a large or write-heavy spend table, pre-create the two cursor indexes before deploying. Prisma cannot combine several concurrent index operations safely in one retryable migration; pre-creating the exact names makes the transactional migration a no-op for those indexes:
+
+   ```sql
+   CREATE INDEX CONCURRENTLY IF NOT EXISTS "deltallm_spendlog_events_org_time_id_idx"
+     ON "deltallm_spendlog_events"("organization_id", "start_time", "id");
+   CREATE INDEX CONCURRENTLY IF NOT EXISTS "deltallm_spendlog_events_time_id_idx"
+     ON "deltallm_spendlog_events"("start_time", "id");
+   ```
+
+3. Deploy the release to every gateway and batch worker. Startup applies the owner columns, immutable batch-snapshot markers, compatibility triggers, and retry-safe transactional index migrations. Schema and index lock acquisition is capped at five seconds, so a busy table fails deployment instead of waiting indefinitely or queuing gateway writes behind the migration.
+
+   If the owner-scope migration times out, allow the blocking transaction to finish or schedule the migration for a quieter window. The migration is atomic and does not need manual schema cleanup. Mark only that failed attempt rolled back, then rerun deployment:
+
+   ```bash
+   uv run prisma migrate resolve --rolled-back 20260810140000_spend_owner_scope \
+     --schema prisma/schema.prisma
+   uv run prisma migrate deploy --schema prisma/schema.prisma
+   ```
+
+   If the cursor-index migration times out because step 2 was skipped, create both indexes concurrently, then recover only that failed migration and rerun deployment:
+
+   ```bash
+   uv run prisma migrate resolve --rolled-back 20260810120000_spend_log_cursor_indexes \
+     --schema prisma/schema.prisma
+   uv run prisma migrate deploy --schema prisma/schema.prisma
+   ```
+4. For a large or write-heavy spend table, create the owner index concurrently after the owner-column migration commits. If the separate owner-index migration timed out, create the index, mark only that failed attempt rolled back, and rerun the deployment:
+
+   ```sql
+   CREATE INDEX CONCURRENTLY IF NOT EXISTS "deltallm_spendlog_events_owner_time_id_idx"
+     ON "deltallm_spendlog_events"("owner_account_id", "start_time", "id");
+   ```
+
+   ```bash
+   uv run prisma migrate resolve --rolled-back 20260810150000_spend_owner_scope_index \
+     --schema prisma/schema.prisma
+   uv run prisma migrate deploy --schema prisma/schema.prisma
+   ```
+
+5. Run the read-only database check:
+
+   ```bash
+   DATABASE_URL='postgresql://...' uv run python scripts/check_spend_reporting_v2_readiness.py
+   ```
+
+   A zero exit status and `"ready": true` confirm that all migrations completed, snapshot-completeness columns have their fail-safe defaults, required indexes are valid, and rolling-compatibility ownership triggers are enabled. Upgraded writers mark snapshots complete even when a key is deliberately ownerless, so steady-state traffic bypasses compatibility lookups. Existing batch sessions and jobs remain unattributed instead of being assigned to a later key owner. The check deliberately does not infer fleet versions; confirm that separately in your deployment platform.
+6. Confirm every gateway and batch worker is on this release, then set `spend_reporting_v2_enabled: true` and perform a configuration rollout.
+7. Sign in as a regular user and verify that **Usage** shows only **Your usage**. Verify a team administrator can switch between team and personal views, and an organization owner can switch between organization and personal views.
+
+## Rollback
+
+Set `spend_reporting_v2_enabled: false` first. This immediately hides team and personal views while keeping platform and organization reporting available. Keep the additive columns, indexes, and compatibility triggers in place during a code rollback; removing them while older and newer writers overlap can permanently lose owner attribution.
+
+Historical spend rows whose owner was not recorded remain unattributed. Do not backfill them from current API-key ownership because keys can be transferred or deleted after the request occurred.

docs/features/authentication.md

@@ -247,21 +247,23 @@ DeltaLLM separates platform roles, organization roles, and team roles.
 | Role | Typical access |
 |------|----------------|
 | `org_owner` | Full organization control |
-| `org_admin` | Manage teams, users, and keys |
+| `org_admin` | Manage teams, users, and keys; view usage from owned keys |
 | `org_billing` | Spend-focused visibility |
 | `org_auditor` | Read-only operational visibility |
-| `org_member` | Basic organization membership |
+| `org_member` | Basic organization membership; view usage from owned keys |
 
 ### Team Roles
 
 | Role | Typical access |
 |------|----------------|
-| `team_admin` | Manage the team and its keys |
-| `team_developer` | Use and create keys, self-service key creation (`key.create_self`) |
-| `team_viewer` | Read-only access |
+| `team_admin` | Manage the team and its keys; view team usage |
+| `team_developer` | Use and create keys, self-service key creation (`key.create_self`), view usage from owned keys |
+| `team_viewer` | Read-only access, including usage from owned keys |
 
 The `team_developer` role includes the `key.create_self` permission, which allows developers to create, regenerate, revoke, and delete their own API keys when the team has self-service enabled. See [API Keys: Self-Service](../admin-ui/api-keys.md#self-service-key-creation) for details.
 
+Personal usage is attributed from the API key owner's immutable request-time snapshot. A later key transfer does not move earlier synchronous or batch usage to the new owner, and an intentionally ownerless key remains unattributed rather than falling back to its current owner.
+
 ### Important Note on `user_role`
 
 `deltallm_usertable.user_role` is metadata for user profile types in the UI. It is not the main authorization source of truth.

prisma/migrations/20260810120000_spend_log_cursor_indexes/migration.sql

@@ -0,0 +1,49 @@
+-- Prisma runs multi-statement PostgreSQL migrations in a transaction. Keep the
+-- migration atomic and bound lock acquisition; production tables should
+-- pre-create these exact indexes concurrently as described in the rollout guide.
+SET LOCAL lock_timeout = '5s';
+SET LOCAL statement_timeout = '15min';
+
+-- Repair an invalid target left by an interrupted manual concurrent build.
+DO $$
+BEGIN
+    IF EXISTS (
+        SELECT 1
+        FROM pg_class AS index_class
+        JOIN pg_index AS index_meta ON index_meta.indexrelid = index_class.oid
+        JOIN pg_namespace AS namespace ON namespace.oid = index_class.relnamespace
+        WHERE namespace.nspname = current_schema()
+          AND index_class.relname = 'deltallm_spendlog_events_org_time_id_idx'
+          AND (NOT index_meta.indisvalid OR NOT index_meta.indisready)
+    ) THEN
+        DROP INDEX "deltallm_spendlog_events_org_time_id_idx";
+    END IF;
+END;
+$$;
+
+CREATE INDEX IF NOT EXISTS "deltallm_spendlog_events_org_time_id_idx"
+ON "deltallm_spendlog_events"("organization_id", "start_time", "id");
+
+DO $$
+BEGIN
+    IF EXISTS (
+        SELECT 1
+        FROM pg_class AS index_class
+        JOIN pg_index AS index_meta ON index_meta.indexrelid = index_class.oid
+        JOIN pg_namespace AS namespace ON namespace.oid = index_class.relnamespace
+        WHERE namespace.nspname = current_schema()
+          AND index_class.relname = 'deltallm_spendlog_events_time_id_idx'
+          AND (NOT index_meta.indisvalid OR NOT index_meta.indisready)
+    ) THEN
+        DROP INDEX "deltallm_spendlog_events_time_id_idx";
+    END IF;
+END;
+$$;
+
+CREATE INDEX IF NOT EXISTS "deltallm_spendlog_events_time_id_idx"
+ON "deltallm_spendlog_events"("start_time", "id");
+
+-- The cursor indexes cover the same leading-column lookups as their shorter
+-- predecessors, so remove the redundant indexes to avoid extra spend-log write cost.
+DROP INDEX IF EXISTS "deltallm_spendlog_events_org_time_idx";
+DROP INDEX IF EXISTS "deltallm_spendlog_events_start_time_idx";