Support Iceberg REST catalog storage credentials in ForwardingFileIo by kaveti · Pull Request #28425 · trinodb/trino

kaveti · 2026-02-23T16:02:42Z

Make ForwardingFileIo implement SupportsStorageCredentials so that storage credentials from the Iceberg REST catalog LoadTableResponse are captured and their config is merged into properties(). Storage credential keys take precedence over base config properties.

This allows storage credential keys (e.g. s3.access-key-id, gcs.oauth2.token) to flow through io().properties() into fileIoProperties, where downstream consumers like
IcebergRestCatalogFileSystemFactory can use them.

Description

Support Iceberg REST catalog storage credentials in ForwardingFileIo

Why this change

Iceberg REST catalogs can return short-lived, scoped credentials alongside table metadata so that
the query engine can access the underlying storage (S3, GCS, ADLS, etc.) without needing long-lived
static credentials. There are two mechanisms for this in the Iceberg REST spec:

Config-based vended credentials — credential keys are embedded directly in the config map
of the loadTable response. Trino already supports this path today.
Storage credentials — a dedicated storage-credentials field in the loadTable response
that carries a list of StorageCredential objects, each scoped to a storage prefix with its
own config map. This was introduced as part of the Iceberg REST catalog spec and is used by
catalogs like Unity Catalog, Polaris, and Gravitino.

Trino currently ignores storage credentials entirely. The Iceberg library's RESTSessionCatalog
calls FileIO.setCredentials() after loading a table, but Trino's ForwardingFileIo does not
implement the SupportsStorageCredentials interface — so those credentials are silently dropped.

This means that any catalog server returning credentials via the storage-credentials field
(instead of or in addition to the config map) will fail with access denied errors when Trino
tries to read the table data.

What this change does

ForwardingFileIo now implements SupportsStorageCredentials. When the Iceberg library calls
setCredentials(), the credential configs are captured and merged into properties().

This is the same io().properties() map that Trino already reads as fileIoProperties throughout
the Iceberg connector (splits, page sources, metadata operations). So all existing downstream
credential handling (e.g. IcebergRestCatalogFileSystemFactory for S3) works without any
additional changes.

Storage credential keys take precedence over base config properties when there is a conflict.
For example, if the base config has s3.access-key-id=static-key and a storage credential
has s3.access-key-id=vended-key, the vended key wins.

Example

A REST catalog loadTable response might include:

{
  "metadata": { ... },
  "config": {
    "s3.access-key-id": "fallback-key"
  },
  "storage-credentials": [
    {
      "prefix": "s3://my-bucket/warehouse/",
      "config": {
        "s3.access-key-id": "AKIA...",
        "s3.secret-access-key": "...",
        "s3.session-token": "..."
      }
    }
  ]
}

Before this change, Trino would only see the config map (fallback-key). After this change,
Trino sees the merged result where storage credentials take precedence (AKIA...).

This works the same way for GCS (gcs.oauth2.token), Azure (adls.sas-token.*), or any other
provider — the change is cloud-agnostic.

Spec references

Iceberg REST catalog OpenAPI spec — LoadTableResult schema with storage-credentials field:
https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml

Additional context

#When storage-credentials is used vs config properties:

The Iceberg REST catalog has two mechanisms for vending credentials:

config map (older, ad-hoc) The REST server puts credential keys like s3.access-key-id, s3.secret-access-key directly into the config map of LoadTableResponse. This is what TestIcebergVendingRestCatalogConnectorSmokeTest exercises — the tabulario/iceberg-rest backend only supports this approach.
storage-credentials list (Iceberg REST Spec) The REST server returns a structured list of StorageCredential objects in LoadTableResponse.credentials(). Each credential has:

A prefix (e.g., s3://my-bucket/warehouse/) scoping the credential to a specific storage location
A config map with the actual credential key-value pairs
This enables:

Prefix-scoped credentials — least-privilege access per storage location
Multi-cloud support — different credentials for S3, GCS, ADLS in the same response
Short-lived, table-specific tokens — the catalog can vend narrowly-scoped STS tokens per table

#Which catalogs use storage-credentials?
Apache Polaris (Snowflake Open Catalog)
Unity Catalog (Databricks)
AWS Glue Data Catalog (REST endpoint)
Gravitino
The tabulario/iceberg-rest test backend predates this spec addition and only returns credentials via config, which is why you see properties being filled but not storageCredentials.

I've added TestIcebergStorageCredentialsRestCatalog which uses a mock REST catalog that injects storage-credentials into LoadTableResponse to validate this flow end-to-end. I've also updated the issue description with this clarification.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

kaveti · 2026-02-23T16:37:35Z

#Note: This change is the Trino-side prerequisite for Iceberg REST catalog storage credentials support.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/fileio/ForwardingFileIo.java

kaveti · 2026-02-28T17:01:36Z

@findinpath i have addressed your review comments

...test/java/io/trino/plugin/iceberg/catalog/rest/TestIcebergStorageCredentialsRestCatalog.java

...eberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/TrinoIcebergRestCatalogFactory.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/TrinoRestCatalog.java

kaveti · 2026-03-08T16:50:16Z

@findinpath could you please review again

kaveti · 2026-03-19T18:52:10Z

@findinpath could you please review again

findinpath · 2026-03-24T10:16:16Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/fileio/ForwardingFileIo.java

+        for (StorageCredential credential : credentials) {
+            merged.putAll(credential.config());
+        }
+        return merged.buildKeepingLast();


This logic makes not much sense to me.
We have the storageCredentials additionally to the fileIo.properties() (considered as a fallback) - meaning that they're separate, not mixed.

agree. we need to retain prefix scoped creds

findinpath · 2026-03-24T10:37:28Z

...eberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/TrinoIcebergRestCatalogFactory.java

            RESTSessionCatalog icebergCatalogInstance = new RESTSessionCatalog(
                    config -> HTTPClient.builder(config)
                            .uri(config.get(CatalogProperties.URI))
                            .withHeaders(RESTUtil.configHeaders(config))
                            .build(),
-                    (context, config) -> {
-                        ConnectorIdentity currentIdentity = (context.wrappedIdentity() != null)


It seems that the changes from apache/iceberg#12591 disregarded the fact that the ioBuilder is not receiving now the storageCredentials.

@nastra was it intentional to skip it during the implementation phase?

@findinpath this was not intentional. I believe Trino is the only engine that goes through that path. Can you please open a PR with a fix?

opened PR: apache/iceberg#15752 , cc @nastra

findinpath · 2026-03-24T10:39:26Z

...eberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/TrinoIcebergRestCatalogFactory.java

-                                : ConnectorIdentity.ofUser("fake");
-                        return fileIoFactory.create(fileSystemFactory.create(currentIdentity, config), true, config);
-                    });
+                    null);


I think we should have a preparatory commit doing this change to see better whether it makes any sense to have it.

findinpath · 2026-03-24T10:44:29Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/fileio/ForwardingFileIo.java

@@ -45,15 +52,23 @@
 import static java.util.stream.Collectors.joining;

 public class ForwardingFileIo
-        implements SupportsBulkOperations
+        implements SupportsBulkOperations, SupportsStorageCredentials


I'm looking at S3FileIO

https://github.com/apache/iceberg/blob/main/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java#L114

https://github.com/apache/iceberg/blob/main/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java#L102

and am rather thinking that we should have a prefixed map of TrinoFileSystem actually (and probably the default one we used to have as fallback).

findinpath

I think the PR needs to be intensively reshaped to have it moving forward.

findinpath · 2026-03-24T10:47:19Z

...test/java/io/trino/plugin/iceberg/catalog/rest/TestIcebergStorageCredentialsRestCatalog.java

+        warehouseLocation = Files.createTempDirectory(null);
+        closeAfterClass(() -> deleteRecursively(warehouseLocation, ALLOW_INSECURE));
+
+        DelegatingRestSessionCatalog delegatingCatalog = DelegatingRestSessionCatalog.builder()


This was introduced as part of the Iceberg REST catalog spec and is used by catalogs like Unity Catalog, Polaris, and Gravitino.

Isn't there an actual implementation we can test against instead of relying on mocked code?

electrum · 2026-03-24T18:19:57Z

lib/trino-filesystem-s3/src/main/java/io/trino/filesystem/s3/S3FileSystemFactory.java

+    private final String staticRegion;
+    private final String staticEndpoint;
+    private final boolean staticCrossRegionAccessEnabled;
+    private final Map<ClientKey, S3Client> dynamicClients;


This is not the right design. S3FileSystemLoader is already responsible for creating and managing clients. We shouldn't have a second system here for doing that.

findinpath · 2026-03-25T12:27:27Z

Waiting on apache/iceberg#15752

Consider rebasing in the meantime on #26640

cla-bot bot added the cla-signed label Feb 23, 2026

github-actions bot added the iceberg Iceberg connector label Feb 23, 2026

kaveti requested review from Praveen2112, dain, ebyhr, findepi and wendigo February 23, 2026 16:10

kaveti marked this pull request as draft February 23, 2026 16:48

findinpath self-requested a review February 24, 2026 16:44

findinpath reviewed Feb 24, 2026

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/fileio/ForwardingFileIo.java Show resolved Hide resolved

findinpath reviewed Feb 24, 2026

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/fileio/ForwardingFileIo.java Show resolved Hide resolved

findinpath reviewed Feb 24, 2026

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/fileio/ForwardingFileIo.java Show resolved Hide resolved

kaveti marked this pull request as ready for review February 28, 2026 17:01

kaveti force-pushed the storage-credentials-support branch 2 times, most recently from 0a7c730 to 882184c Compare March 1, 2026 07:29

findinpath reviewed Mar 2, 2026

View reviewed changes

...test/java/io/trino/plugin/iceberg/catalog/rest/TestIcebergStorageCredentialsRestCatalog.java Show resolved Hide resolved

findinpath reviewed Mar 2, 2026

View reviewed changes

...eberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/TrinoIcebergRestCatalogFactory.java Show resolved Hide resolved

findinpath reviewed Mar 2, 2026

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/TrinoRestCatalog.java Outdated Show resolved Hide resolved

kaveti force-pushed the storage-credentials-support branch 2 times, most recently from f6a6771 to 1e7f22e Compare March 6, 2026 17:05

findinpath reviewed Mar 24, 2026

View reviewed changes

kaveti added 2 commits March 24, 2026 18:40

Support vended credentials from Iceberg REST catalog

b30a90c

Support Iceberg V3 storage credentials in REST catalog

937535c

kaveti force-pushed the storage-credentials-support branch from 1e7f22e to 937535c Compare March 24, 2026 13:11

kaveti mentioned this pull request Mar 24, 2026

Core: Pass storage credentials to ioBuilder-created FileIO in RESTSes… apache/iceberg#15752

Merged

electrum requested changes Mar 24, 2026

View reviewed changes

findinpath added the stale-ignore Use this label on PRs that should be ignored by the stale bot so they are not flagged or closed. label Mar 25, 2026

Conversation

kaveti commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Support Iceberg REST catalog storage credentials in ForwardingFileIo

Why this change

What this change does

Example

Spec references

Additional context

Release notes

Uh oh!

kaveti commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaveti commented Feb 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaveti commented Mar 8, 2026

Uh oh!

kaveti commented Mar 19, 2026

Uh oh!

findinpath Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

kaveti Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

findinpath Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

nastra Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

kaveti Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

findinpath Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

findinpath Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

findinpath left a comment

Choose a reason for hiding this comment

Uh oh!

findinpath Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

electrum Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

findinpath commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

kaveti commented Feb 23, 2026 •

edited

Loading

kaveti commented Feb 23, 2026 •

edited

Loading

kaveti Mar 24, 2026 •

edited

Loading

findinpath Mar 24, 2026 •

edited

Loading