feat(rest): parse storage-credentials and bind a per-table FileIO from vended credentials by plusplusjiajia · Pull Request #719 · apache/iceberg-cpp

plusplusjiajia · 2026-06-10T08:20:54Z

What

Parse storage-credentials from the REST LoadTableResponse and use the vended credentials to bind a per-table FileIO, so a loaded Table can read/write its (S3-compatible) storage without any manual wiring.

Why

REST catalogs hand out short-lived, scoped storage access via vended credentials (STS). Until now LoadTableResult.storage_credentials was dropped on the floor and every table fell back to the catalog's default FileIO, so a vended-credential table could not actually access storage.

How

Serde: add storage-credentials (prefix + config) to the LoadTableResponse (de)serializer, aligned with Java LoadCredentialsResponseParser / CredentialParser.
Credential selection: SelectS3StorageCredential picks the longest matching s3-family prefix, matching Java VendedCredentialsProvider.
Per-table FileIO: implement the REST catalog's TableFileIO hook added in feat(rest): add session-aware REST catalog #750 — when the load response carries an s3-family vended credential, merge catalog + table config + the credential and build an arrow-fs-s3 FileIO (resolving oss:// and other S3-compatible schemes), replacing the fail-closed ValidateNoFileIOConfig
path; otherwise keep the catalog default. This fills the TODO(gangwu) left in feat(rest): add session-aware REST catalog #750.
S3 option alignment: prefer the standard client.region (fall back to legacy s3.region), strip the scheme from a full-URI s3.endpoint, and honor s3.path-style-access for virtual-hosted addressing

plusplusjiajia · 2026-06-16T08:49:47Z

Rebased onto main now that #750 merged. The per-table FileIO binding now lives in the TableFileIO hook — it builds an arrow-fs-s3 FileIO from the vended s3 credential, else keeps the fail-closed path, filling the TODO(gangwu) from #750. @wgtmac @gangwu PTAL.

wgtmac · 2026-06-22T15:51:15Z

+    const std::vector<StorageCredential>& credentials) {
+  const StorageCredential* best = nullptr;
+  for (const auto& cred : credentials) {
+    if (cred.prefix.starts_with("s3") &&


Prefix selection needs the actual storage path. The REST spec and Java S3FileIO choose the longest prefix that matches each path; picking the longest S3 prefix globally makes unmatched paths and other S3 prefixes use the wrong credential.

@wgtmac Good catch. Fixed: SelectS3StorageCredential now picks the longest credential prefix that is actually a prefix of the table's storage location (passed down from metadata->location), instead of the longest s3 prefix globally — matching the REST spec / Java S3FileIO. It also treats s3/s3a/s3n as equivalent so an s3:// vended prefix still matches an s3a:// location.

wgtmac · 2026-06-22T15:51:15Z

  // response. The current table commit response does not define config.
-  ICEBERG_ASSIGN_OR_RAISE(auto table_io, TableFileIO(context, table_config));
+  ICEBERG_ASSIGN_OR_RAISE(auto table_io,
+                          TableFileIO(context, table_config, /*storage_credentials=*/{}));


This rebuilds the returned table with an empty credential list after commit. CommitTableResponse does not carry storage-credentials, so a table loaded with vended credentials loses its FileIO credentials after the first commit.

@wgtmac Fixed. MakeTableFromCommitResponse now reuses the FileIO bound at load (threaded through TableScopedCatalog) instead of rebuilding it with an empty credential list.

wgtmac · 2026-06-22T15:51:15Z

+  for (const auto& [key, value] : table_config) {
+    properties[key] = value;
+  }
+  for (const auto& [key, value] : s3_cred.config) {


This makes the STS values static. Java S3FileIO keeps the credential set and refreshes through credentials.uri when s3.session-token-expires-at-ms is present; long scans or reused tables will fail after the vended token expires.

@wgtmac You're right that the vended STS values are static and will expire on long scans / reused tables. Implementing refresh properly needs a credential-aware FileIO with a background refresher that re-fetches via credentials.uri (mirroring Java's VendedCredentialsProvider + S3FileIO#scheduleCredentialRefresh) — a larger change than this PR. I've left an explicit TODO and would prefer to handle it as a follow-up. Shall I open an issue to track it? Happy to scope it there.

wgtmac · 2026-06-22T15:51:15Z

+  // Longest-prefix "s3" vended credential. Java's VendedCredentialsProvider
+  // resolves per path against the table location; we bind one at load time
+  // (fine for the common one-credential-per-table case).
+  if (const StorageCredential* s3_cred = SelectS3StorageCredential(storage_credentials);


storage-credentials can also carry GCS/ADLS credentials. This path silently ignores non-S3 credentials and falls back to the catalog FileIO; if S3-only is intentional for now, please add an explicit TODO or fail-fast note.

@wgtmac Addressed. When the server vends only non-S3 credentials (GCS/ADLS), TableFileIO now fails fast with NotSupported listing the prefixes, rather than silently using the catalog FileIO. If S3 credentials are vended but none covers this table's location, it falls through to the catalog FileIO (which may use ambient creds). S3-only support is intentional for now — see the TODO in TableFileIO.

plusplusjiajia force-pushed the feat-rest-storage-credentials branch 2 times, most recently from c1216fa to 00ef76a Compare June 14, 2026 10:03

plusplusjiajia changed the title ~~feat(rest): parse storage-credentials from LoadTableResponse~~ feat(rest): parse storage-credentials and bind a per-table FileIO from vended credentials Jun 15, 2026

plusplusjiajia mentioned this pull request Jun 16, 2026

feat(rest): add session-aware REST catalog #750

Merged

feat(rest): parse storage-credentials from LoadTableResponse

d276c5f

plusplusjiajia force-pushed the feat-rest-storage-credentials branch from 968d073 to f8c78cc Compare June 16, 2026 08:43

feat(rest): bind a per-table FileIO from vended credentials

43838fd

plusplusjiajia force-pushed the feat-rest-storage-credentials branch from f8c78cc to 43838fd Compare June 16, 2026 09:17

address review comments

924d8ba

wgtmac reviewed Jun 22, 2026

View reviewed changes

plusplusjiajia force-pushed the feat-rest-storage-credentials branch from dffa3b3 to 634a95e Compare June 23, 2026 08:22

fix(rest): preserve vended credentials on commit and match by location

9d876a9

plusplusjiajia force-pushed the feat-rest-storage-credentials branch from 634a95e to 9d876a9 Compare June 23, 2026 08:40

wgtmac mentioned this pull request Jun 24, 2026

feat(rest): vend storage credentials into per-table FileIO #729

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(rest): parse storage-credentials and bind a per-table FileIO from vended credentials#719

feat(rest): parse storage-credentials and bind a per-table FileIO from vended credentials#719
plusplusjiajia wants to merge 4 commits into
apache:mainfrom
plusplusjiajia:feat-rest-storage-credentials

plusplusjiajia commented Jun 10, 2026 •

edited

Loading

Uh oh!

plusplusjiajia commented Jun 16, 2026

Uh oh!

wgtmac Jun 22, 2026

Uh oh!

plusplusjiajia Jun 23, 2026

Uh oh!

wgtmac Jun 22, 2026

Uh oh!

plusplusjiajia Jun 23, 2026

Uh oh!

wgtmac Jun 22, 2026

Uh oh!

plusplusjiajia Jun 23, 2026

Uh oh!

wgtmac Jun 22, 2026

Uh oh!

plusplusjiajia Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

plusplusjiajia commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

plusplusjiajia commented Jun 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

plusplusjiajia commented Jun 10, 2026 •

edited

Loading