Conversation
ErmiasG
force-pushed
the
HWORKS-3206
branch
3 times, most recently
from
October 2, 2026 07:48
f44172a to
8e0f56e
Compare
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 Trino catalogs and feature groups can now be shared with other projects. User catalogs are shared at catalog, schema, table or column level with column masks, and catalogs can be private to a user. Feature groups shared whole are read through the impersonating metastore catalogs, and feature groups shared with a subset of their features through the non-impersonating iceberg_shared, delta_shared and hudi_shared catalogs. The documentation adds a Sharing page under Query Engine (Trino) for catalog shares (levels, column restrictions, masks, statuses, overlap, revoking, received shares) and feature group shares (which catalog serves which share, what stays denied, which feature groups are not queryable). The catalogs guide gains private catalogs and points its access section at sharing, the mountable secrets guide gains the account-owned bundles private catalogs use, and the admin guide describes the backend-owned access-control rules and the shared feature store catalogs. The default Superset catalog is documented as delta. Screenshots are from a 5.2 cluster. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 A project now holds at most one share of any object in a catalog. A share inside or around one the project already has is refused, so no table is ever answered for by two shares and its access never depends on rule order. The case overlaps served, a broad share with less of one table, moves inside the share: a schema or catalog share carries per-table restrictions, the shared columns and masks of a table, and a restricted table with no shared column is left out. docs: the sharing guide describes table restrictions and the refusal of overlapping shares. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 A catalog is now shared with a project once. The share row is the grant to that project, and what it covers moves into its own tables: whole_catalog on the share, trino_catalog_share_schema for schemas shared or left out, trino_catalog_share_table for tables shared whole, narrowed or left out, and trino_catalog_share_column for the columns of a narrowed table. The narrowest row decides for an object, which is also the order the rules are composed in. Overlapping shares cannot exist any more, so the overlap check, its catalog lock and the per-table exception rows keyed by schema and table are gone. docs: the sharing guide describes one share per project and its scope. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 A project holds one share of a catalog. It covers the whole catalog or chosen schemas; a schema is shared whole or narrowed to chosen tables, and a table whole or narrowed to chosen columns, each optionally masked. Nothing is left out of a wider grant, so what a share covers is exactly its rows. A mask is checked against Trino the way Trino will apply it before the share is saved, and the owner can read a sample of a table through the chosen columns and masks. The sharing guide describes the nested scope and the one share per project. The catalogs guide says the default catalogs are listed once Default is added to the Type filter. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 An adversarial review of catalog sharing found ways a narrowed share could expose columns, ways the rules publication could restore revoked grants or leave rules unconfirmed, and places where the sharing page could not save what it showed. The sharing guide lists the hidden columns a narrowed table denies and the connectors whose tables cannot be narrowed, says that a renamed or added column and an Iceberg or Delta Lake time-travel read are open until the share is saved again, corrects what happens to a mask that reads another table and how a failed or not-found share is fixed, and states that table procedures (ALTER TABLE ... EXECUTE) are not restricted by a share. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 A private catalog is written only from a project where its owner is a Data Owner and read from any other, the split a project catalog makes between its Data Owners and Data Scientists. Before, an owner who could only read a project could write what they read there into their private catalog and read it back in any other project. Administrators are denied the delta_shared, hudi_shared and iceberg_shared catalogs, which nothing queries as admin: Trino runs a SECURITY DEFINER view as its stored owner, so a view recorded in the metastore as owned by admin would otherwise read every feature group through a catalog that reads HopsFS as a superuser. Both were checked on Trino 483 with the chart rules: a Data Scientist owner reads and cannot insert, create, drop or call system.execute, and admin cannot reach delta_shared. docs: the catalogs guide and the administrator guide say so. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 The guides document the behaviours they left out. A private catalog is written only from a project where its owner is a Data Owner, and why: a project the owner can only read cannot be copied out through it. A deleted account's private catalogs stay denied to everyone, a later account of the same username included, until they are removed. A reader of a JDBC catalog can run system.query, which the source database runs as the catalog's database user; the receiving project of a share cannot run the catalog's functions at all. A catalog being deleted, or the project that owns it, cannot be given a share; a share never narrows the owner's own access; and a receiving project sees how many columns are masked but not the mask expressions. The administrator guide says private catalogs are written only as Data Owner. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 The table procedure gap is documented as it was measured on Trino 483. Read access to an Iceberg or Delta Lake table allows ALTER TABLE ... EXECUTE, since file-based access control does not check it (trinodb/trino#22874, HWORKS-3305): a Data Scientist of the project, a private catalog owner acting as one and a share grantee can all optimize, expire or roll back the table, and rollback_to_snapshot drops the writes made since. A table with a column mask refuses table procedures, CALL procedures are refused to everyone, and the connector read_only property refuses them to everyone while reading is unchanged. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 The sharing guide says the connector of a shared catalog cannot change until its shares are revoked, and why. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 The share dialog, share edit and catalog list screenshots are retaken on a fresh 5.2 cluster: the dialog is the checkbox tree with the preview and the sample, and the list shows the Type filter with Default added, so the default catalogs appear beside the project and private ones. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 The administrator guide says what the query engine administrator can read, and that the SQL console no longer reads a project's tables. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 The administrator guide gains Reading the rules file, for debugging access from the Files tab: where rules.json and its last good copy are and what their difference means, the principals and groups the rules match, the first-match order of every section and where the share rules sit in it, the base policy rule by rule, the rules each kind of share adds, with a composed example, and what to look for when a share does not behave. The example is the composer's output for the chart's base. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 HWORKS-3204 now grants the system functions of a user catalog, such as a JDBC catalog's system.query, to the project's Data Owners only. The Data Scientist rule this text described is gone, so it now names Data Owners. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 The sharing guide describes the new column handling: the backend reads a narrowed table's columns on every rules update and denies the ones the share did not check, with the window until the next update and the restart behaviour stated. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn level https://hopsworks.atlassian.net/browse/HWORKS-3206 The feature group section states that an append updates the rules before it returns, and that columns reaching the table outside Hopsworks are denied on the next rules update. Signed-off-by: Ermias Gebremeskel <ermias@hopsworks.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: https://hopsworks.atlassian.net/browse/HWORKS-3206
Draft until the feature lands: logicalclocks/hopsworks-ee#3391, logicalclocks/hopsworks-helm#2411, logicalclocks/hopsworks-front#2135 (themselves waiting on HWORKS-3204).
Changes
user_guides/projects/trino/sharing.md): catalog shares (one per project: the whole catalog or chosen schemas, tables and columns), hidden columns and the connectors whose tables cannot be narrowed, masks and their type rule, editing, statuses, objects that no longer exist, revoking, what a share does not restrict (table procedures, columns added later, time travel), shares a project received (mask expressions are not shown to it); that a share grants no functions of the catalog and never narrows the owner's own access; feature group shares through the query engine, with which catalog serves a whole or a subset share, what stays denied, and which feature groups are not queryable.system.querylets a reader run.rules.json(chart base policy plus per-share rules, validation, restore), and the non-impersonating shared feature store catalogs, which administrators are denied too.system,tpchandtpcds.read_onlyproperty refuse them), and a shared catalog's connector cannot change until its shares are revoked.trino_default_catalogdefaults todelta; the catalog dropdown includes shared catalogs.Screenshots are from a 5.2 dev cluster (seed users); the share dialog, share edit and catalog list were retaken on the final UI.
hopsworks-docs check(strict) andmarkdownlintpass locally.🤖 Generated with Claude Code