[HWORKS-3206] Share Trino catalogs at catalog, schema, table and column level

docs/setup_installation/admin/superset.md

@@ -204,9 +204,11 @@ In **Cluster Settings**, choose **Configuration** under _Infrastructure_ in the
 #### trino_default_catalog
 
 - **Description**: Default catalog to use for the Offline Feature Store Connection
-- **Default**: `hive`
+- **Default**: `delta`
 - **Values**: `hive`, `delta`, `iceberg`, and `hudi`.
 
+The value is applied when a member's connection is created, so connections created before a change keep the catalog they were created with.
+
 Trino connections are created with multi-catalog enabled (Superset's **Allow changing catalogs** option), so users are not limited to this default.
 They select the catalog matching their table format from the **Catalog** dropdown in SQL Lab or when adding a dataset.
 Querying a table whose format does not match the selected catalog raises a Trino `UNSUPPORTED_TABLE_TYPE` error; see the [Trino table type error][trino-table-type-error] troubleshooting in the Superset user guide.

docs/setup_installation/admin/trino.md

@@ -224,6 +224,136 @@ A catalog whose `${HOPSWORKS_SECRET:<name>}` reference no longer resolves cannot
 The repair reports it, leaves any file it already has in place, because that copy resolved when it was approved and still works, and carries on with every other catalog.
 Its owner has to repoint the reference at an existing secret.
 
+## Access control and sharing
+
+The query engine decides who can read what with Trino's file-based access control, from a rules file published into the Trino files store as `access-control/rules.json`.
+Hopsworks owns that file and rebuilds it whenever a share changes, and on a schedule every five minutes by default.
+
+The file is composed from two parts:
+
+- The base policy, from the Helm value `trino.accessControl.rules`, which the chart renders into the ConfigMap `hopsworks-trino-access-control-base`.
+  It grants each project its own catalogs and feature store, and each user their private catalogs, written only from projects where the user is a Data Owner.
+  Administrators see every catalog but read only `system`, `tpch` and `tpcds`, because the query engine's administrator is also the identity Hopsworks itself uses, and a view recorded as owned by it would otherwise read any project's data.
+  The administrators' SQL console therefore cannot read a project's tables; query them as a member of the project.
+- One set of rules per share, for [catalog shares and feature group shares][sharing-catalogs-and-feature-groups].
+  A share names the receiving project's existing `<project>__data_owner` and `<project>__data_scientist` groups, so sharing never changes the group file.
+
+Change the base policy through the Helm value and an upgrade.
+An edit to the published `rules.json` is overwritten by the next rebuild, within minutes.
+
+Every rebuilt file is validated before it is published, and a file that fails validation is not published: the shares that caused it are marked **Failed** with the reason, and the file in place stays as it was.
+After publishing, Hopsworks checks that the query engine still answers once it has re-read the file, and restores the last file that worked if it does not, because Trino refuses every query while its rules file is unreadable.
+
+### The shared feature store catalogs
+
+The chart ships two kinds of catalog over the feature store:
+
+- `delta`, `hudi`, `iceberg` and `hive` impersonate the querying user, so HopsFS permissions apply on top of the access-control rules.
+  They serve a project's own feature groups, and feature groups or feature stores shared whole, which HopsFS grants the receiving project.
+- `delta_shared`, `hudi_shared` and `iceberg_shared` do not impersonate.
+  They read HopsFS as the `trino` user, which is a HopsFS superuser, because a feature group shared with a subset of its features grants the receiving project no HopsFS access.
+
+For the second kind the access-control rules are the only gate.
+The base policy grants nobody access to them, not even administrators, and Hopsworks adds a rule per subset share that allows the receiving project the shared features of that one table and denies the rest.
+They are read-only at the connector as well, so no rule can let a query write through them.
+Do not add rules for these catalogs to the base policy: any rule that reaches one of them reads every project's feature store.
+Administrators are denied them because Trino runs a view as the user recorded as its owner, so a view recorded as owned by an administrator would reach them too.
+
+### Reading the rules file
+
+The rules the query engine enforces can be read under **Cluster Settings** → **Query Engine** → **Files**, as `access-control/rules.json`.
+Beside it, `access-control/rules.json.last-good` is the last file the query engine loaded.
+They differ from a publish until Hopsworks confirms the query engine loaded the new file, a few seconds later.
+If they stay different, the new file is not confirmed yet, for example because the query engine was unreachable, and the next reconcile checks again.
+A file the query engine refused does not stay: the last good file goes back in its place, and the shares the refused file added are marked **Failed**.
+The groups the rules name are in `auth/group.db`.
+
+#### Who the rules match
+
+A query runs as a principal named `<project>__<username>`, for example `seeda__seed1000` for user `seed1000` in project `seeda`.
+Its groups are the member's role in that project, `<project>__data_owner` or `<project>__data_scientist`, and `<owner>__shared_featurestore` for each project `<owner>` whose feature store is shared with that project.
+Group `admin` has one member, the query engine administrator, which is the identity Hopsworks itself uses.
+Project names and usernames cannot contain `__`, so a pattern such as `.*__(.*)` splits a principal unambiguously, and `$1` in a later field stands for what the pattern captured.
+
+Each section of the file (`catalogs`, `schemas`, `tables`, `functions`, `queries`) is checked on its own.
+In a section, the first rule whose user, group and object all match decides, and a request no rule matches is denied.
+The order of the rules is therefore the policy: a broader rule placed first would answer before a narrower one.
+
+#### The order of the rules
+
+Every section keeps the same order, and the rules Hopsworks adds for shares go in one place in it:
+
+1. The administrator rules.
+2. A deny for each private catalog whose owner's account was deleted, until the catalog is removed.
+   It comes before the private-owner rules because a later account with the same username would match them.
+3. The private-owner rules.
+   They come before the shares so that sharing a private catalog with a project the owner belongs to never narrows the owner's own access.
+4. The share rules.
+5. The rest of the base policy: every project's own catalogs and feature store.
+
+#### The base policy
+
+The `catalogs` section of the base policy, in order:
+
+| Rule | Effect |
+| --- | --- |
+| `group: admin`, `allow: none` on `iceberg_shared`, `delta_shared` and `hudi_shared` | The administrator never sees the shared feature store catalogs. |
+| `group: admin`, `catalog: .*`, `allow: read-only` | The administrator sees every other catalog, without writing to any. |
+| `user: .*__(.*)`, `group: .*__data_owner`, `catalog: _$1__.*`, `allow: all` | The owner of a private catalog reads and writes it from a project where they are a Data Owner. |
+| `user: .*__(.*)`, `catalog: _$1__.*`, `allow: read-only` | The owner reads it from any other project. |
+| `catalog: tpch` and `tpcds`, `allow: read-only` | Everyone reads the sample catalogs. |
+| `catalog: iceberg`, `delta`, `hive`, `hudi`, `allow: all` | Everyone reaches the feature store catalogs; the table rules decide what they read. |
+| `group: (.*)__data_owner`, `catalog: $1__.*`, `allow: all` | A project's Data Owners read and write its catalogs. |
+| `group: (.*)__data_scientist`, `catalog: $1__.*`, `allow: read-only` | Its Data Scientists read them. |
+| `catalog: system`, `allow: read-only` | Everyone reads the `system` catalog. |
+
+The `tables` section follows the same pattern: the administrator reads only `system`, `tpch` and `tpcds`, the private-owner rules mirror the catalog ones, and each project reaches the schema `<project>_featurestore` in the feature store catalogs, all of it for its Data Owners, reading for its Data Scientists and for projects its feature store is shared with.
+The `schemas` section gives schema ownership, which is what creating and dropping schemas needs, to Data Owners only.
+The `functions` section lets everyone run builtin functions, and the Data Owners of a project run the `system` functions of their project's catalogs, such as `system.query` on a JDBC catalog.
+
+#### The rules a share adds
+
+Hopsworks writes the names in a share rule as literals between `\Q` and `\E`, so a name containing regular expression syntax matches only itself.
+A share names the receiving project's two role groups in one pattern, `\Q<project>\E__data_(?:owner|scientist)`, so searching the file for `\Qseedc\E__data_` finds every rule a share to `seedc` added.
+
+A share of catalog `seeda__postgresql` with `seedc`, covering table `public.customers` with column `created` unchecked and column `name` masked, adds these rules:
+
+```json
+{"catalogs": [
+  {"group": "\\Qseedc\\E__data_(?:owner|scientist)", "catalog": "\\Qseeda__postgresql\\E", "allow": "read-only"}
+],
+"tables": [
+  {"group": "\\Qseedc\\E__data_(?:owner|scientist)", "catalog": "\\Qseeda__postgresql\\E",
+   "schema": "\\Qpublic\\E", "table": "\\Qcustomers\\E", "privileges": ["SELECT"],
+   "columns": [{"name": "created", "allow": false}, {"name": "$path", "allow": false},
+               {"name": "name", "mask": "'***'"}]},
+  {"group": "\\Qseedc\\E__data_(?:owner|scientist)", "catalog": "\\Qseeda__postgresql\\E",
+   "schema": "\\Qpublic\\E", "table": "\\Qcustomers\\E\\$.*", "privileges": []}
+],
+"functions": [
+  {"group": "\\Qseedc\\E__data_(?:owner|scientist)", "catalog": "\\Qseeda__postgresql\\E", "privileges": []}
+]}
+```
+
+The list of denied columns in the rule is shortened here.
+
+- The catalog rule makes the catalog visible to the receiving project, read-only.
+- The table rule grants `SELECT` on the table and lists the columns it denies: the unchecked ones, and the connector's hidden columns, such as `$path`.
+  A table shared whole has a table rule without `columns`, a schema shared whole has `table: .*`, and a catalog shared whole has `schema: .*` too.
+- The rule after it, with no privileges, denies the table's metadata tables, such as `customers$partitions`, which Trino checks by their own name.
+- The function rule denies the receiving project the catalog's functions, which the base policy would otherwise give it.
+  A share of a private catalog also adds, before that deny, a rule letting the owner keep running the catalog's `system` functions.
+
+A feature group shared whole adds one table rule on `hive|iceberg|delta|hudi` for its table in the owner's feature store.
+A feature group shared with a subset of its features adds a catalog rule on the shared catalog of its format, such as `delta_shared`, and a table rule there that denies every unshared feature and hidden column, followed by the metadata table deny.
+
+#### Debugging a share
+
+- The share is **Active** but a query is refused: find the share's rules by the receiving project's group, then look for a rule above them that matches the same principal and object first.
+- The share's rules are not in the file: the share is still **Applying**, or it is **Failed** and its status says why.
+- A column that should be hidden is readable: it is missing from the rule's `columns`, typically a column added after the share was saved; saving the share again adds it as unshared.
+- `rules.json` and `rules.json.last-good` differ for minutes: the newest file is not confirmed, so check that the query engine is reachable; the reconcile verifies it again and restores the last good file if the query engine refuses it.
+
 ## Credential files a project supplies
 
 A connector that authenticates with a file, such as an Oracle wallet or a Java keystore, cannot be served by a catalog property alone.
@@ -311,7 +441,8 @@ Trino behavior can be customized through cluster configuration variables. To mod
 **Available Variables:**
 
 - **trino_enabled**: Enable or disable Trino cluster-wide (default: `false`)
-- **trino_default_catalog**: Default catalog used for Superset queries (default: `hive`)
+- **trino_default_catalog**: Default catalog of the Superset database connections created for new project members (default: `delta`).
+  Connections created before a change keep the catalog they were created with.
 - **trino_test_coordinator_enabled**: Enable the optional test coordinator that backs the "Test connection" action for user-created catalogs (default: `true`)
 - **trino_catalog_reconcile_enabled**: Rebuild the user-catalog Secrets from the database on a schedule, for a cluster that has lost them (default: `false`, see [Recovering catalog files lost from the mount][recovering-catalog-files-lost-from-the-mount])
 - **trino_catalog_max_per_project**: Catalogs a *newly created* project may create (default: `10`).

docs/user_guides/projects/mountable_secrets/mountable_secrets.md

@@ -12,8 +12,9 @@ A **mountable secret** is a named bundle of files that belongs to your project.
 You upload the files once, then refer to the bundle by name from a catalog property, and Hopsworks substitutes the real location when the catalog is written for Trino.
 The files are stored where project members cannot read or write them directly, and a catalog can only ever reach its own project's bundles.
 
-Only a project Data Owner can list, create or delete mountable secrets.
+Only a project Data Owner can list, create or delete a project's mountable secrets.
 Through the API the same endpoints need an API key with the `MOUNTABLE_SECRET` scope.
+Private catalogs use mountable secrets that belong to your account instead, described in [Mountable secrets for private catalogs][mountable-secrets-for-private-catalogs].
 
 ## Creating a bundle
 
@@ -136,6 +137,27 @@ connection-password=${HOPSWORKS_SECRET:oracle_password}
 Take the host, port and `service_name` from the alias's entry in the wallet's `tnsnames.ora`, and drop `retry_delay`, which means nothing once `retry_count` is zero.
 A catalog can keep this form, and doing so records which consumer group it connects to instead of leaving it to an alias name.
 
+## Mountable secrets for private catalogs
+
+A [private catalog][private-catalogs] follows its owner into every project they are a member of, so it cannot reference a project's bundles: it would carry them into the owner's other projects.
+It references bundles that belong to your account instead.
+
+Open **Account Settings**, then **Secrets**, and use the **Mountable secrets** section below your secrets.
+Creating, listing and deleting work as for a project's bundles, and the same limits apply, counted per account rather than per project.
+
+<figure>
+  <img src="../../../../assets/images/guides/mountable_secrets/account-mountable-secrets.png" alt="Mountable secrets on the account Secrets page" />
+  <figcaption>Bundles that belong to your account, for use by your private catalogs from any project</figcaption>
+</figure>
+
+A private catalog references your bundles with the same `${HOPSWORKS_MOUNT:<bundle>}` forms, and a project catalog cannot reference them.
+Names, sizes, hashes and timestamps of your bundles are visible to you alone.
+The same caution applies as for a project's bundles: where the cluster runs a Trino test coordinator, every mountable secret on it is readable from that coordinator.
+
+Through the API, your account's bundles are at `/users/mountable-secrets`, with the same operations as a project's and an API key with the `MOUNTABLE_SECRET` scope.
+
+When your account is deleted, your bundles are deleted with it.
+
 ## When the feature is unavailable
 
 An administrator can turn the store off for a whole cluster.

docs/user_guides/projects/superset/superset.md

@@ -84,7 +84,7 @@ SQL Lab is an interactive SQL query interface for exploring your feature data.
 </figure>
 
 For the Trino connection, SQL Lab shows a **Catalog** dropdown between **Database** and **Schema**.
-The dropdown appears because the connection has *Allow changing catalogs* enabled, and it lists every Trino catalog (`hive`, `delta`, `iceberg`, `hudi`).
+The dropdown appears because the connection has *Allow changing catalogs* enabled, and it lists every Trino catalog you can read (`hive`, `delta`, `iceberg`, `hudi`, your project's catalogs, and catalogs shared with your project).
 Selecting the catalog that matches your table format lets you query the table without prefixing the catalog in the SQL.
 
 <figure>