fix(storage): write column descriptions where UI and MCP read them (#624)

[padak]

Problem

storage describe-column / describe-batch stored column descriptions as flat KBC.column.{name}.description keys in table metadata — a convention nothing but kbagent itself ever read. Documented columns looked blank in the Keboola UI, were invisible to the MCP server's get_table_detail/search, and never reached Snowflake COMMENT / BigQuery column descriptions. The failure was silent in both directions because table-detail read back its own convention.

Analysis of keboola/ui and keboola/connection (details in the issue comment): the UI writes through PUT .../tables/{id}/definition and the backend mirrors that write into columnMetadata KBC.description — one native write is visible to the UI, the MCP server, and the warehouse. The metadata-only fix suggested in the issue would have left descriptions out of the warehouse and unprotected from Output Mapping overwrite (isDescriptionSystemManaged stays true).

Changes

Write path — describe-column / describe-batch now write via the native definition endpoint:

• PUT /v2/storage/branch/{branch}/tables/{id}/definition with columns: [{name, description}] + isDescriptionSystemManaged: false (async; the CLI waits for the tableDefinitionUpdate storage job)
• unknown column names fail fast before any write (behavior change: the flat write accepted anything)
• remaining legacy flat entries on the touched table are auto-migrated as part of the same write

Read path — table-detail resolves each column description with precedence:

1. native definition.columns[].definition.description (aliases: sourceTable.definition)
2. columnMetadata[col] KBC.description (aliases: sourceTable.columnMetadata, MCP-server parity)
3. legacy flat KBC.column.*

plus an always-present legacy_column_descriptions list and a human-mode warning pointing at describe-migrate. Reads never write.

Migration — new storage describe-migrate --project ALIAS [--table-id ID ...] [--bucket-id ID] [--prune-orphans] [--dry-run] [--yes] [--branch ID]:

• scope: explicit tables XOR one bucket XOR whole project; scan-then-confirm; per-table error accumulation
• conflict (target already has a differing visible description) → skip + report, newer value wins
• orphan (column renamed/deleted) → skip; deleted only with --prune-orphans
• migrated flat entries are deleted so a description later cleared through the native path cannot be resurrected by the read fallback

Client (layer 3) — update_table_definition + delete_table_metadata on KeboolaClient; set_table_metadata docstring corrected (the "no user-writable column-metadata endpoint" claim was wrong).

Structure — the Descriptions command panel moved to commands/_storage_describe.py and the service logic to services/_column_descriptions.py (file-size budgets; follows the _storage_snapshots.py precedent).

Docs — 0.88.0 + changelog; AGENT_CONTEXT, CLAUDE.md command list, commands-reference, gotchas (since v0.88.0), storage-describe-workflow rewritten, SKILL.md regenerated, keboola-expert one-line trigger (61 991 / 62 000 bytes).

Testing

• make check green: 5738 passed, 12 skipped (lint, format, changelog, skill, sentinel-guard, loc checks included)
• new unit coverage: client payload/branch/job-error tests; service migration rules (conflict / orphan / identical / user-wins / delete-failure); describe-migrate scoping, dry-run, error accumulation; table-detail precedence incl. alias fallbacks; CLI exit codes, confirm-abort, --deny-writes
• E2E: _test_storage_describe extended with a native round-trip assertion and a seeded-legacy-key migrate cycle (requires E2E_API_TOKEN/E2E_URL to run)
• not verified live against a real stack in this PR — the E2E harness run is the follow-up

Closes #624

---

[devin-ai-integration]

Devin Review found 2 potential issues.

[padak]

Live verification (project 5946, Snowflake)

Ran the E2E suite against a real stack. The storage path is now verified end-to-end, not just mocked.

Full suite: 93 passed, 13 skipped, 9 failed — none of the failures are regressions from this PR:

Failures	Cause	Status
test_full_cli_e2e (1)	This PR's own new E2E assertion seeded the legacy key on a column that already had a native description, so the service correctly returned conflict and the assertion misread it as a migration failure	Fixed in 75aaaf7 — re-run passes
TestE2ENotificationSubscriptions (7)	Pre-existing: those tests read the CLI envelope one level too high (data["subscriptions"] instead of data["data"][...]). Last touched by ae840ba (#615, 2026-08-19); this PR's diff touches 0 notification lines. The command itself is fine — verified live	Out of scope, tracked separately
test_timeout_triggers_remote_kill_and_exits_seven (1)	Timing race — job still processing when the test asserted terminated	Flaky, unrelated

Re-run of test_full_cli_e2e after the fix: 1 passed (6:51). The migration cycle behaved exactly as designed against the live API:

describe-migrate --dry-run  -> tables_scanned 1, tables_migrated 0, migrated [<the legacy column>]
describe-migrate --yes      -> tables_scanned 1, tables_migrated 1
table-detail                -> description present, legacy_column_descriptions []
describe-migrate --yes      -> tables_migrated 0, migrated []          (idempotent no-op)
describe-migrate --yes      -> tables_migrated 0, migrated []          (conflict correctly skipped)

Positive-only column patch (the review question) was confirmed by a separate live probe: described three columns, re-described one, the other two kept their descriptions verbatim. Details in the resolved review thread.

Remaining unverified: nothing on the write path. The columnMetadata mirror that makes descriptions visible to the MCP server is backed by reading keboola/connection (TableDefinitionUpdateService → StorageColumnDefinitionSynchronizer.syncColumnDescriptions / syncTypedColumnInfo → replaceStorageColumnMetadata), not by an MCP round-trip from here.

[padak]

Rebased onto main (#629 landed first)

fix(storage): surface the table definition on table-detail (#621) (#629) extracted get_table_detail into services/_table_detail.py, which is exactly the function this PR changes — a semantic conflict, not just a textual one. Resolved by rebase (4 commits, linear on top of #629):

• the read-path precedence (native definition → columnMetadata KBC.description → legacy flat key, incl. the alias sourceTable fallback) and legacy_column_descriptions now live in _table_detail.py, where the transform moved. definition from storage table-detail drops the table definition (BigQuery partitioning/clustering) from the API response #621 passes through untouched.
• corrected the docstring fix(storage): surface the table definition on table-detail (#621) #629 inherited in _split_descriptions: it still asserted "Keboola has no user-writable column-metadata endpoint", the very claim this PR disproves. That sentence is now marked as the pre-0.88.0 belief and the legacy tier is labelled as such.
• StorageService keeps inheriting ColumnDescriptionsMixin (which extends BaseService), so the direct BaseService import is gone.

No version bump. Both branches had independently claimed 0.88.0; since 0.88.0 is unreleased, this PR's changelog notes now sit under the existing 0.88.0 key alongside #621's, and pyproject.toml is byte-identical to main. The release covering all three in-flight PRs is cut separately.

make check after the rebase: 5752 passed, 12 skipped — includes #629's own tests/test_storage_table_definition.py.

UI verification is also done: all three column descriptions render on the Schema tab of a live table (project 5946), confirming an independent consumer sees what the CLI wrote.