feat: read back when an instant database is due to be reaped

[rohan-hotdata]

The package can set a database lifetime and cannot read one back. This adds
hl.database_expiry and hl.database_expiries.

Follows #97, which is already merged. Both ship in the same release.

The gap

expires_at is written as a string, either an RFC 3339 timestamp or a relative window such
as "24h". It comes back as a datetime:

Model	Field type
CreateDatabaseRequest.expires_at	str | None
DatabaseDetailResponse.expires_at	datetime | None
DatabaseSummary.expires_at	datetime | None
ManagedDatabase	absent

The server resolves the relative window, so the resolved instant is only ever knowable by reading
it back. A caller that passed "24h" does not know which second it lands on, and a caller holding
a resolved ManagedDatabase cannot ask whether the database has a TTL at all.

That matters because a TTL is the whole disposal story for a short-lived database. The package
sets the thing and then cannot observe it.

Two functions

hl.database_expiry(client, db)   # datetime | None, for one database
hl.database_expiries(client)     # {database_id: datetime | None}, for the workspace

database_expiry reuses _database_detail, the same route database_attachments takes.

database_expiries costs one request per page of the listing, not one per database, because
DatabaseSummary already carries the field. A database with no TTL maps to None rather than
being absent from the mapping, so "lives forever" stays distinguishable from "not in this
workspace".

Why it reads the listing endpoint directly

client.list_managed_databases() is the obvious route and is wrong for this:

def list_managed_databases(self) -> list[ManagedDatabase]:
    listing = self._databases_api().list_databases()
    result: list[ManagedDatabase] = []
    for summary in listing.databases:
        try:
            detail = self._databases_api().get_database(summary.id)
            result.append(managed_database_from_detail(detail))
        except ApiException:
            pass
    return result

Three problems: it returns ManagedDatabase and so drops expires_at; it issues a
get_database per database; and it swallows ApiException per database, so one whose detail read
fails is omitted with no error. It also reads a single page.

That method is unchanged here. Its defects are noted on
#36, which already tracks framework
client gaps. Worth knowing that hotdata_list_managed_databases sits on that path.

Pagination

list_databases takes limit, cursor and search, so a single-page read would report a subset
of the workspace as though it were complete. database_expiries follows the cursor. It also stops
on an empty page and caps at 10,000 records, so a server that keeps returning a cursor cannot spin
the loop. Three tests cover the paging, including the cursor value sent on the second request.

Not on a tool

Reaping runs from the TTL or from an explicit cleanup step, so a model has no decision to make
with the value, and an unexplained timestamp in a tool result is surface it can only misuse. This
follows the line #91 drew for partition_by: the runtime supplies the fact, the model chooses
only when to act.

The asymmetry also favours waiting. Adding the field to hotdata_list_managed_databases later
stays additive. Removing it later would change what every model sees.

Verification

Gate	Result
Suite	634 passed (622 before, +12)
mypy	clean, 27 files
ruff check + format	clean
--resolution lowest-direct	634 passed
Documented examples	every snippet in both new README sections executed

Docs

• README — new "Reading back when a database expires" section, placed after the attach section
  from feat: attach a registered source into a database's scope, and read back when it expires #97.
• CHANGELOG — the expiry entry joins feat: attach a registered source into a database's scope, and read back when it expires #97's entries under [Unreleased]. While there I fixed
  three things in that block: database_attachments sat separated from the attach pair it belongs
  to, the closing paragraph claimed Provisioning boundary: agent-driven database, table, and index lifecycle #61's open question covered all of it when Provisioning boundary: agent-driven database, table, and index lifecycle #61 is about
  provisioning rather than these read-backs, and the expiry entry repeated the
  "not on a tool" line the closing paragraph already made.

Not in scope

No live run. Like #97, the endpoints and fields are verified against the SDK and the declared
floor, but this code path has not been exercised against a real workspace. TTL reaping itself
remains unobserved — this makes it observable, it does not confirm it happens.

[claude]

The scan cap counts distinct ids, so it does not stop a repeating cursor.

Failure scenario: the server answers every request with the same page of 100 summaries and the same next_cursor. seen holds 100 ids and never grows. cursor stays truthy and listing.databases stays non-empty. database_expiries loops forever and never returns, because the dict comprehension keeps overwriting the same 100 keys.

Count records read instead of distinct ids, and stop on a cursor already used:

Suggested change

	seen: set[str] = set()
	while True:
	try:
	listing = api.list_databases(cursor=cursor) if cursor else api.list_databases()
	except ApiException as e:
	raise RuntimeError(api_error_message(e)) from e
	for summary in listing.databases or ():
	yield summary
	seen.add(str(summary.id))
	cursor = getattr(listing, "next_cursor", None)
	# has_more alone has been seen paired with no cursor; without this the loop would
	# either stop early or repeat the first page forever.
	if not cursor or not listing.databases:
	return
	if len(seen) > _MAX_DATABASES_SCANNED:
	logger.warning(
	"stopped paging database listings after %d records; expiries are partial",
	len(seen),
	)
	return
	cursor: str | None = None
	scanned = 0
	used_cursors: set[str] = set()
	while True:
	try:
	listing = api.list_databases(cursor=cursor) if cursor else api.list_databases()
	except ApiException as e:
	raise RuntimeError(api_error_message(e)) from e
	for summary in listing.databases or ():
	yield summary
	scanned += 1
	cursor = getattr(listing, "next_cursor", None)
	# has_more alone has been seen paired with no cursor; without this the loop would
	# either stop early or repeat the first page forever.
	if not cursor or not listing.databases:
	return
	if cursor in used_cursors:
	logger.warning("database listing repeated cursor %r; expiries are partial", cursor)
	return
	used_cursors.add(cursor)
	if scanned > _MAX_DATABASES_SCANNED:
	logger.warning(
	"stopped paging database listings after %d records; expiries are partial",
	scanned,
	)
	return

Add a test for the repeating-cursor case. The current paging tests all end the sequence with a page that has no cursor, so none of them exercises the cap.

[claude]

nit: getattr with a None default hides a field-name mismatch (not blocking). If the SDK response model names the field something other than next_cursor, paging stops after page one and reports a subset as complete, which is the exact failure this function exists to prevent. Nothing catches it, because the tests build pages from SimpleNamespace rather than the SDK model. Read listing.next_cursor directly so a mismatch raises AttributeError, or assert the field name against the declared SDK floor in a test.

[claude]

super nit: the new constant splits the download constants (not blocking). MAX_DOWNLOAD_BYTES and DOWNLOAD_CHUNK_BYTES belong together. Move _MAX_DATABASES_SCANNED below DOWNLOAD_CHUNK_BYTES.

[claude]

super nit: the sentence starting "For the attach pair" has no main clause (not blocking). "For the attach pair that is the open question in #61 ... — which this release does not settle" reads as a fragment. Rewrite as: "For the attach pair, whether provisioning should be agent-callable at all is the open question in #61, which this release does not settle."

[claude]

Review

Blocking Issues

• hotdata_langchain/databases.py:361-380 — the 10,000 record cap counts distinct database ids, not records read. A server that returns the same page and the same cursor on every request never grows seen, so database_expiries loops forever. The cap is the stated defence against that case and does not cover it.

Action Required

1. Count records read, or reject a cursor already used, so a repeating cursor terminates the loop.
2. Add a test where the server keeps returning the same cursor. The three existing paging tests all end on a page with no cursor, so none reaches the cap.

Non-blocking comments are inline: the getattr(listing, 'next_cursor', None) default, constant placement, and one CHANGELOG sentence.

CI checks were queued or in progress when this review ran, so the test results are unknown here.

[rohan-hotdata]

All four findings addressed in 58c2235.

Blocking — the cap did not stop a repeating cursor. Correct, and worse than a missing guard:
the cap was written for exactly that case, and the code comment claimed it covered it. len(seen)
counted distinct ids, so a server answering every request with the same page and the same cursor
never grew the set, the loop never ended, and the dict comprehension kept overwriting the same
keys.

Fixed both ways the review suggested. Records read are now counted instead of distinct ids, and a
cursor arriving a second time ends the loop.

Four tests, since the point stands that none of the three existing paging tests reached this code
— they all ended on a page with no cursor:

• one page returned forever with the same cursor, which terminates after 2 requests
• two cursors alternating, caught on the third request
• a fresh cursor every time, so only the record cap can end it
• the cursor sequence actually sent, asserted as [None, "c1", "c2"]

getattr(listing, "next_cursor", None). Taken, and the reasoning was the right way round: a
field-name mismatch would stop paging after one page and report a subset as complete, which is the
single failure this function exists to prevent. Now read directly. Since that makes correctness
depend on the field name, I checked it against the declared floor rather than the lockfile —
ListDatabasesResponse at hotdata 0.9.0 has count, databases, has_more, limit,
next_cursor.

Constant placement. Moved below DOWNLOAD_CHUNK_BYTES so the download constants stay
together. Its comment now records why the cap counts records rather than ids, so the next reader
does not reintroduce the bug above.

CHANGELOG fragment. Taken as written; the sentence had no main clause.

Re-run locally: 638 passed (up 4), mypy clean, ruff clean, and the --resolution lowest-direct
floor job green.

On the CI caveat in the review: all nine checks passed on the previous head, including the full
3.10–3.14 matrix and the floor job.

[claude]

nit: the public docstring promises the whole workspace and the code can return a subset (not blocking). _MAX_DATABASES_SCANNED and the repeated-cursor stop both end paging with a logger.warning only, so a caller gets a truncated mapping and no programmatic signal. Distinguishing a complete read from a subset is the stated reason this function exists. Document both stops here, so a caller with more than 10,000 databases knows the log is the only notice.

[claude]

All four prior comments are addressed. The loop now counts records, reads listing.next_cursor directly, stops on a repeated cursor, and the new tests cover the repeating cursor, the two-cursor cycle, and the record cap. One non-blocking docstring nit inline.

[claude]

All five prior findings are addressed.

The repeating-cursor loop now terminates two ways: records read are counted rather than distinct ids, and a cursor arriving a second time ends paging. The four new tests cover both stops and the cursor sequence sent.

CI was still queued or in progress when this review started, so this approval does not rest on the test results.