[hary73]

Summary

Chat History is moving to listing only sessions stamped with metadata["tfg.source"] = "ui" (#1118). Sessions created before that stamp would vanish from history, so this adds a one-time data migration that stamps every existing interactive session.

Related: AGE-2528

Changes

• New migration 20261022_000001_session_tfg_source_backfill (Postgres + SQLite mirror) that sets tfg.source = ui on sessions where:
  • source IS NULL (not a schedule run), and
  • external_id IS NULL (schedule runs are the only sessions with an external id; this also excludes schedule runs created before the source column existed), and
  • the key is not already present (never overwrites an existing value).
• Existing metadata (e.g. is_create_agent) is preserved; updated_at is not touched, so list ordering is unchanged.
• New migration 20261022_000002_session_tfg_source_list_idx (Postgres + SQLite mirror): session (tenant_id, (created_by_subject->>'subject_id'), updated_at, session_id, (metadata->>'tfg.source')). Serves a caller's newest-first listing (keyset on updated_at, session_id) without a sort, with or without the tfg.source filter; the filter is checked on index entries without visiting the table.
• The SQLite list query is unchanged, so the SQLite index is for schema consistency only for now.
• Postgres listSessions now adds metadata->>key = value per metadata filter pair next to the existing metadata @> …. Values are strings, so results are unchanged; it lets the planner use per-key expression indexes, which @> alone cannot. Other keys still use session_metadata_gin. The key is inlined as a literal (sql.lit; keys are validated to [A-Za-z0-9._:-]) because expression indexes only match a literal key, not a bound parameter under generic plans.
• down (backfill) is irreversible, matching the other data backfills: backfilled rows are indistinguishable from UI-stamped ones.

Rollout note

This must ship in the same release as (or after) #1118, which switches the stamp/filter key to tfg.source. Sessions created between this migration running and the UI starting to stamp would otherwise be hidden from Chat History.

How was this tested?

Manually, end to end on Postgres 17:

1. Booted the server from main (no backfill) and created: an inline New Chat session, a builder session (is_create_agent: true), a named Try-agent session, an SDK-style session with custom metadata (foo: bar), a session already carrying tfg.source: other, a session with external_id but no source (pre-source schedule run), and two schedule sessions via Run now.
2. Restarted the server on this branch; only 20261022_000001_session_tfg_source_backfill ran.
3. Result:
   • The four interactive sessions gained tfg.source: ui, keeping their existing keys.
   • tfg.source: other was left as is.
   • Both Run-now sessions and the external_id-only session were left untouched.
   • updated_at unchanged on every row.
   • GET /sessions?created_by_me=true&metadata[tfg.source]=ui (the Chat History query) returns exactly the four interactive sessions; the unfiltered list still returns all sessions.

Index check (Postgres 17, 200k seeded sessions, 50 users, ~1/3 tagged ui):

• Before the query change, the Chat History query used no metadata index (walked the tenant newest-first, ~3k rows filtered per page).
• With it, PostgresSessionStore.listSessions (captured via auto_explain) uses Index Scan Backward using session_tfg_source_list_idx, no sort, ~0.2 ms per page; pages are disjoint and all ui.
• Filters on other keys still use session_metadata_gin.
• Postgres session-store contract suite passes, including metadata containment filtering.

---

Note

Medium Risk
Irreversible bulk UPDATE on session must deploy in step with the Chat History filter change; mis-timed rollout could hide sessions from history.

Overview
Prepares for Chat History filtering on metadata["tfg.source"] = "ui" by backfilling that key on legacy interactive sessions and speeding up the filtered list query on Postgres.

A one-time migration (Postgres + SQLite) merges tfg.source: ui into session.metadata for rows with no source, no external_id, and no existing tfg.source, without changing updated_at or overwriting other metadata. The backfill down is intentionally irreversible.

A new composite index session_tfg_source_list_idx (mirrored on SQLite for schema parity) supports per-user session lists in keyset order with optional tfg.source filtering. Postgres listSessions now adds redundant metadata->>'key' = value predicates alongside metadata @> so the planner can use expression indexes like the new one while other metadata keys still hit session_metadata_gin.

^{Reviewed by Cursor Bugbot for commit ee970ea. Bugbot is set up for automated code reviews on this repo. Configure here.}

[changeset-bot]

🦋 Changeset detected

Latest commit: ee970ea

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@truefoundry/trueforge	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 805ab35. Configure here.}

[chiragjn]

1. why do we have updated at? when listing sessions do we use updated_at or created_at
2. Also it should be before (metadata->>'tfg.source'), I expect people to query by time range always with optionally metadata filtering

[hary73]

why do we have updated at? when listing sessions do we use updated_at or created_at

Yes, when listing we order by updated_at DESC

Also it should be before (metadata->>'tfg.source'), I expect people to query by time range always with optionally metadata filtering

the start / end params in the list API filter on created_at, not updated_at - so moving it up won't help for time range queries. We already have an index for supporting these queries session_list_idx (tenant_id, created_at, session_id)

But even if we don't supported range queries on updated_at, it makes sense to move updated_at above in the case where a user would just want to see their recent sessions. The current index performs better on the chat view for users which have a lot of scheduled sessions, but moving updated_at above will create a more generic index.