Dreamer sidebar reports a permanent classify-memories backlog that the executor never processes (expires_at filter mismatch)

[canomogollon]

Bug Report — Dreamer sidebar reports a permanent classify-memories backlog that the executor never processes

Short description

The TUI sidebar's Dreamer panel reports a classify-memories backlog that never decreases, because the sidebar's backlog counter counts expired memories while the classify executor filters them out. The two queries disagree on expires_at.

What happened?

Expected: the Dreamer sidebar's classify-memories counter should reach 0/N once all classifiable memories are scored, or at least reflect the same candidate set the executor actually processes.

Observed: the counter is stuck at a large pending value (e.g. 1020/5300) that never drains, even though the executor reports remaining=0 complete=true on every run.

Root cause (verified against source)

The sidebar backlog probe and the classify executor use different candidate filters:

1. Sidebar backlog — packages/plugin/src/features/magic-context/dreamer/task-gates.ts:

function countUnclassifiedActiveMemories(db: Database, projectPath: string): number {
    const row = db
        .prepare<[string], { cnt: number }>(
            `SELECT COUNT(*) AS cnt
               FROM memories
              WHERE project_path = ?
                AND status IN ('active','permanent')
                AND classified_at IS NULL`,
        )
        .get(projectPath);
    return row?.cnt ?? 0;
}

// ...
case "classify-memories": {
    const total = countActiveMemories(db, projectPath);
    return { pending: countUnclassifiedActiveMemories(db, projectPath), total };
}

This query does not filter expires_at.

2. Classify executor — packages/plugin/src/features/magic-context/dreamer/classify.ts:

function getClassifyCandidates(args: ClassifyArgs): ClassifyCandidate[] {
    const active = getMemoriesByProject(args.db, args.projectIdentity);
    // ...
}

getMemoriesByProject (in memory/storage-memory.ts) filters:

WHERE project_path = ?
  AND status IN (...)
  AND (expires_at IS NULL OR expires_at > ?)

This query does filter expires_at.

Consequence

Memories whose expires_at has passed are:

• counted by the sidebar as pending classification, but
• excluded by the executor from the classify candidate pool.

So the sidebar shows a backlog that the executor will never clear. In my database:

Query	Result
Sidebar (classified_at IS NULL, no expires_at filter)	1020
Executor (classified_at IS NULL and not expired)	4

The executor log confirms it is working correctly:

[dreamer] classify: stage=3 classified=77 changed=77 chunks=1 remaining=0 complete=true

remaining=0 complete=true is accurate for the executor's candidate set — but the sidebar still shows ~1020 pending.

Related observation: expired memories are never archived

expires_at is only used as a retrieval-time filter. Nothing transitions an expired memory to status='archived':

• countActiveMemories counts status IN ('active','permanent') with no expires_at filter.
• There is no archiveExpired / decay task in CANONICAL_DREAM_TASKS.
• curate archives by value/redundancy, not by expiry.

In my database, 1439 memories have expires_at in the past but remain status='active', accumulating since the TTLs (30d and 90d) started elapsing. They are invisible to retrieval but still counted by every backlog probe that uses countActiveMemories (map-memories, curate, compress-cues, classify-memories).

Repro steps

1. Have a project with memories that carry expires_at (TTL-based memories).
2. Let some of them expire (expires_at in the past) while still status='active'.
3. Open the TUI sidebar → Dreamer panel.
4. Observe classify-memories reporting a pending count that includes the expired memories.
5. Run classify-memories (scheduled or via /ctx-dream classify-memories).
6. Observe the executor log reporting remaining=0 complete=true, while the sidebar pending count does not reach 0.

Suggested fix

Make the backlog probes and the executor agree on the candidate set. Two options:

Option A (narrow): add the expires_at filter to countUnclassifiedActiveMemories so it matches getMemoriesByProject:

AND (expires_at IS NULL OR expires_at > ?)

Option B (broad): make countActiveMemories itself exclude expired memories, since every backlog probe that uses it (map-memories, curate, compress-cues, classify-memories) is really asking about the live pool. This would also fix the total denominator shown in the sidebar.

Option B is more consistent, but it changes the meaning of countActiveMemories for any caller that intentionally wants the raw status count — so it may need a separate countLiveActiveMemories helper.

Separately, consider whether expired memories should be archived by a Dreamer task (or by curate) so they do not accumulate indefinitely as status='active'.

Diagnostics

Plugin: v0.42.0
OS: linux x64
Node: v24.14.0
OpenCode: 1.18.30
Client: OpenCode TUI (CLI)
Storage versions: context_db_schema_version=84, plugin_supported_version=84
SQLite integrity_check: ok
Shared DB row counts: tags=106798, compartments=1982, memories=7382, notes=9, dream_runs=475

Relevant DB counts for the affected project:

-- Sidebar counter (no expires_at filter)
SELECT COUNT(*) FROM memories
 WHERE project_path = ? AND status IN ('active','permanent') AND classified_at IS NULL;
-- 1020

-- Executor candidate set (expires_at filtered)
SELECT COUNT(*) FROM memories
 WHERE project_path = ? AND status IN ('active','permanent')
   AND (expires_at IS NULL OR expires_at > <now>) AND classified_at IS NULL;
-- 4

-- Expired but still active
SELECT COUNT(*) FROM memories
 WHERE project_path = ? AND status = 'active'
   AND expires_at IS NOT NULL AND expires_at <= <now>;
-- 1439

Log output

[dreamer] classify: stage=3 classified=77 changed=77 chunks=1 remaining=0 complete=true
[dreamer] classify-memories: stage=3 classified=77 changed=77 chunks=1 remaining=0

Impact

• Functional: none — expired memories are filtered at retrieval and are not injected.
• Observability: high — the sidebar reports a permanent backlog that never drains, which reads as a stuck/failing Dreamer task.
• Storage: low — expired rows accumulate indefinitely (1439 rows in my case, ~1–2 MB).

Environment

• Plugin version: 0.42.0
• OpenCode version: 1.18.30
• Platform: linux x64
• Client: OpenCode TUI (CLI)

[magic-alfonso]

Confirmed, and thank you for the exact query pair — that is the whole defect. Two things were wrong: the Dreamer backlog probes counted every status='active' row while every executor selects through the expiry-filtered pool, so TTL-expired rows (from the legacy WORKFLOW_RULES/KNOWN_ISSUES TTLs) showed as pending forever; and nothing ever transitioned an expired row, so they accumulated as active.

Fixed on master (2151652): every task backlog (map-memories, verify, verify-broad, curate, compress-cues, classify-memories) now counts the same live pool the executor processes, so pending and total in the sidebar and /ctx-dream match what will actually run; and the weekly curate run now archives expired active rows (reason='expired') through the ordinary archive path under the memory lease — permanent rows are never touched, and the change is cache-neutral (it rides the next <memory-updates> delta rather than forcing a rebuild). Your 1,439 rows will be archived on the next curate run after upgrading.

Ships in the next patch release (v0.42.2).

[coleleavitt]

Current-baseline verification against 2d6a8af1 / upstream 5bf26e65: the released fix remains present and the focused gate/expiry coverage passes 8/8 locally. task-gates.ts now counts the executor-visible non-expired pool, while expire-memories.ts archives expired active rows through the module-aware path and preserves permanent rows. This independently confirms the prior resolution; recommendation remains close as fixed. Existing installations still need a successful post-upgrade curate run to archive accumulated expired rows.