GH-34: serve queries from a per-process in-memory index

.agents/conventions.yaml

@@ -2,7 +2,7 @@ project: knowledge-mcp
 version: 1
 entries:
     - id: conv-001
-      summary: Go project standards: gofmt, go vet, go test, GH-N commits
+      summary: "Go project standards: gofmt, go vet, go test, GH-N commits"
       detail: |-
         Format: gofmt (make fmt runs gofmt -s -w .). gofmt -l must output nothing.
         Verify: go test ./... and go vet ./... must pass before work is claimed done; go build ./... must compile all packages.
@@ -13,7 +13,7 @@ entries:
       source: code review, 2026-09-08
       date: "2026-09-08"
     - id: conv-002
-      summary: Knowledge entry content rules for this project's own store
+      summary: "Knowledge entry content rules for this project's own store"
       detail: |-
         Detail is 10-20 lines of query-able facts, not source copies.
         Summary is a topic line (max 100 chars); enforcement text goes in the rule field, which list_knowledge surfaces as constraints.

.agents/decisions.yaml

@@ -2,37 +2,36 @@ project: knowledge-mcp
 version: 1
 entries:
     - id: dec-001
-      summary: Org-level knowledge is indexed per markdown section, not as whole files
+      summary: "Org-level knowledge is indexed per markdown section, not as whole files"
       detail: |-
         Whole-file indexing made org queries return 5-6K tokens of unrelated content (measured 2026-09-08: "daisy test-stability" on chaon-io returned full architecture.md and review.md).
         Split .agents/knowledge/*.md on "## " headings; each section is its own document with key org-<file>::<heading>.
         list_knowledge for an org groups sections by source file; --global entries are merged into org listings with a (global) label.
-        Required metaFileVersion bump to force index rebuilds on existing installs.
-      source: user-requested token-savings fix, 2026-09-08
-      date: "2026-09-08"
+        No version bump is needed anymore: the index is rebuilt from the store files on every startup (GH-34 in-memory index).
+      source: user-requested token-savings fix, 2026-09-08; GH-34 update, 2026-10-07
+      date: "2026-10-07"
     - id: dec-002
-      summary: Merged global entries carry explicit (global) provenance in list output
+      summary: "Merged global entries carry explicit (global) provenance in list output"
       detail: |-
         list_knowledge for any non-global project prepends _global entries; previously they were indistinguishable from the project's own entries.
         Section headers now carry a " (global)" suffix; projects with no local entries get an explicit "no project-specific knowledge entries found" note.
         Query results already carried a project field, so only the list presentation changed.
       source: user-requested provenance fix, 2026-09-08
       date: "2026-09-08"
     - id: dec-003
-      summary: Query text is escaped so special characters are always literal
+      summary: "Query text is escaped so special characters are always literal"
       detail: |-
         User queries are plain text, not lucene syntax; characters like : ( ) && || " were previously interpreted as operators.
         escapeQueryString escapes syntax-active characters (context-aware for +/- and &&/||); mid-word hyphens like test-stability stay intact.
         Decision documented so future search work does not re-introduce raw QueryStringQuery usage.
       source: user-requested robustness fix, 2026-09-08
       date: "2026-09-08"
     - id: dec-004
-      summary: Background indexing with --no-index-on-startup for large stores
+      summary: "Startup indexing: always built in memory (no skip flag)"
       detail: |-
-        Synchronous indexing on startup blocked server readiness for large knowledge stores.
-        Changed to background goroutine (go idx.IndexAll(resolver)) with ready channel for query synchronization.
-        Added --no-index-on-startup flag to skip indexing entirely; useful when startup delay is unacceptable.
-        Queries block on ready channel until indexing completes; stale data remains available if index has data.
-        readyOnce (sync.Once) prevents double-close panic when NewIndex already closed ready channel.
-      source: GH-25 background indexing, 2026-09-20
-      date: "2026-09-20"
+        The former background-indexing flag --no-index-on-startup is now a deprecated no-op (accepted, warns, ignored).
+        The index is always built in memory at startup; queries block on the ready channel until IndexAll completes.
+        readyOnce (sync.Once) prevents double-close panics.
+        Recorded so a skip-indexing flag is not reintroduced.
+      source: GH-34 memory-only index, 2026-10-07
+      date: "2026-10-07"

.agents/subsystems.yaml

@@ -2,28 +2,28 @@ project: knowledge-mcp
 version: 1
 entries:
     - id: sub-001
-      summary: Bleve search index: lifecycle, rebuild triggers, org section docs
+      summary: "Search index: per-process in-memory bleve, scoped keys, org sections"
       detail: |-
-        Located at <config>-derived path: single-flag configs use <root>/.agents/.index; multi-entry configs use $XDG_STATE_HOME/knowledge-mcp/<sha8>.index; --index overrides.
-        metaFileVersion (search/index.go, currently 3) + recorded project-name set are written to <index>.meta.json; changed version or names force a wipe+rebuild from yaml source of truth.
-        Corruption auto-recovery: bleve.Open failure deletes index dir and rebuilds (search/index.go NewIndex).
-        Documents: summary, detail, rule, category, project fields; scoped keys <project>/<entryID>. Query results de-scope the ID.
-        Org-level docs (.agents/knowledge/*.md) are split on "## " headings into sections with keys org-<file>::<heading>; content before first heading becomes Overview (main.go splitSections).
-        User query text is escaped (search/index.go escapeQueryString) so special characters are literal, never lucene syntax.
-      source: code review, 2026-09-08
-      date: "2026-09-08"
+        The index is built in memory (bleve.NewMemOnly) at server startup from the store YAML/MD files — every restart re-indexes.
+        There is no disk state, no index lock, and any number of instances may query concurrently.
+        Documents carry summary, detail, rule, category, project with scoped keys <project>/<entryID>.
+        Query results de-scope the ID.
+        Org-level docs (.agents/knowledge/*.md) are split on "## " headings with keys org-<file>::<heading>.
+        User query text is escaped (escapeQueryString) so special characters are literal, never lucene syntax.
+        --index and --no-index-on-startup are accepted warning-logged no-ops.
+      source: GH-34 memory-only index, 2026-10-07
+      date: "2026-10-07"
     - id: sub-002
-      summary: Project resolver: kinds, addressing, dynamic fallback
+      summary: "Project resolver: kinds, addressing, dynamic fallback"
       detail: |-
         projects/resolver.go builds refs: KindProject (has .agents/ yaml), KindOrg (root with .agents/knowledge/), KindGlobal (--global path, addressed _global).
         Roots scan immediate children only, skipping .agents and .git; symlinks followed; unreadable entries warn.
         Addressing: bare name when unique, else <qualifier>/<name>; org roots always win their bare name; duplicate root basenames are a startup error.
         Resolve falls back dynamically for paths created after startup (resolveDynamicBare/Qualified); Covers checks known paths and root prefixes.
-        Index location is keyed by hash of canonical root/project paths (github sha256) so different configs get different indexes.
-      source: code review, 2026-09-08
-      date: "2026-09-08"
+      source: code review, 2026-09-08; GH-34 update, 2026-10-07
+      date: "2026-10-07"
     - id: sub-003
-      summary: Knowledge store: atomic writes, per-file locks, entry lifecycle
+      summary: "Knowledge store: atomic writes, per-file locks, entry lifecycle"
       detail: |-
         knowledge/store.go: Save writes temp file, fsyncs, renames (atomic); LoadOrCreate returns empty KnowledgeFile when missing.
         tools package keeps per-path RWMutexes (fileLocks) around read-modify-write cycles; write/update lock, list takes RLock.
@@ -34,12 +34,12 @@ entries:
       source: code review, 2026-09-08
       date: "2026-09-08"
     - id: sub-004
-      summary: Background indexing with --no-index-on-startup flag
+      summary: "Startup indexing: always-on background kick, no skip flag"
       detail: |-
-        IndexAll runs as a goroutine (go idx.IndexAll(resolver)) when --no-index-on-startup is false (default).
-        When --no-index-on-startup is true, indexing is skipped entirely; server starts with stale or empty index.
-        Queries block on ready channel until indexing completes; ready channel closes when IndexAll finishes or if index has stale data.
-        readyOnce (sync.Once) prevents double-close panic if NewIndex already closed ready channel.
-        Useful for large knowledge stores where startup indexing would delay server readiness.
-      source: GH-25 background indexing, 2026-09-20
-      date: "2026-09-20"
+        IndexAll always runs in a background goroutine at startup (there is no skip flag).
+        Queries block on the ready channel until IndexAll completes.
+        readyOnce (sync.Once) prevents a double-close panic on the ready channel.
+        --no-index-on-startup and --index are deprecated, accepted, warn, and do nothing.
+        The index lives only as long as the process and is rebuilt from the store files on every startup.
+      source: GH-34 memory-only index, 2026-10-07
+      date: "2026-10-07"

README.md

@@ -135,7 +135,6 @@ Layout:
 
 ```
 <store>/
-  .index/                          # bleve index (default; --index wins)
   <project>/.agents/               # per-project store (qualified names nest:
                                    #   <root>/<project> -> <store>/<root>/<project>/.agents)
   <org>/.agents/knowledge/         # org-level markdown
@@ -147,8 +146,10 @@ Layout:
 - `init_knowledge` initializes the central store for resolvable paths and
   errors for others (add them via `--project`/`--root` first).
 - The store directory itself must exist and is never created for you.
-- The store is single-writer: its search index and files do not support
-  concurrent access, so point only one server at a store.
+- The store files are single-writer: their YAML reads/writes do not support
+  concurrent access across server instances, so point only one server at a
+  store for writing. Queries are safe from any instance — the search index
+  is per-process in-memory and never shared.
 
 ### Shared global store
 
@@ -177,31 +178,33 @@ General notes:
 - Each `--root` discovers its immediate children as projects (one level deep — pass deeper directories as additional flags).
 - Duplicate project basenames across roots are addressed as `<root>/<project>` (e.g. `work/api`); `list_projects` shows which names need qualification.
 - `query_knowledge` accepts org root names to search that root's `.agents/knowledge/` files. Org-level documents are indexed **per `##` section**, so a query returns the matching section(s), not the entire file.
-- Multi-entry configurations store the search index under `$XDG_STATE_HOME/knowledge-mcp/` (default `~/.local/state/knowledge-mcp/`). Single-flag configurations keep the index inside their own `.agents/` — unless `--store` is set, in which case the index lives at `<store>/.index`.
-- `--index <path>` overrides the index location in all modes.
+- The search index is built **in memory** per server process at startup and
+  does not survive a restart. There is no index on disk anywhere, no lock,
+  and no contention — any number of instances may query concurrently.
+- `--index <path>` is deprecated and ignored (the search index is in-memory).
+- Old leftover `.agents/.index/` and `<store>/.index/` directories and `$XDG_STATE_HOME/knowledge-mcp/` artifacts from earlier releases are inert and can be deleted.
 
 ### Startup Options
 
-`--no-index-on-startup` skips background indexing on startup. The server starts
-immediately with stale or empty index data; queries block until indexing completes
-if no stale data is available. Useful for large knowledge stores where startup
-indexing would delay server readiness.
+Both `--no-index-on-startup` and `--index` are deprecated no-ops. The server
+always builds its search index from the store files at startup (backgrounded;
+queries wait for it once). They remain accepted so existing configurations
+keep working without changes.
 
 ### Debugging
 
 `knowledge-mcp debug` runs a one-shot startup diagnosis against the same
 flags as server mode and exits non-zero when it finds a problem (unresolved
-project, stale or missing index, blocked queries):
+project, blocked queries):
 
 ```bash
 knowledge-mcp debug --root /path/to/org
 ```
 
 The report has three sections — Resolution (projects/stores resolved),
 Startup trail (phase timings and a watchdog-bounded index self-query that
-flags the cold-index query block), and Environment (version, index
-location/source, doc count, freshness, install record) — followed by
-`ok`/`warn`/`fail` findings.
+flags a slow index kick), and Environment (version, doc count, install
+record) — followed by `ok`/`warn`/`fail` findings.
 
 For live diagnostics, run the server with `--debug` (optionally
 `--log-file <path>` to tee). Lines go to stderr, which MCP clients such as