Survive transient fetch failures during file indexing and SystemCard load

[jurgenwerk]

There was a situation in staging where the AI assistant showed 0 skills and single model:

The reason this happened:

1. The indexer re-indexed a file — skills/index.md, the default room skill. To do that it downloads the file's bytes, and that one download failed with net::ERR_HTTP2_PROTOCOL_ERROR — the HTTP/2 connection between the prerender page and the realm server was reset mid-request — so the markdown parser could not run.
2. Instead of trying again, the indexer fell back to indexing the file as a generic file, and saved that as a normal success — so nothing ever retried it. The file permanently lost its markdown fields, including kind: skill.
3. The SystemCard — the card that lists the available LLM models and the default skills — links to that file through a field typed "markdown file". A generic file does not satisfy that type, so loading the SystemCard threw and the whole card became unloadable.
4. The host caches that failure for the entire session: the model picker drops to the built-in fallback list, and new chat rooms are created with an empty skill list. The empty list is saved into the room's state, so the room stays at 0 skills even after the index is repaired.

Net effect: one reset connection during indexing broke the assistant until someone manually reindexed.

Reindexing solved it for staging — the fresh pass rewrote the bad row, and models and skills came back. (Rooms created during the broken window keep their empty skill list, since room state is persisted; they need the skill re-added or a new session.)

Steps taken to not have this happen again, one commit each:

1. A failed download now produces an error row instead of a wrong success row. Error rows are retried on the next indexing pass, so the index heals itself. The generic-file fallback stays for its real purpose: content the specific parser cannot handle.
2. The download retries twice with a short pause before giving up, so most dropped connections never surface at all.
3. A linked card or file that does not satisfy the field's type is dropped from that field (with a console warning) instead of making the whole linking card unloadable. Direct assignment still validates strictly.
4. The host retries a failed SystemCard load on a backoff schedule, and applies the default skills when it opens an empty room that never had any — so neither failure is permanent anymore.
5. A Grafana alert fires on the log line that signals the generic-file fallback, which was previously the only — and unwatched — sign that a file row silently degraded.

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

Host Test Results

    1 files      1 suites   1h 44m 51s ⏱️
4 686 tests 4 672 ✅ 14 💤 0 ❌
4 701 runs  4 687 ✅ 14 💤 0 ❌

Results for commit ce414c5.

Realm Server Test Results

    1 files      1 suites   20m 2s ⏱️
2 403 tests 2 403 ✅ 0 💤 0 ❌
2 486 runs  2 486 ✅ 0 💤 0 ❌

Results for commit ce414c5.