#333 child: Make the first Docker answer independent of Compose and cold-start work

[Joncallim]

Parent: #333. Baseline from the performance child should exist before final acceptance.

Problem

DockerMap already keeps optional host providers off the Docker publication path, but two remaining choices still make the first useful answer slower than necessary:

1. collect_docker_snapshot_candidate() waits for bounded Compose filesystem projection inside the same five-second Docker snapshot budget after Docker inventory has already completed.
2. daemon startup awaits refresh_cache() before binding the HTTP listener.
3. Docker container, network and volume list reads are independent but currently performed sequentially.

Goal

Publish the earliest truthful Docker model as soon as Docker evidence is available. Treat Compose correlation as later enrichment tied to the exact Docker observation it was computed against.

Architecture direction

Target shape:

listener ready
    ↓
initial state = collecting / no authoritative Docker model
    ↓
Docker inventory (bounded)
    ↓
publish Docker snapshot/runtime/findings that require Docker-only evidence
    ↓
Compose projection worker (bounded + single-flight, keyed to Docker observation revision)
    ↓
if still coherent: publish Compose binding + Compose-dependent findings
if Docker changed: retain/suppress as stale according to an explicit contract

The exact internal types may differ, but Compose filesystem work must no longer be able to delay a completed Docker observation.

Scope

• Bind/start the daemon HTTP server before the first collection completes.
• Define a safe explicit initializing/unavailable publication state; do not serve mock as if initialization were failure when mock fallback is disabled.
• Run independent Docker inventory reads concurrently under one overall observation deadline if the gateway/client semantics permit it.
• Split Compose runtime correlation from the Docker critical path.
• Preserve the existing Compose projection single-flight guarantee and filesystem bounds/symlink rules.
• Bind Compose results to a stable Docker observation identity, not to wall-clock timing.
• Recompute/publish Compose-dependent Findings only when the enrichment matches current Docker evidence.

Acceptance criteria

• Listener availability does not depend on the initial five-second observation budget.
• A slow Compose filesystem cannot delay first authoritative Docker publication.
• A timed-out Compose projection leaves Docker data usable and explicitly lacks current Compose correlation.
• A Compose result that completes after Docker changes cannot be attached to the newer observation.
• No overlapping Compose filesystem tasks after timeout/unwind.
• Container/network/volume Docker reads are concurrent, or the PR records controlled evidence showing why they must remain sequential.
• Docker gateway authority/route allowlist is unchanged.
• Source transitions (Docker ↔ unavailable/mock where permitted) remain fail closed.
• Epic: Build deterministic Findings and drift analysis from provenance-backed evidence #69 Docker-only rules keep exactly their truth semantics.
• Compose mount-drift rule appears only after coherent Compose binding exists.
• Controlled benchmark demonstrates improvement or at minimum eliminates the Compose/cold-start coupling without regression elsewhere.
• Live-Docker and production-image tests pass.

Non-goals

• No new Compose semantics.
• No project-identity UI work.
• No persistent cache across daemon restart.
• No broad filesystem watcher.
• No change to provider scheduling cadence.

Risk: MEDIUM · read_only_product_behavior: true · security_sensitive: true (publication/source coherence) · data_migration: false · routing_mode: stable

[Joncallim]

Baseline 4 is now the comparison authority for this issue

#335 is merged (aa023a7) and Baseline 4 is the baseline that #336/#337/#338 must be
judged against. Do not assume a win; measure against these numbers.

• methodology dockermap-v1/time-to-answer-methodology-8; product + capture-harness revision
  bdce6ae354d757d3318c514e10d620edb497918e
• composite: 44 records (38 end-to-end + 6 controlled-poll-phase), 3 runs × 15 measured
  samples each, 1 980 raw samples; conditioning is fixed 60 burn-in then 15 measured with
  burn-in excluded from every summary
• promotion rule: max(baseline × 1.25, baseline + 2 ms), compared only in a compatible
  pinned environment; provenance fields (sourceRevision, harnessRevision,
  daemonBinarySha256) and the informational dockerRevision are recorded but are not
  compatibility requirements
• published record: docs/testing/TIME_TO_ANSWER_BASELINE.md
  (interpretation contract: docs/testing/TIME_TO_ANSWER_EVIDENCE.md)
• evidence root: /srv/jonas/evidence/dockermap/time-to-answer/bdce6ae/;
  reproduce the published summary with
  npm run perf:summarize -- --artifact /srv/jonas/evidence/dockermap/time-to-answer/bdce6ae/time-to-answer-baseline-4-composite.json

Two facts worth carrying into this work:

1. The dominant term is the Stage-5 poll-wait floor. publicationToNodeObservationMs
   is a phase-normalized figure (median of three run p95s) of 1 897.79 / 1 897.61 / 1 897.67 ms
   at 25 / 100 / 250 containers — 89.5 % / 85.1 % / 66.4 % of the summed reference-fixture
   stage medians. It is the fixed 2000 ms polling mechanism's phase response, explicitly not
   network latency, and the production cadence is deliberately unchanged by #333 child: Establish end-to-end time-to-answer baselines and regression gates #335.
2. Compose is small on these fixtures but is measured separately: composeEnrichmentMs
   1.02 / 1.05 / 0.97 ms at 25 / 100 / 250, with 9.55 ms for the declared slow-but-bounded
   Compose scenario. Nothing was decoupled by the baseline — a container-level change should
   be justified against these numbers, not an assumed large win.

The supporting Stage-6/7 seam-isolation control is FAIL at this checkpoint
(publication-level causal identity unavailable); it is disclosure evidence only and does not
gate these timings.

[Joncallim]

Architecture hand-off — 29 September 2026

Disposition: continue/reconcile the existing implementation; do not start a competing rewrite. Remote main inspected at cd70425fe777b02077b32b65d0ccf8041c1838b8; draft #347 is at 09c56153b9e61a04fa579f05e4c87f23f383ccc3. The source-level findings below include inspection of its cache_refresh.rs patch. The unpublished #337 stack depends on this work; reconcile its current owner/HEAD before editing. This comment does not unpark #347 or authorize capture, push or merge.

1. Important review finding: retained Compose evidence needs a validity lifecycle

In the inspected patch, an unchanged Docker observation and matching private Docker-side inputs can retain the previous compose_runtime_binding. The detached worker returns early when projection yields None or times out. Consequently, unchanged Docker input does not by itself prove that retained Compose file evidence is still current. A previously readable configuration can change or become unreadable while the container stays unchanged.

A second problem with using compose_mount_finding_projection equality as the entire retention decision is that two different scans can produce identical or empty Findings. That is not proof of identical configuration, current coverage or renewed validation. Successful revalidation must update the private evidence/coverage record even when public Finding semantics do not change.

Treat these as mandatory revalidation cases on the actual working branch; do not assume the remote patch is still its latest implementation.

2. Publication state machine

Keep transport readiness distinct from model readiness:

• Initializing: listener responds, but cache-backed authority routes return the closed initializing/unavailable response. Internal bootstrap mock is never presented as a live answer.
• DockerReady: a complete validated Docker collection attempt publishes snapshot/runtime/Docker-only Findings atomically.
• Unavailable: failed authority is explicit; retained data, where supported, remains old data, not a new healthy publication.
• MockReady: only under the existing explicit mock policy, with sample provenance and a new source generation.

Audit every cache-backed route, not just /snapshot: health/status, runtime, Findings, inventory and any revision endpoint must not expose bootstrap data as live. A valid empty Docker inventory counts as authoritative; a listening socket or successful health transport alone does not.

Independent container/network/volume requests run concurrently within one observation deadline. No partial list success can become a complete snapshot. A failed/timed-out attempt invalidates the affected gateway client according to existing recovery policy. Concurrency does not imply Docker supplied a transactional point-in-time inventory; preserve the current bounded observation claim.

3. Compose worker contract

Use a private work identity containing source generation, Docker observation identity and an attempt/ownership token. Keep exact Compose binding inputs private. Derive project identity, diagnostics and declarations from the same bounded captured file bytes using the existing binding scanner; retain its fingerprint only privately.

The state is explicitly not-yet-validated / collecting / current / stale-or-unavailable, with closed failure reasons and the original observation time. Do not add arbitrary provider error strings to the public contract. Coordinate the minimal public coverage projection with #337/#340 rather than inventing three parallel states.

Lifecycle requirements:

1. Publish Docker first. Queue at most one Compose computation and at most one latest pending observation; never an unbounded work backlog.
2. The single-flight permit lives inside the blocking closure. An async timeout does not make an already-running filesystem task stop. Do not release the permit until that task actually exits; no second blocking projection may overlap it.
3. Compute outside the publication lock. At commit, recheck source generation, Docker/private binding identity and job ownership. An obsolete success is discarded. An obsolete failure cannot invalidate a newer accepted result.
4. Failed revalidation removes current eligibility for Compose-dependent Findings, or transitions retained evidence to explicitly stale under a reviewed bounded validity policy. It cannot leave an indefinitely fresh binding. Keep Docker-only facts usable.
5. Successful equivalent scans refresh private validation metadata without manufacturing a semantic model change solely from timestamps. Changed configuration/eligibility/coverage advances the relevant semantic publication even when the Finding list remains empty.
6. Retention while a new validation is pending must have explicit semantics. Prefer marking Compose as collecting/not currently revalidated rather than silently renewing freshness. Any grace period must be bounded, clock-monotonic and documented as last-validated evidence, not proof that files have not changed.
7. No filesystem read, parse or network await while holding the publication write lock. Prepare bounded candidate material, then atomically commit after checking the captured input version; if another publication changed a dependency, recompute/retry within bounds rather than overwrite it.

This issue owns the enrichment lifecycle; #337 owns how its semantic state participates in facet attestation. Neither may weaken the other's acceptance boundary.

4. Adversarial pass and required tests

Use barriers/channels or injected deterministic clocks, not sleeps, for concurrency assertions.

• Block all initial Docker reads: a real HTTP probe must still reach the listener; no cache-backed route publishes bootstrap live/mock data incorrectly.
• Barrier-prove all three Docker reads start before any is released; one failure/timeout prevents a partial authoritative result.
• Block the actual detached Compose path, not only the old collector injection seam: first Docker publication must complete, one permit remains occupied after timeout, and repeated refreshes start no second filesystem task.
• Change Docker/source while Compose runs, including Docker→mock/unavailable→Docker with superficially equal content: the old completion cannot attach.
• Keep Docker unchanged, then edit/delete/deny access to Compose files: prior Compose Findings cannot stay fresh indefinitely.
• Revalidate successfully with identical Findings but changed private configuration/coverage: validity is updated correctly and unrelated geometry need not rerun.
• A late failed attempt cannot clear newer valid enrichment; equal Docker human-readable uptime/order-only changes must not create false material changes.
• Changing files during bounded capture, symlink substitution, oversized files, caps and malformed input fail closed without widening roots or authority.
• Docker-only rule outputs remain semantically identical; mount-drift appears only with eligible exact binding. UI displays unavailable Compose coverage without calling the workload failed.

5. Implementation sequence, rollback and acceptance

First add regression tests for the retained-binding failure and the real detached-worker lifecycle. Then implement the smallest state/coverage repair in cache_refresh.rs, with only necessary daemon route/contracts/API validation changes. Preserve fixed gateway permissions, filesystem bounds and provider cadence. Keep #337's dependent stack and single-editor ownership intact.

Run focused Rust/core/daemon and contract tests, the full npm run check, live Docker, browser and production-image/deployment gates at the exact final SHA. Obtain a fresh independent implementation/security review; this architecture pass is not that execution evidence.

#347's recorded Baseline-4 failures remain failures. Parent #333 must govern any new paired process→first-model metric or changed stage methodology. Never substitute sums of independent percentile summaries, waive the browser/query failures, or use an older green build as exact-head evidence.

Rollback restores a coherent prior implementation and preserves source gating; it must not re-enable raw Docker access, expose bootstrap mock, or suppress the comparison failure. Keep #336 open until the repaired architecture and controlled evidence are proven.

Sources: existing PR, private exact binding, bounded captured-byte scanner, baseline authority. No runtime tests were executed in this architecture review.

[Joncallim]

Architecture amendment — second optimisation pass, 29 September 2026

Amends the first hand-off. Continue the existing owner/stack; no replacement implementation, unpark, capture, push or merge is authorized here. Rechecked the remote #347 source at 09c56153b9e61a04fa579f05e4c87f23f383ccc3; the actual unpublished dependent work still needs owner-side reconciliation.

Avoid fixing stale evidence by creating permanent refresh churn

Separate worker activity from evidence eligibility. Starting another validation must not itself remove an otherwise eligible binding, emit a new Findings set, reset preferences and then restore everything on success. This replaces the first hand-off's default suggestion to publish collecting/not-current on every routine revalidation.

Retain a last-validated binding only while all of these hold: the source and exact private Docker-side dependencies still match; a documented finite validity deadline has not expired; and no relevant failure/change has invalidated it. Display its actual last-validated meaning, not a claim that the files have remained unchanged. Successful validation renews validity from actual completion; launching or timing out a job does not.

The implementation must select and test the finite deadline explicitly against the existing refresh cadence and attempt budget. Do not introduce a configurable general cache-TTL subsystem. If no bounded policy is established, the conservative collecting/unavailable fallback remains required rather than indefinite retention.

A relevant input change, failed validation, timeout or deadline expiry withdraws current eligibility atomically, even when the previous/new Finding lists are both empty. Deadline expiry must be driven by daemon-owned scheduling independently of the blocking worker's eventual return. A stuck filesystem task cannot leave evidence fresh forever. Successful equivalent validation updates private validation metadata without forcing a semantic publication solely for elapsed timestamps.

Prevent equivalent refreshes from starving the worker

Keep at most one filesystem task and at most one latest pending input. Coalesce equivalent input instead of assigning it a new ownership token that makes the running job obsolete. The attempt token belongs to the actual job; the dependency identity describes its inputs. Neither is the broad model revision, which unrelated provider publications can change.

On completion, accept only matching source, exact private dependencies and valid job ownership. A late obsolete failure cannot clear a newer binding. A genuinely changed Docker input makes the old completion unusable; it never gains compatibility because its resulting Findings happen to be equal.

The single-flight permit still lives inside the blocking closure until it exits. Perform file IO and candidate preparation outside the publication lock. Do not add a watcher, an unbounded task queue or overlapping replacement workers to improve a latency number.

Additional deterministic cases

With controlled barriers/clock advancement, prove: repeated equivalent refreshes do not starve one compatible validation; many equivalent successful validations create no alternating empty/populated Review publications; an expired binding becomes ineligible while a timed-out worker is still blocked; changed private inputs with identical public Findings still invalidate correctly; and an unrelated provider completion does not discard compatible Compose work. Count publications and filesystem tasks, not just final output.

Preserve first-Docker publication, route readiness, gateway restrictions and all existing #69 semantics. Keep #347's failed promotion and the exact-head gate requirements unchanged.

Source: actual detached worker, retention and publication code. These are design repairs and required tests, not executed test results.