Skip to content

docs: the Phase 4 handoff #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
craig-dt merged 1 commit into from
Sep 1, 2026
Merged

docs: the Phase 4 handoff #187

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions docs/RESUME-phase-4.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Resuming after Phase 4 on `fl-replay`

Written 2026-09-01, on `docs/RESUME-ls-9.md`'s precedent. **Authoritative state is
`docs/status.yaml` — its `next_action` leads with this and its `log:` carries the reasoning.** This
file is the operational half.

## The situation in one paragraph

**Phase 4 is one step from complete and the store now does what it was built for.** Bare `blfile`
emits **880 flows across 24 captures**, every one with a recorded `gs://` origin — it emitted **zero**
on 2026-08-27. Tier 2 is selectable for all 24 captures, and the cross-tier merge rule runs on real
rows. Only **P4-6** remains, and it is deliberately off the critical path.

## Where things stand

| | |
| :-- | :-- |
| `main` | `5d24c3d` — #184 merged |
| Phase 4 | P4-0…P4-5 **done**; **P4-6** outstanding (one `--both` run on the smallest capture) |
| store | 50 runs · 50 sightings over **24 distinct captures** · 2,362 flow_labels · 35 unmatched |
| `blfile` | 880 flows, 24 captures, all `uri_status: gs`; rebuilds at exit 0 |
| box suite | **1 failure** — the decided Zeek 8.2.1-vs-8.0.9 divergence. It was 9 on 2026-08-27 |
| toolchain | Suricata **8.0.6**, Wireshark **4.6.6**, ja4 **v0.18.8** — all pinned *and* `apt-mark hold`ed |
| corpus | described in `docs/corpus-2026-08-28.md` — **read that before using it as ground truth** |

## Read this before touching the corpus

`docs/corpus-2026-08-28.md` exists because three reviews said the numbers alone would let a consumer
over-trust it. The four things that matter most:

1. **Split by capture, never by flow.** One capture is **41%** of the corpus; the top three are 58%.
A random flow-level split measures memorisation.
2. **52% of the flows are port 22.** Top four ports: 69%. No flow carries TLS enrichment. This is not
a general malicious-traffic corpus.
3. **177 of 880 flows (20%) carry only `indicator-reference` sources** — the flow to your own
resolver or proxy, not the malicious activity.
4. **Licence obligations ride on the labels**: 462 entries CC-BY-SA-4.0 (share-alike), 408
`proprietary:vendor-signature (not redistributed)`. `blfile` writes **no `NOTICE`**.

## What the review process cost, and it is the most transferable thing here

`CLAUDE.md` already says re-cut and re-review after acting on findings. Phase 4 is the evidence for
why, and it went further than LS-9's three rounds: **Craig asked "has this had a fresh-eyes review?"
three times, and all three times the honest answer was no.** Each time the gap had the same shape —
a review ran, produced substantial new content, and *that content* shipped unreviewed.

The sharper lesson is what rounds 2 and 3 found. **They repeatedly found the code sound and my
justifications false.** In one change: a fabricated measurement pasted into a block captioned "one
real X"; "`build` already refuses elsewhere" (it does not); "`coverage` gains fields as loss
conditions are added" (it does not); "a digest so the next added field cannot repeat this" (it
covered a strict subset). Each was a confident sentence that one command contradicted.

**So: when you write a justifying sentence, run the thing that would falsify it.** That is cheaper
than a review round and it is what the reviews kept doing on my behalf.

Two more, both cheap to internalise:

- **A sabotage that does not do what you think is worse than none.** Two of mine went wrong mid-round
— one edited `SourceSpec` instead of `SourceEntry`, one produced a pytest *collection error* rather
than a failure. Both returned green, and green meant nothing.
- **Verify the reviewer.** One invented four loss-condition field names that do not exist; another's
SQL measured a different set from the one it claimed. Both were still worth their round.

## Things that will bite if you do not know them

- **`--offline` from `gs://` is the route, and `--both` is not.** A tier-2-only run records the
origin, attests tier 2, and **supersedes nothing** — the arithmetic held on production: 400 tier-1
only + 8 both = 408, exactly the pre-phase baseline. `--both` would have replaced all 408 with
re-observed device results *and* still not exercised cross-run merge, because one `--both` run
correlates both tiers into a single row.
- **`tcpprep` is tier-1 only.** `capture_2026-07-21` failed the replay path twice and labelled
cleanly offline. A capture that is bad for replay is not a bad capture.
- **`flabel-run` refuses to start without `/var/lib/flabel/.provisioned`** (exit 2). The provisioning
script writes it last, after every tool-pin assertion.
- **`flabel-deploy`'s busy guard fires on any process whose command line mentions flabel** (#181),
including your own shell. It will refuse on an idle box.
- **Documents written before #184 cannot be rebuilt** — `schema_version` moved to `1.1`. They are
still readable; only reproduction is refused, and the message says so.

## Do this first

1. **P4-6** — one `--both` run on the smallest capture, off the critical path. Then Phase 4 is done.
2. **#183** — the scheduled false-positive review failed on 2026-08-30 and **nobody has looked**.
That job is Goal 5's real FP control, and `ci.yml` refuses a push if it goes dark seven days.
3. **PRD Q11, unrecorded**: are these 24 captures publicly-published, or internal/customer traffic?
The PRD restricts flabel to the former and calls the latter a gate. The licence findings make this
sharper, and nobody has written the answer down.

## Still open and unowned

- **#162** — real project id and SA email in plaintext on this public repo. A standing `CLAUDE.md`
guardrail violation, and a decision (history rewrite vs accepted exposure) rather than a patch.
- **#185** an authoritative run with a NULL `run_block` is dropped from the pin (latent: 0 of 50).
**#179** `flabel-run` reuses a staged capture without checking its digest while still asserting the
`gs://` origin (latent: triangle green). **#181** the busy guard. **#161**, **#164**, **#159**.
- **#103** — Goal 5's sensitivity half is still unmet; the malicious canary is unsourced.
- **#153** stays open on `collection_id` alone, deferred.
- **The Zeek pin.** `Dockerfile.toolchain` still says `8.0.9-0`, which matches neither the box nor any
row in the store and cannot be rebuilt from apt. The GHCR digest is the real pin. Bumping to 8.2.x
is the rebuildable fix — Craig's call, recorded 2026-08-27.
9 changes: 7 additions & 2 deletions docs/status.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ prd_gdoc_url: "https://docs.google.com/document/d/1WwKukQ71L6JQWkSpKvoR4lgwB96w3
# place, so it means creating a new Doc and updating this URL). Also: the superseded v0.1 Doc
# 1q_muS6AmCuvcNva4C1WtRE3_KXArJPxnapttsU6UQIE needs deleting by hand — no delete tool exists.
current_stage: scaffold # PHASE 3 only. Phases 1 and 2 are complete; see stages.label_store
next_action: "PHASE 3 IS COMPLETE AND MERGED (LS-9 = #177, main at a56939d). There is no LS-10, and #153 stays open on collection_id alone (deferred, Craig). READ docs/RESUME-ls-9.md for the operational handoff and the review-process change that came out of this phase — the standing rule is now RE-CUT THE DIFF AND RE-REVIEW AFTER ACTING ON FINDINGS, because LS-9 took three rounds and rounds 2 and 3 each found defects in the previous round's fixes. ZEEK IS DECIDED: the box stays on 8.2.1 (Craig, 2026-08-27); what is left unowned is that Dockerfile.toolchain still pins 8.0.9-0, which matches neither the box nor any row in the store and cannot be rebuilt from apt, so the GHCR digest IS the pin now. THE NEXT TASK IS SCOPING, NOT BUILDING, and be honest about it: THERE IS NO PHASE 4 DEFINED ANYWHERE IN THIS REPO. The delivery phases are 1 (tier 2, signed off 2026-08-14), 2 (tier 1, #122/#128) and 3 (the label store); PLAN.md is Phase 1 and closed, PLAN-label-store.md is Phase 3, and Phase 2 was built on a branch with no plan document at all (Craig, 2026-08-17). The term Phase 4 appears ONLY in docs/RESUME-ls-9.md, where the model coined it — do not treat it as an existing scope to look up. WHAT THE EVIDENCE SAYS THE NEXT PHASE SHOULD BE: the store is correct and the corpus in it is not yet fit for the headline requirement. Bare blfile emits ZERO flows, refusing all 408 selectable ones for want of a recorded origin, because every one of the 25 captures predates --source-uri; no tier-2 knowledge is selectable at all (#142); and cross-tier composition therefore still has only fixtures behind it (#144). A re-run of the corpus through the LS-5 wrapper would address all three at once and is the strongest candidate. THE COMPETING CANDIDATE IS #162 — the real project id and SA email in plaintext on this public repo — which is the only open item that is a STANDING GUARDRAIL VIOLATION rather than work not yet done, and whose fix is a decision (history rewrite versus accepted exposure) rather than a patch."
next_action: "READ docs/RESUME-phase-4.md FIRST, and docs/corpus-2026-08-28.md BEFORE TOUCHING THE CORPUS. PHASE 4 IS ONE STEP FROM COMPLETE and the store now does what it was built for: bare blfile emits 880 FLOWS ACROSS 24 CAPTURES, every one with a recorded gs:// origin, where it emitted ZERO on 2026-08-27. Tier 2 is selectable for all 24 and the cross-tier merge rule runs on real rows (#144 CLOSED on 8 flows composed from two different runs). main is 5d24c3d with #184 merged. Store: 50 runs, 50 sightings over 24 DISTINCT captures, 2362 flow_labels. Box suite is at ONE failure, the decided zeek 8.2.1-vs-8.0.9 divergence — it was 9 on 2026-08-27. Toolchain aligned: suricata 8.0.6, wireshark 4.6.6, ja4 v0.18.8, all pinned AND apt-mark held. DO FIRST: (1) P4-6, one --both run on the smallest capture, off the critical path, and Phase 4 is done; (2) #183, the scheduled false-positive review FAILED on 2026-08-30 and NOBODY HAS LOOKED — that job is Goal 5 real FP control and ci.yml refuses a push if it goes dark seven days; (3) PRD Q11, UNRECORDED: are these 24 captures publicly-published or internal/customer traffic? The PRD restricts flabel to the former and calls the latter a gate, and the licence findings make it sharper. THE PROCESS LESSON PHASE 4 PAID FOR, and it goes further than LS-9 three rounds: Craig asked `has this had a fresh-eyes review?` THREE TIMES and all three times the answer was NO — each time a review had run, produced substantial new content, and THAT content shipped unreviewed. And rounds 2 and 3 repeatedly found THE CODE SOUND AND THE JUSTIFICATIONS FALSE. When you write a justifying sentence, RUN THE THING THAT WOULD FALSIFY IT. STILL UNOWNED: #162 (plaintext project id on a public repo, a standing guardrail violation), #185, #179, #181, #161, #164, #159, #103, #153, and the Zeek pin decision."
blocked_on:
- "NOT BLOCKING SIGN-OFF ANY MORE — DECIDED (Craig, 2026-08-13): the malicious canary gap is ACCEPTED for Phase 1. The PRD success criterion ships explicitly not met, #24 stays open, and the skipif stays as the visible marker rather than being removed. Residual risk, to be named in the verify report: the SCHEDULED Goal 5 gate (feeds.yml) has only a negative control, so nothing in the scheduled path proves the live 85k ruleset can still produce a label at all. The PR suite does cover that — test_canaries.py labels end to end against a small real snapshot every PR — so a total labelling outage is caught, just not by the gate whose job it is."
- "PRD Q1/Q2/Q10: stakeholders, target release, success-metric review dates. Left TBD deliberately, not invented."
Expand Down Expand Up @@ -101,7 +101,7 @@ stages:
prd: { status: skipped, reason: "no new product surface — this phase feeds the existing one; the PRD goals it serves are 1, 2 and 5" }
eng_review: { status: completed, completed: "2026-08-27", reason: "fresh eng-reviewer on revision 1: 2 BLOCKING, 6 HIGH, 5 MEDIUM, 5 LOW. Findings VERIFIED before acting — one BLOCKING was refuted by one grep. Revision 2 changes the ROUTE from --both to --offline." }
plan: { status: completed, artifact: "docs/PLAN-phase-4.md (revision 2)", completed: "2026-08-27", steps_total: 7, steps_done: 6, detail: "P4-0 through P4-5 done; only P4-6 (one --both run, off the critical path) remains. Revision 2 rewrote the route after review; see build.detail." }
build: { status: in_progress, steps_total: 7, steps_done: 6, detail: "PHASE 4 HAS MET ITS PURPOSE. P4-0 to P4-5 DONE; only P4-6 remains (one --both run, deliberately off the critical path). THE THREE OPENING SYMPTOMS ARE ALL FLIPPED, measured: bare blfile emits 880 flows across 24 captures with NO refusal line, where this morning it emitted ZERO and refused 408; all 880 records carry uri_status gs; tier 2 is selectable for all 24 captures; and 17 captures compose their two tiers from TWO DIFFERENT RUNS, which closes #144 on real rows. Store: 50 runs, 50 SIGHTINGS OVER 24 DISTINCT CAPTURES, 2362 flow_labels. Reconcile clean on all 50. blfile --rebuild of a post-corpus-change document exits 0, so reproduction survived. Corpus snapshot 682c24ae30631504, suricata 8.0.6, editcap 4.6.6, ja4 v0.18.8, config digest 4ecf7728. See docs/corpus-2026-08-28.md before treating the corpus as ground truth." }
build: { status: in_progress, steps_total: 7, steps_done: 6, only_remaining: "P4-6, one --both run on the smallest capture, deliberately off the critical path", detail: "PHASE 4 HAS MET ITS PURPOSE. P4-0 to P4-5 DONE; only P4-6 remains (one --both run, deliberately off the critical path). THE THREE OPENING SYMPTOMS ARE ALL FLIPPED, measured: bare blfile emits 880 flows across 24 captures with NO refusal line, where this morning it emitted ZERO and refused 408; all 880 records carry uri_status gs; tier 2 is selectable for all 24 captures; and 17 captures compose their two tiers from TWO DIFFERENT RUNS, which closes #144 on real rows. Store: 50 runs, 50 SIGHTINGS OVER 24 DISTINCT CAPTURES, 2362 flow_labels. Reconcile clean on all 50. blfile --rebuild of a post-corpus-change document exits 0, so reproduction survived. Corpus snapshot 682c24ae30631504, suricata 8.0.6, editcap 4.6.6, ja4 v0.18.8, config digest 4ecf7728. See docs/corpus-2026-08-28.md before treating the corpus as ground truth." }
label_store:
research: { status: skipped, reason: "the 2026-08-20 design session did the work; artifact is inline-labeling/label-store-design.html" }
prd: { status: skipped, reason: "inline-labeling/database.md is Craig's requirements doc and was treated as the PRD" }
Expand Down Expand Up @@ -440,3 +440,8 @@ log:
- "2026-08-28 ROUND 3 ALSO CAUGHT A FALSE JUSTIFICATION IN MY OWN COMMENT. _record raise said `build already refuses rather than writing past that assumption elsewhere`. It does NOT — build writes runs: [] whenever nothing is selected, and read_prior then refuses that document for non_empty. There is an existing test recording exactly that as known and accepted, on THE MOST LIKELY DOCUMENT AN OPERATOR HAS. Sentence corrected to say what is true and to name the disagreement rather than deny it"
- "2026-08-28 AND THE #171-SHAPED TEST WAS ITSELF #171-SHAPED. test_the_declaration_validates_the_shape_production_actually_wrote passed sightings=[], which exercises _record no-sighting fallback — a shape production NEVER WROTE. The archive real shape is a not-recorded SIGHTING: uri null but filename, link_type and snaplens POPULATED. So the one shape the archive took stayed unvalidated inside the test written to validate it. Fixed to sighting(recorded=False)"
- "2026-08-28 REPRODUCTION RE-MEASURED AND LOGGED rather than asserted, after round 3 flagged `verified on the 880-flow corpus` as a not-measured flipped to verified: `blfile: REPRODUCED v11.json — 880 flow(s), identical over records with built_at excluded (§6.5)`, exit 0, 2026-08-28. Also M2/M3: unmatched declared (int,float) like its sibling because _total sums values out of an unvalidated verbatim STRING column, and tiers_supplying gains non_empty because its own docstring argues [] is a falsehood rather than an absence"
- "2026-09-01 #184 MERGED (#186) AFTER THREE REVIEW ROUNDS, and the rounds are the story. Round 1: adding a field to every record CHANGED THE REPRODUCTION SURFACE — SCHEMA_VERSION was still 1.0, so --rebuild of any prior document compared every record and exited 1 saying `the rows those runs hold have changed`; measured against the P4-5 baseline, 409 difference lines on the new key alone. Round 2: I had FABRICATED A MEASUREMENT, and closing the origin shape had left coverage open so the hole moved down a level. Round 3: the digest pin I added to prevent recurrence covered a STRICT SUBSET of what differences compares, and my first fix for that did not work either"
- "2026-09-01 THE PATTERN ACROSS ALL THREE ROUNDS, and it is the thing to carry forward: THEY FOUND THE CODE SOUND AND MY JUSTIFICATIONS FALSE. `build already refuses elsewhere` — it does not. `coverage gains fields as loss conditions are added` — it does not, _coverage returns a literal five-key dict. `a digest so the next added field cannot repeat #184` — it covered declared paths while differences compares every key of every record. Each was a confident sentence that ONE COMMAND contradicted. Write the sentence, then run the thing that would falsify it"
- "2026-09-01 AND CRAIG ASKED `HAS THIS HAD A FRESH-EYES ENG REVIEW?` THREE TIMES THIS PHASE — #177, #180, #182 — and all three times the honest answer was NO. Same shape each time: a review ran, produced substantial new content, and that content shipped unreviewed. CLAUDE.md already says re-cut and re-review after acting; Phase 4 is the evidence for why, and the gap is not the first review, it is everything the first review causes you to write"
- "2026-09-01 #144 CLOSED on real rows: 8 flows carry sources from BOTH tiers, composed from two DIFFERENT runs — tier 1 from an August replay, tier 2 from a 2026-08-28 offline run — which is merge.py rule 2 executing outside a fixture for the first time. Guarded against the obvious false positive: 0 cases of one 5-tuple producing two different flow_keys, so the 400 disjoint flows are genuine firewall-vs-Suricata disagreement rather than a broken join across two toolchains. Noted on closing that 8 of 880 is thin, and that a richer overlap wants its own issue rather than this one held open"
- "2026-09-01 HANDOFF WRITTEN: docs/RESUME-phase-4.md. Note the two docs a new session must read in order — the RESUME for state, and docs/corpus-2026-08-28.md BEFORE the corpus is used as ground truth, because the numbers alone invite over-trust: one capture is 41% of it, 52% of the flows are port 22, 20% carry only indicator-reference sources, and 462 source entries are CC-BY-SA-4.0 share-alike while blfile writes no NOTICE"
Loading