scanner_ranges: 厂商自报段接入 + eval 真值层排除 + 自报段标签呈现

CHANGELOG.md

@@ -6,6 +6,8 @@
 
 ### 新增 Added
 
+  - **Scanner self-declared ranges: source + eval truth-layer exclusion + label pinning (issues #212 #213 #214, docs closeout #215; scanner_ranges batch; census docs/research/2026-10-26-scanner-ranges-census.md, ADR 0011)**: new `backend/ipdb/_sources/scanner_ranges.py` (cdn_edges-shaped _FEEDS multi-publisher table) ingests 66 MISP warninglists non-nt scanning lists (CC0, CIRCL mirror, ~29,070 CIDR v4+v6 mixed, date-versioned daily) + Palo Alto Xpanse reference-page direct read (9 CIDR — the only major absent from MISP); OpenFilters/internet-scanners (same upstream) deliberately NOT wired — one route per truth, echo discipline; GreyNoise API-only stays out (red line); evidence face is the user-ruled accusation form: Evidence(classification="scanner", verdict="malicious", native_categories=["研究扫描 · <厂商/机构>(自报段)"], r=0.90, stale_days=7, provenance_tier B) — malicious is the behavioral-type assertion (词表 "aggressive scanning"), intent lives in the native label, adjudication stays with the user; independent two-vote corroboration with behavior witnesses (sw_scanners) is legal (observation line independent, no lineage overlap). Eval truth layer (ADR 0011, deliberate product/eval asymmetry = feature): assembly-side exclusion — `layer_builders.scanner_ranges_prefixes` reads prefixes from the raw middleware CSV (backend/data/scanner_ranges.csv, first column, never reverse-parsed from LMDB; missing CSV raises FileNotFoundError — the exclusion face must not silently shrink; empty merged table parks DEGRADED via truth_layer scanner_ok) feeding the existing `build_truth_layer(scanner_prefixes=)` hook; `corpus.build_benchmark` skips the scanner_ranges source and strains every stratum through `apply_scanner_filter` post-cap (宁少算, no backfill; no-source / missing-CSV degenerates bit-identical to the status quo); S12 corpus versioning and the S17 truth-fingerprint-change trend-reset note are pinned by tests (40.124.175.29 two-arm: member diff exact, removed_by_scanner_filter 0→1, both scanlist fingerprints change); the truth-side ops runbook rewire (legacy censys+Xpanse+Shadowserver hand-merged asset → scanner_ranges loader) is measured advice left for the ops round. Product surface: zero new UI mechanism — the native label rides the existing native_categories chip chain (_merge.py:306-331 → _api_models details → SourceDetailRow chip), pinned by double-ended contract tests (real lookup() full chain on 40.124.175.29: malicious verdict stands, label in details, corroborated=False, verdict_conflict stays dark; LookupResultOut outbound shape re-pinned; frontend chip full-literal visible); tamper-red proven both ends. Gates: source tests 11/11 offline (66+1 name-set verbatim vs spec; negative assertions: no OpenFilters, no -nt-, no GreyNoise), backend eval 489/0, backend targeted 1791/0 (source+core) and 111/0 (label chain), frontend vitest 11/11, full backend 2251 passed / 9 skipped (4 dispositions: 2 legit metadata-snapshot updates landed with the ticket, 1 pre-existing baseline failure unchanged, 1 load-flake single-run green).
+  - **扫描器自报段三件套:源接入+eval 真值层排除+标签钉测(issues #212 #213 #214,文档收口 #215;scanner_ranges 批;普查 docs/research/2026-10-26-scanner-ranges-census.md,ADR 0011)**:新增 `backend/ipdb/_sources/scanner_ranges.py`(cdn_edges 同构 _FEEDS 多发布者表)接入 MISP warninglists 66 个非 nt 扫描列表(CC0,CIRCL 镜像,~29,070 条 CIDR v4/v6 混排,日期版本日更)+ Palo Alto Xpanse 参考页直读(9 CIDR——唯一不在 MISP 的大厂);OpenFilters/internet-scanners 同上游刻意不接——每真值一路,防回声;GreyNoise API-only 红线外;证据面=用户裁定的指控形态:Evidence(classification="scanner", verdict="malicious", native_categories=["研究扫描 · <厂商/机构>(自报段)"], r=0.90, stale_days=7, provenance_tier B)——malicious 是行为类型断言(词表 "aggressive scanning"),意图活在 native 标签里,改判权留用户;与行为证人(sw_scanners)独立两票 corroborated 合法(观测线独立,无谱系重叠)。eval 真值层(ADR 0011,产品/eval 刻意不对称=特性):装配侧排除——`layer_builders.scanner_ranges_prefixes` 从原始中间件 CSV(backend/data/scanner_ranges.csv,首列)取前缀表(勿从 LMDB 反解;缺 CSV raise FileNotFoundError——排除面不得无声缩水;合并表空由 truth_layer scanner_ok 落 DEGRADED),喂现成 `build_truth_layer(scanner_prefixes=)` 挂点;`corpus.build_benchmark` 跳过 scanner_ranges 源+封顶后逐分层过 `apply_scanner_filter`(宁少算不回补;无源/缺 CSV 与现状逐位一致退化);S12 语料版本与 S17 truth 指纹换版趋势重置注记以测试钉住(40.124.175.29 两臂:成员差精确、removed_by_scanner_filter 0→1、双 scanlist 指纹皆变);truth 侧 ops runbook 换源(legacy censys+Xpanse+Shadowserver 手合资产→scanner_ranges loader)为测量建言,留 ops 轮人裁。产品面:零新 UI 机制——native 标签走既有 native_categories chip 链(_merge.py:306-331→_api_models details→SourceDetailRow chip),双端契约测试钉死(40.124.175.29 真 lookup() 全链:malicious 判决照旧、标签在 details、corroborated=False、verdict_conflict 不点亮;LookupResultOut 出海形状复钉;前端 chip 全文字面量可见);双端 tamper-red 红证。门:源测试 11/11 offline(66+1 名单与 spec 逐字;负断言:无 OpenFilters/无 -nt-/无 GreyNoise)、后端 eval 489/0、后端定向 1791/0(源+core)与 111/0(标签链)、前端 vitest 11/11、后端全量 2251 过/9 跳(4 挂处置:2 快照合法更新随票落地、1 基线同挂、1 负载 flake 单跑绿)。
   - **Artifact-name table single source (issue #202; C1-C3 refactor batch, umbrella #201; pure refactor, disk names byte-identical)**: artifact filenames were raw string literals in three consumption layers (gates.mjs ~45 sites, steps.mjs ~31, selftest-pipeline's 17-name hand-copied list) — adding 09-budget.json in #199 meant hand-syncing three layers (live example); new `ops/agent/artifacts.mjs` exports `ARTIFACTS` (18 fixed names, frozen) + `ARTIFACT_NAMES` + the DEEP parameterized families (`deepArtifactName`/`deepTranscriptName` factories + `DEEP_ARTIFACT_RE`/`DEEP_TRANSCRIPT_RE` matchers); all three layers now take names from the table; a two-way closure sentinel pins table<->disk-set coherence (table gains an entry without a writer = red; a step emits a new artifact not in the table = red); golden/cases.json and the canned-faux artifact-name set are bit-identical pre/post (md5-equal); umbrella closure folds in the fourth write surface — finding-exit.mjs's EXITS_ARTIFACT now takes `ARTIFACTS.ANCHOR_SWEEP_EXITS` (C1-merger find). Known non-swept literals by spec design: faux.mjs transcriptName x3 and pipeline.mjs HEALTH_PIPELINE out/SKIP_WHEN (outside the spec's three layers; the disk-level sentinel backstops). Gates: selftest-pipeline 78/0 (tamper proof: injected fake table entry -> closure sentinel FAIL "工件缺:10-c1-tamper.json" -> restored green), t4 63/0, contract 18/0, golden 9/0, standard-check 47 OK, backend ops subset 271 passed 1 skipped.
   - **工件名表单源(issue #202;C1-C3 重构批,伞 #201;纯重构,磁盘工件名字节不变)**:工件文件名原是三层裸字符串(gates.mjs ~45 处/steps.mjs ~31 处/selftest-pipeline 17 名手抄清单)——#199 加 09-budget.json 时手抄三层就是活例;新 `ops/agent/artifacts.mjs` 导出 `ARTIFACTS`(18 固定名 frozen)+`ARTIFACT_NAMES`+DEEP 参数族(deepArtifactName/deepTranscriptName 工厂+DEEP_ARTIFACT_RE/DEEP_TRANSCRIPT_RE 匹配面);三层全改从名表取值;双向闭包哨钉名表↔磁盘集合自洽(名表加项无写出面=红,步面新工件不入表=红);golden/cases.json 与罐装 faux 工件名集合重构前后位等价(md5 同);伞收口补入第四写面——finding-exit.mjs 的 EXITS_ARTIFACT 改取 `ARTIFACTS.ANCHOR_SWEEP_EXITS`(C1 merger 发现)。spec 设计内留白:faux.mjs transcriptName ×3+pipeline.mjs HEALTH_PIPELINE out/SKIP_WHEN 字面量(三层措辞之外;磁盘级哨兜底)。门:selftest-pipeline 78/0(红证:名表注入假名→闭包哨 FAIL "工件缺:10-c1-tamper.json"→还原回绿)、t4 63/0、contract 18/0、golden 9/0、standard-check 47 OK、backend ops 子集 271 passed 1 skipped。
   - **Monthly-admission refusal block + ctx shape consolidated to single sources (issues #203 #204; C1-C3 refactor batch, umbrella #201; pure refactor, zero behavior change — C2+C3 scope; C1's artifacts-name table lands with its own worker)**: C2 — the monthly kill-switch refusal block was verbatim-duplicated across both faces (agent.mjs runRound legacy face ↔ pipeline.mjs runPipeline, the #200 spec-axis finding); `ops/agent/pipeline/budget.mjs` now exports the single source: `refusalOf` (pure text face) + `admitOrRefuse` (stateful shell: monthlyAdmission reads → on refusal the `[admission]` log line + the ledger trace row via appendRefusalTrace, write failure logged never flipping the verdict) ; both faces rewire to it and only map their exits (legacy throws {code} → failure envelope exit 1; pipeline emits emitFail envelope exit 1) — refusal message / [admission] log / ledger row / exit codes byte-identical, pinned by a new selftest-pipeline sentinel against a real ledger tree; the monthly-cap tolerance doctrine is centralized as the single annotation at admitMonthly (exact comparison, no ε — equality passes, spec-pinned "only > cap rejects"; the +1e-9 dust-tolerance precedent stays on the round-budget face and must never leak here; if a monthly ε is ever added, admitMonthly is the only legal site). C3 — runResume's hand-rebuilt ctx had drifted from mkCtx's shape with only human memory holding them together; pipeline.mjs now exports `ctxShape` — the ctx field-list single constructor; mkCtx passes validated fresh values, runResume passes state.json-recovered values with missing knobs re-read from env (budget_knobs/model_face fallbacks unchanged; budget_cny stays unvalidated and un-filled — an old round's undefined passes through verbatim); knob validation stays in mkCtx exactly as-is (not moved, not loosened — ctxShape does zero validation by design), state passes by reference, modelCalls/dupRegistry/usage are fresh per construction; two new sentinels: the ctxShape closed key-set (23 keys incl. the ctxPaths four) and the unique-construction-site check ("dupRegistry: new Map()" / "modelCalls: 0" / "usage: { input: 0" each appear exactly once in pipeline.mjs — a hand-rebuilt field list recurring is red). Gates: selftest-pipeline 81/0 (baseline 78; red proofs: unimplemented shell 78/1 "admitOrRefuse is not a function", tolerance flip <=→< reddened the equality-admits pin, removing ctxShape's breakerKnobs reddened the closed-set sentinel), t4 63/0, contract 18/0, golden 9/0 (canned face doubles as the bit-equivalence spot check), standard-check 47 OK (挂点31/建议10), backend ops subset 271 passed 1 skipped, legacy smoke exit-2 paths unchanged.

GLOSSARY.md

@@ -21,6 +21,7 @@
 - **合法空**:bare-IP 多列表 feed 中 HTTP 200 + 空 body 的子列表——发布方「当前无此类条目」的声明:落 0 字节文件、下一 epoch 出清该列表条目、不计失败。仅限 200;404/超时/锚空仍是失败。firehol 等头部化格式不适用(空文件无头即坏)。锚:_sources/blocklist_de.py(download)
 - **活性锚**:同批下载中证明出版方存活的列表(blocklist_de = all.txt 聚合);锚空 = 本轮零替换 + 记真失败(fail-closed),防灾难性全空清库。判例:blocklist_de ircbot 2026-10-15(grill Q1-Q3)。锚:_sources/blocklist_de.py(download)
 - **载入期基础设施过滤**:聚合类源 rebuild 载入面排除 curated 公共基础设施单址集——聚合器转载的指控对公认解析器不构成行为证据,排除在投影层(数据文件零改写,无损管线不破)。真源=infra_services curated 集(单真相,勿查可能纪元落后的 LMDB);基类默认关,事故源显式 opt-in(infra_noise_filter=True);宁少算=仅精确单址命中,含 infra 的更大 CIDR 不排;IP_RADAR_INFRA_NOISE_FILTER=0 + 重建即复原,排除计数+条目日志化。判例:8.8.8.8(myip_ms 单源)/114.114.114.114(cps+ziyadnz 假印证)/1.1.1.1(cps)三公共解析器误报(2026-10-24 事故,ADR 0008 发现固化;哲学同族=资产弃权)。锚:_sources/_infra_filter.py
+- **自报段**:IP 段由扫描主体自己公布、经机构镜像(MISP warninglists,CC0;Xpanse 直读)到达的厂商声明——身份证据非行为观测;镜像同源(OpenFilters/internet-scanners)只接一路防回声,与三方自采观测面(sw_scanners)无谱系重叠;兼作 eval 真值层排除清单(ADR 0011)。判例:scanner_ranges 接入 grill 2026-10-26。锚:backend/ipdb/_sources/scanner_ranges.py:129(下载面 _FEEDS 66+1 表在 :119,harvest 证人 :150)
 
 ## 证据
 
@@ -31,6 +32,7 @@
 - **benign(两层,勿混)**:①指控缺席 = 弃权——观测级弃权拼写 verdict=""(R17A-1,序列化特判保留,勿兑底 malicious),顶层无指控观测时缺省序列化 verdict="benign"、confidence=0(_types.threat_summary);不投票、不亮黄灯,是「无证据」不是「判良性」——弃权绝不作良性背书(见「无证据≠清白」)。②良性断言 = 非指控信号源明确断言 benign(infra_services 升格为首个此类源,#188:curated 公共 DNS/NTP 单址双面 harvest——service 资产面照旧+classification_type=服务角色(dns/ntp)+verdict="benign";弃权拼写 "" 期作废);与指控同场才算真对立(verdict_conflict=True——良性断言 type-agnostic 注入每个指控组同场,注入面在 _registry.lookup);宁少算:良性证人不投票(组 conf/顶层重融合不变)、不压制指控 verdict(precedence 恒指控胜)、不产 STIX Indicator;conflict 留痕人裁不自动压制(锚巡检 conflict 投影/finding 出口 issue 标记);无指控组时以自身 ctype 成纯良性面(verdict=benign conf≈r,区别于①弃权 benign/0)。锚:_evidence.py(to_dict 弃权特判)、_types.py(threat_summary)、_merge.py(_assess_classification)、_registry.py(lookup 分组注入)、_sources/infra_services.py(harvest 双面)
 - **资产弃权**:源只陈述基础设施属性(proxy/hosting/tor),不构成滥用指控——verdict="" 不计票、不产 STIX Indicator,仅走 attributes 通道。判例:cloud_ranges(首创)→ip2proxy DCH(SM-F2)→ip2proxy VPN/PUB/TOR(ADR 0001,2026-10-12)。锚:_sources/ip2proxy.py(_proxy_evidence)
 - **行为源**:以观测到的滥用行为(扫描/爆破/C2 回连/举报)立证的源,如 bruteforce/ciarm/abuseipdb;与资产属性源相对——「挂着代理」由资产弃权陈述,「在作恶」由行为源指控。锚:_sources/bruteforce.py
+- **公开研究扫描**:scanner 分类的自报段证人形态——verdict=malicious 是行为类型断言(词表 "aggressive scanning"),非独立过错指控,意图活在 native 标签「研究扫描 · <厂商/机构>(自报段)」里;与行为源指控构成独立两票(观测线独立,corroborated 合法);改判权留用户,非良性断言(自报不构成良性背书,无证据≠清白)。判例:grill Q3 2026-10-26(用户裁「还是恶意但打标签,用户自己评估」)。锚:backend/ipdb/_sources/scanner_ranges.py:150-161(harvest 构造 native_categories=["研究扫描 · <厂商/机构>(自报段)"] :161)
 - **无证据≠清白**:conf 0 = 无源命中,不是"干净";缺数据直说缺。锚:_api_models.py(FieldOut)
 
 ## 融合

NEEDS.md

@@ -24,7 +24,7 @@
 | ASN 业务类型(ISP/content/edu) | 2026-10-10 | 候选 | CAIDA as-classification 已死(官方 legacy、下载关闭,2026-10-10 查证);其真值本就是 PeeringDB 自报 info_type → 正路 = peeringdb 既有源扩展(org/net 端点 join,闸门判例),立项另裁 |
 | 攻击者 IP 第一方测量(蜜罐/传感网) | 2026-10-10 | 已闭 | knock_knock 已交付(PR #104,blacklist=binarydefense 双生判例) |
 | DDoS 攻击源 | 2026-10-10 | 已闭 | rtbh 已交付(PR #104,ddos 槽首个专职证人;许可用户签核 2026-10-10) |
-| IP 是否研究扫描器(Censys/Shodan 等已知扫描段) | 2026-10-12 | 已立项（2026-10-12 eval 闭环批裁：真值层剔除同源语料污染，产品面同型呈现） | 官方公布段静态文件→attributes 通道，cdn.notice 同构判例；兼作威胁真值层排除清单（语料版本随附） |
+| IP 是否研究扫描器(Censys/Shodan 等已知扫描段) | 2026-10-12 | 已交付待合(SDD 批 PR #216:#212 源接入+#213 eval 排除+#214 标签钉测+#215 文档收口;ADR 0011;报告 docs/research/2026-10-26-scanner-ranges-census.md) | 官方公布段静态文件→attributes 通道,cdn.notice 同构判例;兼作威胁真值层排除清单(语料版本随附);通道裁定=MISP warninglists -scanning 66 列表(CC0)+Xpanse 直读,80.2% 池内污染实测,grill 全裁 2026-10-26(恶意保持+「公开研究扫描/自报段」标签人裁;eval 语料装配侧排除,ADR 0011;单批三件套);truth ops runbook 换源接线(legacy censys+Xpanse+Shadowserver 手合资产→scanner_ranges loader)留 ops 轮人裁 |
 | 我们输出的 verdict 在真实执法点(阻断/检测消费端)的 TP/FP 表现 | 2026-10-12 | open | 业界金标准 = 消费端反馈回流(ThreatConnect CAL 社区众包/MISP sightings/GTI 分析师纠正,docs/research/2026-10-12-industry-loop-practices.md);前置 = 产品需有消费 verdict 并能回报处置结论的执法点 + 查询隐私裁决,2026-10-12 用户记「后续想做」;本批以蜜罐锚+纠错检验作无遥测近似 |
 | 土耳其政府 SGB 情报(siberguvenlik.gov.tr/api/address/index) | 2026-10-10 | open | JSON API,本轮未验活未取样;与 RTBH 同区域第二票,区域扩展时再验 |
 | geo_asn/attributes 通道源的接入关读数口径(现走硬门得零信息报告) | 2026-10-13 | open | 对抗审查第1轮:读数覆盖域=threat/asset 两轴(verdict N/A-ASSET 早退+metrics 仅遍历 classifications),geo_asn 现役 7 源即此态;标量轴/排除清单覆盖率口径未裁,待用户(扩张 Q4/Q18 gate 覆盖域须裁) |