Repository navigation
Prove fresh Layev telemetry through collector cache report and visible UI #3
Description
Gap and evidence
Draft PR #1 adds the collector, raw-v2 cache, report and React model-runs view, but the pipeline test skips without a fresh export and stops at report JSON. No populated loopback browser result or cache-only visible readback has been recorded. The current backend suite is blocked by #2.
Acceptance
- Generate a fresh genuine Layev train/checkpoint/calibration/evaluation export with optimizer steps, parallel-question counters and verifiable attempt receipts; record source/config/split/artifact hashes.
- In isolated registry/cache paths, execute producer export -> collector -> raw-v2 cache -> cached_metrics -> report/report_service -> existing React UI, with the native pipeline test unskipped.
- Prove recovery after missed polls/restart, uncertain lineage shown as uncertain, and cache-only report/UI readback after the source export disappears; preserve legacy Flow filters, cloud inventory and authoritative scheduler status.
- Show useful tokens, padding, question batches, losses/reward/calibration and measured resource units without inventing local SkyPilot IDs or converting unknown to zero.
- Run supported backend suite, Ruff, frontend typecheck/Vitest/build; inspect a populated loopback page, save screenshot and browser-error record, and read back the actual /api/report. Keep PR #1 draft until these gates are satisfied.
Related native model work: Jev-Engineering/layev#3 and Jev-Engineering/layev#4.
Activity
romgenie commentedon Sep 27, 2026
Additional isolated producer evidence — NOT full Overwatch acceptance
A fresh CPU miniature software exercise in this session completed 8 optimizer steps, checkpointing, exact same-policy interrupted/resumed continuation, separate calibration and a software evaluation export. It observed one state-prefix pass followed by one 16-question batched suffix pass. A missed-attempt recovery exercise used isolated registry/cache paths and the actual Layev-vendored provider/adapter modules. The report preserves local association, unknown cost fields and measured CPU RSS; it does not invent a SkyPilot job.
The calibrated checkpoint SHA-256 is
adf8413b5e25098f7d2af0ee69f158fa02fe8086f02b887b434ac80d8b364809. Its explicit evidence class is tiny-synthetic-fixture: maximum observed training branch 169 tokens, maximum aggregate 400, 16 examples and 8 optimizer steps; zero 32k/64k examples. A separately started real loopback Layev HTTP service served 12 requests/93 questions with exact direct-vs-HTTP matches, one prefix per request and bounded batches (the 20-question cases used 16+4). These are actual HTTP/software measurements, not a native Qwen, quality or Jev result.Crucial limitations: source-export Python 3.13.5/torch 2.10.0+cpu, not the supported locked Overwatch environment; this did NOT execute Overwatch's collector→cached_metrics→report_service→React UI path, inspect the owner's actual /api/report, or produce a populated browser screenshot/error record. The source export was based on verified old Layev core preimages plus this session's uncommitted tokenizer candidate, not a Git clone; its source report explicitly records git_status=unavailable. These receipts are not a current PR-head backend result.
The delivery bundle preserves the JSON receipts and test logs without checkpoints, weights, credentials or private datasets. Keep all issue acceptance boxes open and PR #1 draft pending #2's authorized backend environment and the actual full UI gate. No existing workstation run, registry, cache, running service or WSL instance was modified.
CompleteDotTech commentedon Sep 28, 2026
Real producer partial readback — 2026-09-27/28
A fresh local Layev pinned-weight, two-step CUDA training export is available outside Git. Its content-free telemetry SHA-256 is
0d6aec562ccf958850b3567005569c193687b4935f9732b90080ac2991daef25; it includes two optimizer steps, an actual checkpoint artifact, an evaluation artifact and batching evidence. Using the current draft PR #1 adapter source, the real export decoded and rendered through the direct cache presentation adapter with run IDlayev-native-probe-20260927-a, two steps,verification: verified, local association, unknown SkyPilot status and no warnings. The private direct-adapter receipt is source-bound to that export.The full producer -> collector -> raw cache -> report -> visible UI gate is still OPEN. Its unskipped test was attempted under Python 3.13.12 using a separate partially provisioned environment and stopped at missing
pyzstdbefore collector execution. This is not the locked Overwatch environment; the approved privatetscoreindex route is still unavailable locally and in CI, so no full backend pass or browser readback follows. The original export was not modified, and no fake provider was used to claim the initial report. Keep #3 and dependency #2 open.CompleteDotTech commentedon Sep 28, 2026
Updated local acceptance probe, 2026-09-28: the same genuine Layev trained telemetry export (SHA-256
0d6aec562ccf958850b3567005569c193687b4935f9732b90080ac2991daef25) was supplied to the unskippedtests/test_kev_laya_pipeline.pypath. In a separate temporary Python 3.13.12 target, I added the public locked-version dependencies needed to advance import resolution. The test reached the realskypilot==0.13.0rc1import and then failed before collection because SkyPilot imports Unixresource, which is unavailable on Windows. No reader/provider was replaced by a fake and no collector/cache/report/UI success is claimed. The existing WSL interpreter is Python 3.12.3, below Overwatch's supported >=3.13.12 minimum.The approved private
tscoreCodeArtifact route is still absent for the full locked backend check. Issue #3 remains open pending a supported Linux Python 3.13.12 environment with that route and the genuine producer -> collector -> cache -> report -> visible UI readback.CompleteDotTech commentedon Sep 28, 2026
Native local integration update on draft PR #1, head
aef193862499734af552860744db667b64afebd1: in an isolated Linux Python 3.13.12 environment with the real SkyPilot 0.13.0rc1 import, the actual trained Layev export at SHA-2560d6aec562ccf958850b3567005569c193687b4935f9732b90080ac2991daef25passed the collector → persisted raw-v2 cache → report → cache-only-after-copied-source-removal test. The full source backend suite with that export set passed 120 tests, zero skips. No provider was mocked in the collection pass.An isolated loopback service rendered the verified two-step run in the React UI. Browser inspection showed useful-token/padding/question-batch units, loss/reward, unknown calibration/Sky/W&B fields, local association and measured CUDA allocation; console errors and warnings were both zero. A private evidence receipt and screenshot were retained outside the repository (SHA-256
b0796164c3bdb0df9f033f05a34487681d3e4c8c187dad24f24c245081ad8865and6d8ad1c5879c783d223ddbc33555542f5107adc411e1aa871bbb982942b8). The owned loopback service was stopped afterward.This uses public dependencies, not the locked private TypeSafe/TSPath environment. Full hosted backend CI still stops at missing authorized private index credentials; live S3/W&B collection and remaining lineage/recovery cases are not established. Keep #3 open.
CompleteDotTech commentedon Sep 28, 2026
Recovery readback: a separate collector process cached the final real resumed export. After the copied source was removed, a new Python process read raw-v2 cache and verified attempt index 1, its actual parent attempt ID, and two optimizer steps. Cache SHA-256:
d41d531e8189225640291ca251673df2bc9634e8d1847a5fc00f91bb6d861412. Updated private evidence receipt SHA-256:7e48be3829d14636c78f01419e3f9f27c53cdccb0801b5ec15902fb468d49303. This proves persisted readback of a real resumed attempt; it does not establish remote transports or the locked private-index CI gate.CompleteDotTech commentedon Sep 28, 2026
Additional real local stages: using the existing resumed Qwen checkpoint and generated marker suite, I completed a separate CUDA calibration stage and development evaluation stage into new paths without changing the training run. Overwatch's genuine local collector read all three telemetry exports and produced three verified model runs with zero collection warnings. The React UI visibly rendered training, fitted calibration and development evaluation together; Playwright observed zero console errors/warnings. The private stage receipt is SHA-256
34424ec764d475c306056a6c87339906f353a6c97a5c7c174eca1421fe1fcf0b; the full-page screenshot is SHA-2562021f19863b40f1405e4da3a227f4f2a9dd858fd64cdd9eee0153f4fc4801874.This generated marker suite is synthetic. The development evaluation was 5/15 correct (33.3%) with ECE 0.6574, and all three fitted per-type temperatures reached the upper bound. These are preserved poor results, not representative quality or a corrected regression. The locked private-index CI and live remote collection gates are still open; keep #3 open and PR #1 draft.
CompleteDotTech commentedon Sep 28, 2026
Cache-only visible readback is now observed in a new service process: the copied registered export was absent before startup, yet
/api/reportreturned the previously verified real run and asource_unavailable_or_invalidwarning. The React UI showed the same run and warning; Playwright recorded zero console errors/warnings. Private evidence receipt SHA-25664d80f39f0508b18c4cbcd470b5d86697c5a7dbced298a2800575d1ec7479575; screenshot SHA-25652adce20c938f72ed917b5ff15faad2238f9a0abcbafcebf968dafa94bce08ac. This is a local missing-source case, not remote transport proof. The owned loopback service was stopped.CompleteDotTech commentedon Sep 28, 2026
Controlled negative-path update (separate from genuine producer evidence): a copy of the real export was altered to skip an unobserved attempt and omit hashed receipts. The collector/report kept the last verified attempt 1 and its two optimizer steps, classified the injected attempt 3 as
last_verified_with_newer_uncertainty, and emittedunverified_attempt_lineage. The React UI visibly labeled the newer observation uncertain and showed an identity alert; Playwright found zero console errors/warnings. Private negative receipt SHA-25631d606162d97cadc739897fc419527638b280e1033865707e91468e21eaa7ffa; screenshot SHA-256de96c9379ba12875a332db12ff35ff985565b59eac9d3ca56f613c4e4719dc3e. This is an injected failure case, not a real failed lineage event. Locked CI and remote transport gates remain open.CompleteDotTech commentedon Sep 28, 2026
Local-only producer-to-report progress on clean PR #1 head
aef193862499734af552860744db667b64afebd1: on Ubuntu WSL with Python 3.13.12, an isolated public-dependency-only environment ran the existingtests/test_kev_laya_pipeline.pyunskipped against the preserved genuine two-step native training export. Source telemetry SHA-2560d6aec562ccf958850b3567005569c193687b4935f9732b90080ac2991daef25; run IDlayev-native-probe-20260927-a; optimizer steps2and parallel batch size greater than one were checked by the test. Result: 1 passed, 0 skipped, with five upstream SkyPilot/SQLAlchemy deprecation warnings.The test used isolated registry/cache paths, executed the real local collector, wrote
runs-v2.json, built the cached report, then removed its copied export and verified cache-only report readback. Its private report receipt SHA-256 is3b6d531f71ea1657d633b2b74abe5a9cb7be80c7045a485b876027b18bf869c4; it recordskev_layacollector statusok, model-run verificationverified, two optimizer steps and cache-only readback true. The original producer file was unchanged.This is a local-only partial integration result from a preexisting training export. It does not provide a fresh complete train/calibrate/evaluate export, browser screenshot,
/api/reportservice readback, missed-poll/restart matrix, private TypeSafe/tspath locked install, full backend suite, live W&B/S3 or completed UI acceptance. PR #1 stays draft and #3 remains open. A prior attempt accidentally selected an evaluation-only snapshot with zero optimizer steps and failed the test's precondition; it was not substituted for training evidence.CompleteDotTech commentedon Sep 28, 2026
Follow-up to the local-only partial pipeline receipt on clean PR #1 head
aef193862499734af552860744db667b64afebd1: an owned loopback Overwatch service used the genuine two-step training export, isolated registry/raw-cache paths, and explicit local-models-only mode. The actual/api/reportreturned HTTP 200 with one verified run (layev-native-probe-20260927-a), two optimizer steps, zero cloud resources and null SkyPilot/W&B association. Private initial report SHA-256:0d583bd0d30f4fee7144731a3c468c9f0b660fee7bb6c6620489e11d05dca481.Headless Chromium rendered the existing React page with the populated decision-model card: useful/padding/compute tokens, two question-batch passes, six questions, loss/reward fields, explicit unknown calibration, measured CUDA memory and local association. The full-page screenshot is retained privately (SHA-256
117626bc8d51e275d9faac7f91d29164463f248b8592a7c831d5ccedc3c69294); it is not attached publicly because the artifact table displays local filesystem paths. Browser console: zero warnings and zero errors;/api/reportwas a 200 network request.After byte-verifying a disposable copy against the untouched source, I removed only that copy and triggered a manual refresh. The report retained the same verified run and the UI showed
source_unavailable_or_invalidrather than silently dropping it. After stopping and restarting the owned service against the same cache with the copied source still absent,/api/reportagain returned the run with two steps and a source-unavailable warning. Private after-removal/restart report SHA-256:039d639ddefd82ad4ef3c554b5db07474102cf9419486b344dc66adb67042b7b/8eacfd0d4da1844ae80885d2425797f6e5cd17fc1b86be89eca498181999e0fd; warning-state screenshot SHA-256aa36f85354d725a44b498ed692f0f6d553fd1cf33cd52a9a0347c6e177ae2888. The original export hash remained0d6aec562ccf958850b3567005569c193687b4935f9732b90080ac2991daef25. The loopback service was stopped cleanly.This advances local collector/cache/report/UI and source-disappearance/restart evidence only. The source is a preserved short training export, not a newly generated complete train/calibrate/evaluate package; the supported locked private backend, full UI acceptance matrix and real remote collection remain pending. #3 and PR #1 remain open/draft.
CompleteDotTech commentedon Sep 28, 2026
Controlled synthetic negative consumer probe, separate from the genuine producer evidence below: I copied the already verified local cache and derived a private test payload from the genuine two-step export by removing its root attempt receipt. The original producer/export and previous cache were untouched. The altered payload still passes the telemetry schema validator, while the lineage verifier correctly refuses to treat it as a verified root. Private probe script/payload SHA-256:
e452e3a642d0624fc25c2ad679a025250d2a6e5f2208164ba0a0f475fa8a5096/f0b8add8ed14a2b73666205fdfeb3a5a476b277d17bbe46e1d3e7d869d1649e7.The real local-only collector consumed that injected payload in an isolated copy of the cache.
/api/reportretained the prior verified run and reportedlast_verified_with_newer_uncertainty, an unverified observation withsequence_payload_conflict, and a warning; report SHA-2568b81aae150990c91fe21a8e6fd084025db2c7efbe101a21f38d048ad1234717d. Headless Chromium visibly showed “Last verified attempt; newer observation is uncertain” and “Attempt identity requires verification”; private screenshot SHA-2568190370e03dd947ffc05f01bb5a44ef8d9e7555056c1cfd3208f1b153d1b037c. Browser console had zero warnings/errors. The owned service was stopped, both test ports closed, and the genuine export hash remained unchanged.This proves the consumer/UI warning behavior under a deliberately injected lineage conflict. It is not evidence that a real trainer produced an uncertain attempt, and it does not complete the fresh native producer, private locked backend or remote criteria. #3 and PR #1 remain open/draft.
CompleteDotTech commentedon Sep 28, 2026
Fresh Stage 3 producer-to-UI evidence (2026-09-28; partial acceptance):
904656cc2102ca834e765061288954a342b9511bgenerated actual supervised training (160 optimizer steps), 40-step supervised and reward extensions, checkpoints, held-out calibration, evaluations, and telemetry. Producer report SHA-2561714175064e50953bd40da0efd4a5da13d5f6a9805d851b6b91db3b20d15123e; protocol041357cbd9cbe1024461e21df7574a65f81f9aefec6eddc87e9c9cfc6b74888d; suite manifest3ed3666edf3312a4dac09e95ea9ad1e3cd95d90bc4f9f0377f0c2d0513749c64. The 160-step training export SHA-256 is86ee95506529b68e897fedfb586ba50dc40a603f50e5d585336575f998a6a65a; the calibrated-stage export is234c9e7af82cc20de0fabf5da58653e4d2fdbcc4bbd767fd078ac9f311840ce2and correctly reports zero optimization steps.aef193862499734af552860744db667b64afebd1, the existing unskipped pipeline test against the fresh 160-step producer export passed 1/1 under supported Python 3.13.12. It exercised the real collector, persisted raw-v2 cache, cached report and cache-only read after removal of only a disposable source copy./api/reportreturned HTTP 200, four verified local runs, zero cloud resources and zero source warnings (private response SHA-2565f067ced3b46fb2dedac081dce17b72121cb3da0fe578bea0052f7d5ddb88cd4). Headless Chromium rendered four model cards, including 160 steps, useful/padding tokens, batched questions, measured CPU RSS and fitted held-out calibration. Its/api/reportrequest returned 200 and browser console recorded zero warnings/errors. Private full-page screenshot SHA-256e6d116adc537ac2153fc4166f5cf80293405eef411bb3c5ccd3aa56f72fc9417(retained privately because UI artifact links include local paths). The owned service was stopped after capture.This is a genuine producer and consumer pipeline using a tiny synthetic CPU fixture, not a representative quality or native long-context result. The known
case=99999regression remains wrong for Noul and Score. Private TypeSafe-index locked backend CI, live remote collection and remaining full acceptance scenarios are still open; PR #1 remains draft and this issue remains open.