Repository navigation
Validate pinned Qwen weights and CUDA training and serving #3
Description
Pinned source-gate delivery — 2026-09-27; PR #18 merged
The opt-in pinned backbone source gate is merged: verified SSH-signed source d7c6aefd6d92ffed938d1a2c7073e0a80dd425f6, verified GitHub-signed merge 6a99afde9df06be616a2ac75fb63e4e933bdfd96. All five PR jobs and all five post-merge main jobs succeeded; the primary checkout is clean and synchronized.
The gate fixes the accepted weight digest and exact size to the pinned Qwen Git LFS pointer, verifies measured bytes against it, and checks the four auxiliary files only against the local acquisition manifest. It is read-only and does not run automatically in native-init. Review fixed a Windows metadata false rejection before delivery. Local validation passed 752 tests with 11 explicit skips, source audit, critical lint, compileall, lock check and CPU numerical comparison. The package-free no-input gate returned blocked/exit 2.
No actual 988097824-byte Qwen weight payload was downloaded or verified by this delivery. No independent Transformers oracle, native initialization/reload/serving, CUDA FP32/BF16, context, quality or Jev result follows. Keep all five acceptance boxes unchecked and retain the existing #2 dependency. A real source-gate receipt on acquired weights is a prerequisite, not issue closure.
Current prerequisite readback — 2026-09-27, after tokenizer PR #13
The tokenizer initialization defect described in the original dependency note is no longer the current source blocker. The reconciled v4 repair is merged through PR #13 at e1db691cc7710ff808fa1220b16628c0d675d16c. Fresh GitHub reads confirm source commit 078eaf7195b118cdd5c907bc12b0c556d8dae32f is verified/valid SSH-signed, the merge is verified/valid GitHub-PGP-signed, and all five post-merge jobs succeeded in run 36312254077. Do not reapply the earlier v3/v4 archive patches over this merged source.
The primary-workstation receipt on #2 reports the real Qwen tokenizer gate passed 36 literal cases under Python 3.13.12 / tokenizers 0.22.1, with source-output accounting, metadata reload, exact budgets and v3 refusal checked. The recorded tokenizer SHA-256 is c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539 at revision 060db6499f32faf8b98477b0a26969ef7d8b9987. Its reported full workstation suite result is 413 passed / 4 explicit skips (official SDK, CUDA FP32, CUDA BF16, Windows symlink privilege). These workstation tests were not independently rerun by this readback session.
The remaining execution prerequisites are actual pinned model weights and the supported local hardware/runtime. That workstation receipt states the pinned model weights are absent and its installed PyTorch build is CPU-only. This does not establish that the PC lacks CUDA-capable hardware; hardware capability and the authorized local CUDA setup still need inspection. Keep CUDA work on the user's PC unless an acceptance criterion genuinely requires another authorized environment. No remote CUDA replacement was provisioned.
The native blocked-by relationship to #2 is retained: that issue remains open for actual pretrained initialization/checkpoint/serving preprocessing-identity evidence. The weights-based validation here must supply that integration evidence rather than treating tokenizer metadata reload as trained-model reload. The former tokenizer-construction failure is not a reason to repeat old patches or defer preparing the real-weights gate. No native-model, CUDA, context, quality or Jev comparison pass follows from the tokenizer-only result or green hosted jobs.
Gap and evidence
The validation record reports CPU fixture success but has not executed the pinned pretrained Qwen weights, reference-model comparison, native LoRA/full-weight/activation-checkpointed training, or CUDA FP32/BF16 gates. Thus model behavior on real weights and hardware remains unknown. Native validation already refuses a fixture, which is the correct boundary.
Original dependency: #2. The earlier pinned-tokenizer construction failure is superseded by the PR #13 delivery above; the remaining native integration acceptance is not.
Acceptance
- Fetch and hash-verify the pinned Qwen/Qwen2.5-0.5B revision and publish a reproducible environment/hardware receipt without storing weights or secrets in Git.
- Initialize and reload actual pretrained weights; compare independent backbone outputs with the pinned Transformers oracle at the declared tolerance.
- Exercise native inference plus LoRA, full-weight and activation-checkpointed train/resume paths where supported; retain exact branch-isolation and shared-prefix numerical checks.
- Run CUDA FP32 and BF16 where hardware supports them; record skips as unsupported, not passes. Measure peak GPU memory and latency on declared shapes.
- Produce commands, artifact/checkpoint hashes, test results and remaining hardware limits. Do not substitute CPU doubles or configuration flags for native evidence.
Native partial acceptance readback — 2026-09-27/28
Protected PR #24 merged at 937fee343fdc758cd241e3e2bb48090a60ddd55f; all five PR and five post-merge jobs passed. The primary checkout was synchronized cleanly. Source gate verified the actual pinned Qwen revision 060db6499f32faf8b98477b0a26969ef7d8b9987 weight bytes: SHA-256 88c142557820ccad55bb59756bfcfcf891de9cc6202816bd346445188a0ed342, size 988,097,824. Python 3.13.12, torch 2.10.0+cu128, Transformers 4.57.1, tokenizers 0.22.1 and RTX 3060 12 GiB are recorded in the native evidence. No weights are in Git.
The real-weight initialization and trained checkpoint reloaded successfully. A fresh merged-main CUDA probe on the two-step trained checkpoint SHA-256 b6e0db32a65bbf240a5cdec8c938040054a45284e5d4cf35cd310160262b6efb passed the unchanged independent Transformers hidden-state tolerance at 62 input tokens, with maximum absolute difference 0.00011444091796875. Its raw local receipt SHA-256 is c3ff436ac722f9e5d49a76d404047f7f273888c601a81d41a012c1b2fa84bc12. The failed initial CUDA and trained-oracle receipts remain published alongside the corrected source-bound evidence. Bounded FP32 LoRA with activation checkpointing resumed from step 1 to step 2 and served locally; peak allocated training memory was 4,018,332,672 bytes.
The first two original boxes are complete. Keep #3 OPEN: full-weight training, BF16 where supported, broader gradient/branch-isolation/shared-prefix numerical coverage and a declared shape/resource matrix are outstanding. Two synthetic optimizer steps and a 62-token inference request do not establish trained long context, representative quality or Jev parity.
Activity
CompleteDotTech commentedon Sep 28, 2026
Real-weight mixed-question numerical diagnostic — 2026-09-27/28
An opt-in local CUDA probe loaded the actual two-step pinned-Qwen LoRA checkpoint SHA-256
b6e0db32a65bbf240a5cdec8c938040054a45284e5d4cf35cd310160262b6efband exercised three mixed Choice/Score/Noul branches, 201 logical tokens, on RTX 3060 with torch 2.10.0+cu128. The model source SHA-256 wasb2207f6c7ed04146b0d47874baaf0a84c6d31b8d2ce9e6391c98205bc67c3f4e. The raw private failure receipt SHA-256 isd0c8819ab08456a0625ebc8494c005a3398ead1fc5abc0d30a2c32aa14d6ad65, and the probe source SHA-256 is0c598d9bdb992b492a5f45bedd15a3e648bf055d6493c376598314db7416f7cc.At the unchanged fixture tolerances (
atol=rtol=1e-5logits,2e-5gradients), batched versus full-row reference failed: maximum absolute logit difference0.00021648406982421875, gradient difference0.028961181640625. Altering only the Score instructions left the other two branch outputs bitwise equal, so branch isolation passed. A separate eval-only diagnostic localized logit drift to both batching (batched versus serial cached maximum0.00016880035400390625) and shared-prefix recomputation (serial cached versus full-row maximum0.00014209747314453125). These are observations of the current floating-point paths, not a proof of a specific low-level kernel cause. No tolerance was changed and no numerical pass is claimed.PR #26 now also records a successful pinned-weight BF16 LoRA CUDA two-step resume and rank-0 full-weight CPU two-step resume, plus the explicit full-weight CUDA memory limit on the tested 12 GiB GPU. Keep the numerical acceptance and issue #3 OPEN until this real-weight discrepancy is resolved or a scientifically justified separately declared native tolerance is accepted through normal review.
CompleteDotTech commentedon Sep 28, 2026
Protected BF16/full-weight evidence delivery — 2026-09-27/28
PR #26 merged normally at
8c722fd87a8ec4049adab018aa90417a539defd9after all five required PR checks passed. Source commits2f76a08d72143d1e6bd3e41a4cef16ed5665df52andb1285338ef0391d9d99ae9d89a3ca4879366aa2aare verified SSH-signed. All five post-merge main jobs passed. The primary checkout was fast-forwarded to main, clean at remote divergence 0/0.The published native receipt binds actual pinned Qwen weights to a two-step BF16 LoRA/activation-checkpointed CUDA resume (final checkpoint SHA-256
dd996659ff2453bd2995258b71a40c5ab8d4fbc42e0d73a32d901bbca1559ec5) and a two-step rank-0 full-weight FP32 CPU resume (final checkpoint SHA-256f5bc3d7a645cbddc6f56e2cfeeecc7b035bd9667c21ab16585927de795ced27c). Both passed lineage/exposure verification and short independent pinned-Transformers inference comparisons at the existing combined tolerance. Synthetic training reached only 69 maximum branch and 144 aggregate tokens. The tested full-weight CUDA train/resume profile remained memory-limited at an explicit 8.64 GiB cap on RTX 3060; no unmerged optimizer diagnostic was claimed as a source fix.The separate actual-weight mixed-question gradient/logit probe remains a failure at the unchanged strict fixture tolerance, recorded in the prior comment. Branch isolation passed, but batching and shared-prefix recomputation both contribute measured logit drift. Thus the third/fourth original acceptance boxes remain unchecked. Broader declared shapes/resource comparisons, full native numerical resolution, trained long context, representative quality, and Jev comparison remain open. Do not close #3 from the passing short probes or green CI alone.
CompleteDotTech commentedon Sep 28, 2026
Native actual-weight numerical localization on the existing two-step Qwen LoRA checkpoint, with the fixed tolerances unchanged: FP32 batched vs serial-cached max logit difference was
0.0001688004; serial-cached vs independent full-row was0.0001420975; batched vs full-row was0.0002164841(maximum tolerance ratio 4.81). Intermediate backbone hidden states already diverged by up to0.0007515; switching only the batched pointer-head arithmetic to the canonical per-branch readout changed logits by at most0.000003815.CUBLAS_WORKSPACE_CONFIGplus deterministic algorithms left the FP32 result byte-identical. An FP64 diagnostic made batched and serial-cached outputs identical and full-row differed by at most0.000001907; that is diagnostic evidence of shape-sensitive FP32 CUDA arithmetic, not an FP32 acceptance pass or a production change.No source, tolerance, checkpoint or run identity was modified. The original strict logits/gradients gate remains failed and #3 remains open. Private localization receipt SHA-256:
9b20a8a8ac248896b2bd27fdcc3ffb2ec8ee927bdc421739406e49963816c072.CompleteDotTech commentedon Sep 28, 2026
Read-only actual-checkpoint FP32 localization on the RTX 3060: checkpoint SHA-256
b6e0db32a65bbf240a5cdec8c938040054a45284e5d4cf35cd310160262b6efb; current-mainmodel.pyblobc92b3731ccde5968a8943d8c5dbd37c971b5bc40. Private hook probe/JSON SHA-256:d3aed13a58cedbeb3fc76ac5a49809ddbb3b2572f71b7c61ac0ded04ff31c6b51/fa16cc1bd6efdb1d05a8d504089f7abb8f1c261097580401f3df1681a3ddad3d.On the same real unpadded suffix tokens, embedding and layer-0 input norm matched bitwise across batched, serial-cached and independent full-row paths. The first observed difference was layer-0 Q projection: maximum absolute 7.62939453125e-6 in both comparisons. Layer-0 K projection also differed by that amount for batched versus serial. The difference grew through later operations (final norm max 6.0081e-4 batched/serial, 7.51495e-4 serial/full). This localizes the first difference to projection/GEMM on equal inputs; it does not establish the sole cause or satisfy the unchanged logit/gradient tolerances. No source or precision policy was changed. A targeted projection arithmetic/backward and cost test is next; #3 remains open.
CompleteDotTech commentedon Sep 28, 2026
A follow-up actual-checkpoint diagnostic temporarily changed projection arithmetic in memory, keeping FP32 outputs and the existing 1e-5 absolute/relative logit criterion. Script/JSON SHA-256:
292c735f830577fc7e372ac4a5845af27ab26901e11d73abb56b7e244947b286/5c3cca84bfd4f88c7155b04f87dbe392467c195bb869de5066d85eb92499e6b5. No repository source or checkpoint changed.On the same three-question input, baseline maximum tolerance ratios were 3.751 batched/serial, 2.942 serial/full, 4.811 batched/full. Double accumulation in Q projection only gave 0.930, 0.922, 1.040; Q/K/V projections gave 2.780, 1.545, 3.277. Double accumulation in all backbone linear modules with FP32 outputs gave 0.0555, 0, 0.0555. That is forward evidence on one input only. Backward/gradient parity, latency, memory across cases and other shapes remain unmeasured; double arithmetic may be impractical. #3 remains open under unchanged acceptance tolerances.
CompleteDotTech commentedon Sep 28, 2026
Actual trained-checkpoint backward diagnostic, private script/JSON SHA-256
7b5817a0a933648e7b78961be8545f39a53d7a21b1a3e834942e0fb5b11dbc42/d031b36931f0fa37ef0498c9eff79ce0f36f1d76688e9e9d9b9961f3a3d3e94. The first non-checkpointed double-autograd attempt hit the explicit 70% GPU memory cap. Retrying through the checkpoint's existing activation-checkpointing path completed without source or weight changes.On one synthetic three-question request, double accumulation for all backbone linear modules with FP32 outputs met unchanged tolerances: max logit ratios 0.0555/0/0.0555 and worst of 100 trainable gradient ratios 0.3134/0.2698/0.3624 (batched/serial, serial/full, batched/full). Baseline failed both. Single forward+backward timings rose from 0.5–0.8 seconds baseline to 4.3–5.0 seconds double accumulation. This is a promising numerical diagnostic but too costly and narrow to call a source fix or native acceptance. Further shape/mode, performance and narrower-arithmetic tests remain; #3 stays open.
CompleteDotTech commentedon Sep 28, 2026
Narrower actual-checkpoint projection diagnostic: Q+MLP, Q/K+MLP, Q/K/V+MLP and MLP-only higher-precision accumulation each still failed at least one unchanged strict forward logit comparison on the three-question input. Full attention projections plus MLP reproduced the prior pass. Private script/JSON SHA-256:
1b50442cbde7db878714a7222f7461f7e97d12a736ed835ff3b2542853bf7465/3fb17c3d27ea9f39862e71ccb1545169c6881353cc9b90ee966bc277d75c1a40. This rules out these simple subsets for this input, not other FP32 algorithms. No repository source or tolerance changed; #3 remains open.56 remaining items
CompleteDotTech commentedon Sep 29, 2026
Source-free native diagnostic on clean
5fedf05with the pinned trained checkpoint found a passing long BF16 backward candidate on the local RTX 3060 under the unchanged 70% process memory cap. It preserves the grouped fused BF16 attention forward, accumulates cached prefix key/value gradients in double precision, and uses a query-tiled causal Q/K/V VJP plus a shape-stable LoRA backward.For one 10,520-state-token/two-question actual-weight row, batched cached, serial cached, and full-row logits/probabilities matched exactly. All 100 trainable gradient tensors passed the frozen 2e-5 tolerance for all three pairwise comparisons; batched/serial gradients were exact, and the worst serial/full ratio was 0.01742 (absolute 7.63e-6). Peak allocated GPU memory was 4.57/4.50/4.62 GB for batched/serial/full. The local script and receipt SHA-256s are
8e169001ae3182cacc5f35508bf1891f906f206fb9b5a8047b93cf0a7a5a8e66andddbc4e25a42f29ae085da4fddc159339e8ac3fc2f0b47b0c1d929c35499656da.This is a source-free diagnostic, not a merged production repair. Exact-limit whole-model training, FP32, resume identity, serving, and the rest of the native matrix remain unproven. Acceptance stays open.
CompleteDotTech commentedon Sep 29, 2026
PR #51 delivered a measured long BF16 gradient repair, but it does not satisfy the complete native acceptance for this issue. FP32 long gradients, short BF16 post-change gradients, exact-limit whole-model training/resume, serving, and the full applicable native matrix remain open. I restored the issue state after the merge-linked automatic closure.
CompleteDotTech commentedon Sep 29, 2026
PR #52 merged as GitHub-verified
886a1156231c7846776e02d049977c7abed8568e; all five required PR and exact-merge jobs passed. It repairs the short CUDA BF16 training derivative and schedules a single joint objective through branch VJPs. The pinned actual-weight four-question, 287-logical-token comparison has exact batched/serial/full logits and probabilities; all 100 trainable gradients pass the unchanged 2e-5 tolerance (worst cached/full ratio 0.011664). Portable receipt:docs/evidence/native/short-bf16-gradient-20260929.json.This is partial #3 progress. A 331-token branch in the CUDA BF16 fixture exposed a medium-path output mismatch; lengths 257–8191 still use ordinary SDPA. Exact-limit source-free backward resource feasibility passed under the 70% RTX 3060 cap, but production optimizer/resume, full applicable precision/context matrix, calibrated service and #4 chain are not yet proven. The original remaining boxes stay unchecked.
CompleteDotTech commentedon Sep 29, 2026
PR #53 has merged as verified commit
11de27a81e386ad5ee5ecd8aeec3df36363f89d1; all five required PR checks and all five exact-merge post-merge jobs passed.The pinned actual-weight CUDA BF16 trainer completed one optimizer step on a frozen generated 19-question row under the unchanged 70% RTX 3060 process cap. It observed a 32,768-token maximum branch and 65,536 logical aggregate tokens, with recomputed-logit maximum absolute delta 0. The checkpoint SHA-256 is
ad921524f8aadd67ba1c117caada476bd3b3f5364d55ebd8c398f0bf5af7f27e; it reloaded on CUDA andverify-exposurereturnedverified/native_32k_64k=true. Peak PyTorch GPU allocation was 7,254,551,040 bytes. The portable receipt includes source/input/output hashes and limitations.This closes the exact-limit training mechanics/resource gap only. The complete native precision, medium-length BF16, serving and representative quality criteria still need their own evidence. This issue remains open with its original acceptance boxes unchanged.
CompleteDotTech commentedon Sep 29, 2026
PR #54 merged normally as verified commit
9504b6ee37a778e4e8567bc4a0f642ea660a59f3; all five required checks and all five exact-merge post-merge jobs passed.The source-bound pinned actual-weight CUDA BF16 probes now cover a mixed short/medium request (state 13; branch totals 97/583/113/977) and a medium-prefix request (state 613; branch totals 697/1183/713/1577). Batched, serial cached, and independent full-row logits matched exactly in both. All 100 trainable gradients passed the unchanged 2e-5 tolerance in each pairwise comparison; worst tolerance ratios were 0.007794 and 0.009824. The process cap stayed at 70% on the RTX 3060. The portable receipt preserves source/checkpoint/request hashes and private receipt hashes.
The repair uses bounded tiled medium attention, double K/V cache gradients and shape-stable CUDA BF16 RMSNorm. An 8,191-token single-KV-group forward/backward resource probe passed; it is not an 8,191-token full-model parity result. The pinned 62-token FP32 Transformers hidden oracle also passed. Long fused parity, the full native mode/resource matrix, calibrated serving and representative quality remain open; the original acceptance boxes are unchanged.
CompleteDotTech commentedon Sep 29, 2026
Merged-main native follow-up on source
9504b6e: the pinned actual-weight 10,520-state-token, two-question BF16 batched/serial/full comparison passed exact logits/probabilities and all 100 gradients at the unchanged 2e-5 tolerance (receipt SHA-2568c25d4338cc3552e6c8cc7dff2f599a039b631ddac56e516916b0b3286fba15f). The same request in FP32 passed logits/probabilities but failed gradients: 14/100 batched/serial, 11/100 serial/full, 10/100 batched/full; worst tolerance ratios 21.59, 7.98, 13.60 respectively (receipt SHA-256c9187c25112b835a8a1b6fd097c64bb8d1b9ea776c945932eb092e040fdd4b65). Both runs used the pinned checkpoint SHA-25699f797ffd0521adbee964d130d73eb381c9b6bd3b7896ce61b5007886d19ccf9and unchanged 70% RTX 3060 process cap. This is generated-data numerical evidence, not representative quality. FP32 long gradients remain a concrete blocker; issue stays open. Overwatch remains archived and untouched.CompleteDotTech commentedon Sep 30, 2026
Verified numerical/source delivery — PR #56
PR #56 merged at
4f31cc76e38ef9848950e1667875571f8e190903, from verified SSH-signed source4ffd9f1f1263d225e4f130f174cf6070c635d235. All five required PR checks and all five post-merge checks passed; the primary checkout synchronized cleanly. The merged-source CUDA graph ownership regression also passed.The source delivery repairs canonical FP32 dispatch/RoPE/shared derivatives, accounts for materialized double GQA storage and versions the CUDA cache planner, and releases completed BF16 branch graph owners before optimizer allocation. It provides opt-in public HF/seam reproducers and source-bound portable receipts.
Actual pinned trained weights passed the short independent HF oracle (96 Q/V adapter gradients, exact hidden states), four short/seam cases, 8191/8192 prefix boundaries and a 10520-token state. Each generated native comparison checked all 100 trainable LoRA/head gradients across batched/serial/full modes at unchanged tolerances. Public HF and four short/seam commands were separately executed successfully. These are LoRA numerical checks, not full-weight gradient parity.
Noncheckpointed rank-8 FP32 and BF16 each passed two-step training, same-run resume and finite mixed-output reloads. Independently initialized pointer heads mean these cells are not a matched precision comparison. See the rank-8 receipt.
Keep #3 open. Rank-0 FP32 resume OOMs during backward. The graph repair allows the exact same-init BF16 first optimizer step and reload, but resumed backward still OOMs at the unchanged 70% cap. An expandable-segments experiment failed because this Windows runtime reports it unsupported. The failures and artifact/attempt chains are preserved in the scoped full-weight evidence. Exact-limit FP32 training also OOMs during forward. These observations do not establish universal hardware unsupported status, complete native mode/serving acceptance, representative quality or Jev benefit.
Final CPU suite: 858 passed, 55 explicit skips; source integrity, critical lint, compilation, the original 45-feature/450-cell mapping, wheel runtime/license bytes and pruned source archive inventory passed. Historical numerical and rank-8 receipts retain their original source hashes; the graph cleanup is separately attributed. Overwatch remains archived by user direction.
CompleteDotTech commentedon Sep 30, 2026
CLI checkpoint lifetime repair — September 30, 2026
PR #59 merged normally at verified merge ab9f200; verified signed source ead3850. All five exact-head required checks passed in run36668771022 and all five post-merge checks passed in run36669030675. Primary main is clean and synchronized0/0.
The CLI now retains the parent digest separately and releases its unused deserialized checkpoint payload before train/resume. Four CPU weak-reference regressions prove payload collection and unchanged init/resume arguments for train and reward-train. Full CPU suite862passed/55explicit skips; compile/indexedsource/criticallint/lock/matrix/CPU numerical/localresume gates passed.
This corrects an observed object-lifetime cause of extra checkpoint retention. Actual full-checkpoint RSS savings and native full-weight feasibility remain unmeasured. Existing native default-backend OOM results remain failed. Explicit CPU-state streaming integration and its actual full-weight numerical qualification are still pending; latest host availability fails the fixed12/16GiB admission floors. No workload eviction, cap increase or floor reduction was used. Keep #3 open.
CompleteDotTech commentedon Sep 30, 2026
Partial delivery: PR #60 adds the explicit versioned CPU-state AdamW backend and hash-bound synthetic CUDA qualification receipts.
b5d8299b768564c09c541e3dfc04b3d28eef2fd6; signed evidence:dfdc4c12db27fff93a16976bc882d6379578e68c; verified signed merge:20474a0d600125bc8fe23bc7463974f9280b6987.36672911774; all five post-merge checks passed on exact merge20474a0in run36673746553. Primarymainis clean and synchronized (divergence0 0).Actual pretrained fullweight optimization/resume and captured-gradient stock optimizer equivalence remain pending under the unchanged 16 GiB available-host requirement. Independent rank0 derivative readout is separately prepared against the clean merged 33-file source, preserving its historical checkpoint birth, with a 12 GiB host requirement. Latest admission available host RAM was 6764613632 bytes; neither probe started. Existing default backend OOM receipts remain failures. No native derivative/context/representative-quality/Jev claim or acceptance checkbox change is made; issue #3 remains open. Overwatch remains archived per user instruction.
CompleteDotTech commentedon Oct 3, 2026
PR #62 merged at verified commit 5f8b70e after all five required checks passed. Local main is synchronized. The portable trained rank0 FP32 derivative failure is preserved in docs/evidence/native/rank0-derivatives-failed-20261002.{json,md}; worker/supervisor hashes and all five comparison counts were checked against the original artifacts.
Only batched/serial passes. HF backbone fails 132/290 gradients; HF task fails 194/294; serial/full and batched/full each fail 22/294. Correct inventory is 294 tensors / 494,090,176 scalars. Original incorrect-inventory failure remains preserved. No tolerances or acceptance flags changed, and this derivative probe constructed no optimizer.
Local validation: 875 passed, 62 skipped, required compileall, source audit and feature-matrix verification passed. Postmerge CI run 37087435476 passed all five jobs on merged commit 5f8b70e. This is partial evidence delivery: keep #3 open; derivative acceptance, full optimizer resume/precision qualification and downstream acceptance are unfinished.
CompleteDotTech commentedon Oct 3, 2026
The partial actual full-weight FP32 optimizer evidence is merged in PR #63 at verified commit 69dab1d. All five required checks passed before normal protected merge; local main is synchronized cleanly. Postmerge run 37091096520 passed all five jobs on merged commit 69dab1d.
One actual pretrained rank0 update saved a reloadable checkpoint with 294 CPU optimizer states / 3,952,722,584 bytes. Separate stock CUDA AdamW comparison found exact equality of all updated weights, moments and step counters. Before portable publication, the saved checkpoint hash and all 294 expected reference-shard hashes were rechecked. See docs/evidence/optimizer/fullweight-fp32-step1-partial-20261002.{json,md}.
The original oracle's comparison phase left the 16 GiB host floor unchecked and its end value fell below that floor. The arithmetic result is therefore separate from complete budget compliance. Subsequent guarded failures remain preserved; no floor/tolerance was lowered and the completed training step was not repeated. Worker numerical_oracle_status=not_run remains intact alongside its later oracle receipt.
Local validation: 875 passed, 62 skipped, compileall, feature matrix and source snapshot/index audits passed. Keep #3 OPEN: resume/BF16, full resource qualification and native shared-prefix/derivative acceptance remain unfinished. No representative quality, trained context or Jev claim.
CompleteDotTech commentedon Oct 3, 2026
PR #64 merged as
8ed5e2aa9aa8cdc9f1cd5446d8a8ab86ceadc795with a verified signature after all five required PR checks passed. Local main is clean/synchronized; local validation was 875 passed, 62 skipped plus compilation and preserved matrix verification.Portable rejected-candidate evidence: summary, receipt.
Cached/full gradient failures remain 38/294 for rounded attention products, 21/294 for per-call FP64 projection parameter products, and 27/294 for branch-owned prefix VJPs. Batched/serial comparisons pass; independent-reference failures remain. Module input hooks include residual contributions; the corrected normalization-only replay is documented. No model code, acceptance flag or numerical criterion changed, and no native/resume/BF16/quality completion is claimed. Issue #3 remains open.
Post-merge workflow 37094947716 passed all five jobs on the exact merge commit.
CompleteDotTech commentedon Oct 3, 2026
Offline acceptance audit executed unchanged main
8ed5e2aa9aa8cdc9f1cd5446d8a8ab86ceadc795in a supported CPython 3.13.16 / torch 2.10.0+cpu environment. Full suite: 882 passed, 55 explicit skips for native SDK/CUDA/network/resource profiles. Focused publication/native-VJP checks: 24 passed, nine CUDA skips. Compilation, source audit, 45-feature by 10-configuration matrix, and 24-configuration CPU numerics passed. Independent tiny random HF Qwen2 full/shared-prefix gradient checks passed across three seeds and 26 parameters at unchanged tolerances. These are CPU/synthetic numerical checks, not Jev/provider quality or CUDA qualification.No new causal defect was reproduced, so no speculative patch or generated weights were published. Initial unsupported Python 3.14/missing dependencies, CUDA resolver selection and CPU-index resolution attempts are retained separately from the successful supported environment. The unchanged CPU dependency lock was used; no new resolved lock was fabricated.
Original native device/context/quality and authentic agentic host/provider/telemetry gates remain open. No training, paid provider request, telemetry publication, resource acquisition, workload restart or production activation occurred. Existing v6/v8 receipts and unknown unsaved updates are retained; expired workload authority was not revived.