Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/skills/fine-tuning/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ Train language models with SFT, KTO, and GRPO locally or on supported cloud prov
| Check SFT loss masking (tokenizer only) | `python tuner.py doctor sft-mask --dataset-path <jsonl> --model <tokenizer>` |
| Inspect native training host candidate | `python tuner.py doctor native-training --json` — read-only platform/Python/package metadata; see `reference/native-local-training.md` |
| Diagnose an offline SFT worker rejection | Read the exact retained `trainer.stderr.log`; closed phase/category/milestone labels locate a boundary, not a cause or retry authority; see `reference/sft-training.md` |
| Diagnose Torch import in a retained Docker image | `python scripts/diagnose_docker_torch_import.py --help` for bounded guarded, worker-context prefix, and worker-entry probes; each live command needs exact review; see `reference/sft-training.md` |
| Native MLX prepared SFT candidate | Internal exact-token batch/loss and runtime comparison only; no native launch CLI or API provider yet; see `reference/native-local-training.md` |
| Plan a derived training image | `python scripts/qualify_derived_training_image.py plan --config Trainers/image_profiles/<profile>.yaml` |
| Warm Space scaffold | `python3 Trainers/cloud/scripts/manage_space.py render --template vllm_warm --output-dir /tmp/my-space --base-image ghcr.io/<org>/<image>:<tag>` |
Expand Down
97 changes: 93 additions & 4 deletions .agents/skills/fine-tuning/reference/sft-training.md
Original file line number Diff line number Diff line change
Expand Up @@ -450,10 +450,16 @@ See the checked-in example README for commands and exact fixture coverage.

The checkout-owned `examples/training_qualification/prepare_docker.py` closes
material setup for the public Docker consumer. With an installed engine,
`plan` validates declared release/model/layout, `prepare` authenticates genuine
sources and existing immutable v1 input before publishing canonical material,
and `prepare-model` explicitly downloads the pinned public snapshot into the
private artifact-mount cache. See `examples/training_qualification/README.md`
`plan` validates declared release/model/layout, `prepare-model` first prepares
the pinned public snapshot into a fresh private artifact cache, and `prepare`
then authenticates fresh source and existing immutable v1 input before publishing
canonical material. A private read-only seed cache may reduce downloads only
after pinned upstream metadata and every reused Git/LFS member are verified.
The old seed claim is never adopted. `prepare` binds the complete model receipt;
public start rechecks its bytes before Docker effects. A missing pinned revision
in an otherwise private seed takes the normal cold path; redirects reject.
Cold public reads use the receipt retained with the attempt and do not require
current model weights or the original preparation root. See `examples/training_qualification/README.md`
for the template and exact commands. Full public preflight remains deferred
until host composition; missing image/runtime/model qualification still refuses.
The CPU `probe-filesystem` command measures actual native writer modes and host
Expand Down Expand Up @@ -492,6 +498,89 @@ the consumed claim, roots, container and retained logs; the diagnostics never
authorize replay. See `docs/review/offline-sft-worker-diagnostics.md` for the
October 20 evidence boundary, CPU regressions and fresh-run prerequisites.

For a fresh, credential-free Docker Torch import reproduction, review the exact
command before effects. The checked-in `scripts/diagnose_docker_torch_import.py`
requires `--docker`, an empty `--docker-config`, `--release`, independently
verified `--expected-release-sha256`, `--daemon-id`, and a nonexistent private
`--output-dir`. Guarded mode additionally requires `--source-root`,
`--source-lock`, independently reviewed `--expected-source-lock-sha256`,
`--expected-manifest-sha256`, and `--expected-closure-digest` from the retained
attempt. It validates every staged worker member before executing mounted code;
the result records those commitments. Run from the reviewed checkout so its
trusted closure verifier is the one checked in with the helper.

The helper claims a unique name before Docker create, authenticates the exact
new container before start, and retains it. Docker CLI output is capped at 8 KiB
in memory; the container requests JSON log rotation at 1 MiB with one file.
The import execution has a
120-second host deadline. At expiry the helper issues at most one stop request
to the authenticated new container and separately inspects terminal state;
cleanup takes additional bounded time, and a stop acknowledgment alone is not
quiescence. An uncertain create/start/stop remains consumed in `claim.json` and
`events.jsonl`, with no automatic replay or container removal. The default
probe asks for at most 2 CPUs, 4 GiB, no network/GPU/model mount, and a
read-only root.
Bare mode mounts no source. Guarded mode mounts the independently authenticated
retained source and runs the worker closure check, module guard, trainer path
setup, environment bootstrap and Torch import. Neither runs training. `PASS`
only says Torch imports in the reduced probe; it does not diagnose an earlier
failure. Review the pinned package/runtime source before changing a profile,
image or runtime policy.
The exact r2 CPU evidence and its unresolved runtime difference are in
`docs/review/docker-torch-import-r2-diagnosis.md`.

The guarded-only `--gpu-device 0` option is an import-only comparison that
requires separate exact root review before any live use. It requests only
device 0, verifies Docker retained that device request before start, removes
the CPU probe's visibility suppressors, and records the scope in its claim.
`gpu_used=null` means actual device use was not measured. The same 8-KiB
output budget, 120-second execution deadline, owned cleanup, source
authentication and closed child codes apply. It neither invokes the trainer
nor loads a model; a PASS or failure cannot retrospectively diagnose an
earlier trainer attempt. See the exact proposed command and remaining
environment/user/filesystem gaps in the r2 diagnosis note.

For the worker-context child, keep supplied environment bytes distinct from
the environment observed after isolated Python startup. Require every
supplied key unchanged; the only reviewed addition is
`LC_CTYPE=C.UTF-8`, recorded as a closed delta with separate digests.
Any other addition or change rejects. Do not reset the observed environment
or reinterpret it as a historical capture of the failed r2 child. See the r10
repair and exact unexecuted command in the diagnosis note.

The optional `--worker-context` mode requires independently reviewed
`--expected-workload-sha256`, `--expected-runtime-sha256`, and
`--expected-trainer-sha256` alongside guarded source commitments. It
reconstructs the pinned workload's isolated child environment, launches a
separate `-I` child, loads the authenticated worker closure and guard, and
executes only the authenticated trainer prefix through `import torch`.
The prefix excludes the `UNSLOTH_IMPORT` marker and all later trainer/model
code. It records closed environment, path, file and cwd digests and keeps the
existing claim, output budget, deadline and owned cleanup. Its writable
`/artifacts` is a small private tmpfs with no model weights. Review the exact
workload/source hashes, device scope, host user, caps and filesystem before
each fresh use. A reconstructed child environment is a source-and-plan
deduction; it is not a historical capture of the failed child.

`--worker-context --worker-entry` selects a separate, still import-only
diagnostic. It reconstructs the pinned runtime's trainer argument vector,
including a one-byte sealed stand-in for the prepared dataset descriptor,
then invokes the authenticated worker's real `main()` and nested
`runpy.run_path` trainer entry. A source-bound line trace raises at the
`UNSLOTH_IMPORT` marker after Torch import, while a narrow import-call and
module-finder barrier prevents Unsloth entry even if tracing is disabled or
that module name was cached. A PASS
requires the worker's closed stop label and real trainer `__main__` module
binding. The worker still performs its own argument, closure and import-guard
checks. It records closed environment digests before and after Torch import,
separate from the supplied and interpreter-startup digests. The diagnostic
starts through `-I -c` to install that boundary, so it
does not reproduce the original direct `-I` script startup. The sealed data
byte, reduced filesystem and absent model weights also differ from the failed
attempt. Review and admit a fresh one-use command before any live execution;
do not infer the historical failure cause from either result. See the r12
proposal in the diagnosis note.

After an intentional worker source change, check and refresh existing
manifests with the checked-in maintainers in dependency order:
`regenerate_offline_sft_worker_closure.py`,
Expand Down
1 change: 1 addition & 0 deletions .claude/skills/fine-tuning/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ Train language models with SFT, KTO, and GRPO locally or on supported cloud prov
| Check SFT loss masking (tokenizer only) | `python tuner.py doctor sft-mask --dataset-path <jsonl> --model <tokenizer>` |
| Inspect native training host candidate | `python tuner.py doctor native-training --json` — read-only platform/Python/package metadata; see `reference/native-local-training.md` |
| Diagnose an offline SFT worker rejection | Read the exact retained `trainer.stderr.log`; closed phase/category/milestone labels locate a boundary, not a cause or retry authority; see `reference/sft-training.md` |
| Diagnose Torch import in a retained Docker image | `python scripts/diagnose_docker_torch_import.py --help` for bounded guarded, worker-context prefix, and worker-entry probes; each live command needs exact review; see `reference/sft-training.md` |
| Native MLX prepared SFT candidate | Internal exact-token batch/loss and runtime comparison only; no native launch CLI or API provider yet; see `reference/native-local-training.md` |
| Plan a derived training image | `python scripts/qualify_derived_training_image.py plan --config Trainers/image_profiles/<profile>.yaml` |
| Warm Space scaffold | `python3 Trainers/cloud/scripts/manage_space.py render --template vllm_warm --output-dir /tmp/my-space --base-image ghcr.io/<org>/<image>:<tag>` |
Expand Down
97 changes: 93 additions & 4 deletions .claude/skills/fine-tuning/reference/sft-training.md
Original file line number Diff line number Diff line change
Expand Up @@ -450,10 +450,16 @@ See the checked-in example README for commands and exact fixture coverage.

The checkout-owned `examples/training_qualification/prepare_docker.py` closes
material setup for the public Docker consumer. With an installed engine,
`plan` validates declared release/model/layout, `prepare` authenticates genuine
sources and existing immutable v1 input before publishing canonical material,
and `prepare-model` explicitly downloads the pinned public snapshot into the
private artifact-mount cache. See `examples/training_qualification/README.md`
`plan` validates declared release/model/layout, `prepare-model` first prepares
the pinned public snapshot into a fresh private artifact cache, and `prepare`
then authenticates fresh source and existing immutable v1 input before publishing
canonical material. A private read-only seed cache may reduce downloads only
after pinned upstream metadata and every reused Git/LFS member are verified.
The old seed claim is never adopted. `prepare` binds the complete model receipt;
public start rechecks its bytes before Docker effects. A missing pinned revision
in an otherwise private seed takes the normal cold path; redirects reject.
Cold public reads use the receipt retained with the attempt and do not require
current model weights or the original preparation root. See `examples/training_qualification/README.md`
for the template and exact commands. Full public preflight remains deferred
until host composition; missing image/runtime/model qualification still refuses.
The CPU `probe-filesystem` command measures actual native writer modes and host
Expand Down Expand Up @@ -492,6 +498,89 @@ the consumed claim, roots, container and retained logs; the diagnostics never
authorize replay. See `docs/review/offline-sft-worker-diagnostics.md` for the
October 20 evidence boundary, CPU regressions and fresh-run prerequisites.

For a fresh, credential-free Docker Torch import reproduction, review the exact
command before effects. The checked-in `scripts/diagnose_docker_torch_import.py`
requires `--docker`, an empty `--docker-config`, `--release`, independently
verified `--expected-release-sha256`, `--daemon-id`, and a nonexistent private
`--output-dir`. Guarded mode additionally requires `--source-root`,
`--source-lock`, independently reviewed `--expected-source-lock-sha256`,
`--expected-manifest-sha256`, and `--expected-closure-digest` from the retained
attempt. It validates every staged worker member before executing mounted code;
the result records those commitments. Run from the reviewed checkout so its
trusted closure verifier is the one checked in with the helper.

The helper claims a unique name before Docker create, authenticates the exact
new container before start, and retains it. Docker CLI output is capped at 8 KiB
in memory; the container requests JSON log rotation at 1 MiB with one file.
The import execution has a
120-second host deadline. At expiry the helper issues at most one stop request
to the authenticated new container and separately inspects terminal state;
cleanup takes additional bounded time, and a stop acknowledgment alone is not
quiescence. An uncertain create/start/stop remains consumed in `claim.json` and
`events.jsonl`, with no automatic replay or container removal. The default
probe asks for at most 2 CPUs, 4 GiB, no network/GPU/model mount, and a
read-only root.
Bare mode mounts no source. Guarded mode mounts the independently authenticated
retained source and runs the worker closure check, module guard, trainer path
setup, environment bootstrap and Torch import. Neither runs training. `PASS`
only says Torch imports in the reduced probe; it does not diagnose an earlier
failure. Review the pinned package/runtime source before changing a profile,
image or runtime policy.
The exact r2 CPU evidence and its unresolved runtime difference are in
`docs/review/docker-torch-import-r2-diagnosis.md`.

The guarded-only `--gpu-device 0` option is an import-only comparison that
requires separate exact root review before any live use. It requests only
device 0, verifies Docker retained that device request before start, removes
the CPU probe's visibility suppressors, and records the scope in its claim.
`gpu_used=null` means actual device use was not measured. The same 8-KiB
output budget, 120-second execution deadline, owned cleanup, source
authentication and closed child codes apply. It neither invokes the trainer
nor loads a model; a PASS or failure cannot retrospectively diagnose an
earlier trainer attempt. See the exact proposed command and remaining
environment/user/filesystem gaps in the r2 diagnosis note.

For the worker-context child, keep supplied environment bytes distinct from
the environment observed after isolated Python startup. Require every
supplied key unchanged; the only reviewed addition is
`LC_CTYPE=C.UTF-8`, recorded as a closed delta with separate digests.
Any other addition or change rejects. Do not reset the observed environment
or reinterpret it as a historical capture of the failed r2 child. See the r10
repair and exact unexecuted command in the diagnosis note.

The optional `--worker-context` mode requires independently reviewed
`--expected-workload-sha256`, `--expected-runtime-sha256`, and
`--expected-trainer-sha256` alongside guarded source commitments. It
reconstructs the pinned workload's isolated child environment, launches a
separate `-I` child, loads the authenticated worker closure and guard, and
executes only the authenticated trainer prefix through `import torch`.
The prefix excludes the `UNSLOTH_IMPORT` marker and all later trainer/model
code. It records closed environment, path, file and cwd digests and keeps the
existing claim, output budget, deadline and owned cleanup. Its writable
`/artifacts` is a small private tmpfs with no model weights. Review the exact
workload/source hashes, device scope, host user, caps and filesystem before
each fresh use. A reconstructed child environment is a source-and-plan
deduction; it is not a historical capture of the failed child.

`--worker-context --worker-entry` selects a separate, still import-only
diagnostic. It reconstructs the pinned runtime's trainer argument vector,
including a one-byte sealed stand-in for the prepared dataset descriptor,
then invokes the authenticated worker's real `main()` and nested
`runpy.run_path` trainer entry. A source-bound line trace raises at the
`UNSLOTH_IMPORT` marker after Torch import, while a narrow import-call and
module-finder barrier prevents Unsloth entry even if tracing is disabled or
that module name was cached. A PASS
requires the worker's closed stop label and real trainer `__main__` module
binding. The worker still performs its own argument, closure and import-guard
checks. It records closed environment digests before and after Torch import,
separate from the supplied and interpreter-startup digests. The diagnostic
starts through `-I -c` to install that boundary, so it
does not reproduce the original direct `-I` script startup. The sealed data
byte, reduced filesystem and absent model weights also differ from the failed
attempt. Review and admit a fresh one-use command before any live execution;
do not infer the historical failure cause from either result. See the r12
proposal in the diagnosis note.

After an intentional worker source change, check and refresh existing
manifests with the checked-in maintainers in dependency order:
`regenerate_offline_sft_worker_closure.py`,
Expand Down
1 change: 1 addition & 0 deletions .skills/fine-tuning/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ Train language models with SFT, KTO, and GRPO locally or on supported cloud prov
| Check SFT loss masking (tokenizer only) | `python tuner.py doctor sft-mask --dataset-path <jsonl> --model <tokenizer>` |
| Inspect native training host candidate | `python tuner.py doctor native-training --json` — read-only platform/Python/package metadata; see `reference/native-local-training.md` |
| Diagnose an offline SFT worker rejection | Read the exact retained `trainer.stderr.log`; closed phase/category/milestone labels locate a boundary, not a cause or retry authority; see `reference/sft-training.md` |
| Diagnose Torch import in a retained Docker image | `python scripts/diagnose_docker_torch_import.py --help` for bounded guarded, worker-context prefix, and worker-entry probes; each live command needs exact review; see `reference/sft-training.md` |
| Native MLX prepared SFT candidate | Internal exact-token batch/loss and runtime comparison only; no native launch CLI or API provider yet; see `reference/native-local-training.md` |
| Plan a derived training image | `python scripts/qualify_derived_training_image.py plan --config Trainers/image_profiles/<profile>.yaml` |
| Warm Space scaffold | `python3 Trainers/cloud/scripts/manage_space.py render --template vllm_warm --output-dir /tmp/my-space --base-image ghcr.io/<org>/<image>:<tag>` |
Expand Down
Loading
Loading