Add Python-configured independent worker deployments

[kailash109]

Model deployments were tied to a manually maintained catalog. This adds Python infrastructure configs and independently deployed trainer/inference worker apps, while keeping one Tinker-compatible frontend for all models.

Configs export a Deployment composed from typed Model, Compute, Trainer, and Inference dataclasses. Variants use ordinary dataclasses.replace. Unknown orchestration fields are rejected; backend tuning stays in explicit dictionaries. There is no custom inheritance parser or dotted override syntax.

Backend settings are resolved once and stored with the deployment. Worker launchers consume those settings. SGLang constructs ServerArgs directly; Miles-specific argparse handling stays inside its adapter. Megatron constructors share one ownership rule: extra options may add fields, but cannot replace managed fields. This prevents optimizer keyword collisions and removes attribute-patching override loops.

Compute, trainer timeout, inference startup timeout, and replica scaling are wired through. Unsupported trainer minimum instances and generic inference timeout fields are rejected. The merged multi-node Miles launcher is integrated through Compute.nodes, including a two-node Qwen3.8-27B 256K example.

Worker hashes track settings and explicit per-role code releases, without a whole-source fingerprint. Unchanged worker apps are reused, and existing jobs retain their recorded workers. Frontend selection belongs to the deployment command, not model configs. Earlier draft manifests require migration or a fresh registry; automatic old-worker cleanup is not included.

Validation: 651 CPU tests passed, 1 skipped. Coverage includes config typos, duplicate backend settings, all 15 examples, compute propagation, direct SGLang construction, saved records, update isolation/retries, and multi-node launcher wiring. Ruff and whitespace checks pass. No redeployment or GPU validation was performed for this revision. Historical live results in docs/deployment-validation.md are explicitly identified as earlier revisions.

See docs/deployment-configs.md for the configuration-to-launch code path and examples.

[micahtyong]

Some design considerations:

1. Agreed with folks during standup today, let's not use YAML and push for config-as-code. At Applied, we tried using YAML and people would get confused around what configs were implicit via "base" YAMLs vs what they needed to explicitly set. Like Sabri said during the Engram meeting, managing configs as code files isn't necessarily a bad thing. But I think this PR is a step in the right direction, especially if we use a data-classes. This is also ultimately what we ended up moving towards at Applied. Clients could define specs like this:

# src/lilo/presets/qwen35.py
QWEN35_9B_LORA_16K = DeploymentSpec(
    name="qwen35-9b-lora-16k",
    model=Model(id="Qwen/Qwen3.5-9B", revision=..., max_context_length=16_384),
    trainer=Trainer(backend="miles", resource=H100x4, options=MilesLoRA(rank=32, ...)),
    inference=Inference(resource=H200x1, replicas=Scaling(max=8), sglang=SGLang(tp=1, ...)),
)

and if we really want to avoid duplicating code, can do something like

QWEN35_9B_LORA_64K = QWEN35_9B_LORA_16K.model_copy(update={"model": ..., "trainer": ...})

2. I'm curious how the hashing mechanism works, and importantly how it will play into the long-term solution for "validating model configs". With implementation_fingerprint, any change to lilo source code requires updating all the miles config definitions. I wanted to check, is this intentional? I suppose this is saying that every Lilo code-change requires revalidating our configs? From chatting with Andrew, I believe our training-gym does something similar so I wonder if there's any logic we can reuse there
3. Do we want to have a process for deprecating old model configs?

My review agent flagged a few other implementation details like

Single startup failure takes a model offline for all tenants. run_trainer.failed writes deployment_failure:; _spawn_engine and the reconciler then refuse to launch any trainer for that definition until an operator runs lilo deployment retry. A transient (HF hiccup, NCCL init flake, single bad host) becomes a manual-intervention outage on a shared frontend. No auto-expiry or retry budget.

Manifest grows monotonically and lives in an env var. Retained inactive generations are never pruned ("not implemented"), and the entire manifest JSON is baked into the image env and read on every import (manifest_from_env(), also in pool_environment). Each row is ~2-3 KB; you'll hit env-size limits (128 KiB per var on Linux) somewhere in the tens of generations, and every frontend/pool container imports and re-validates all of them.

but these can come after we align on the config design / validation story