scripts: add engine definition smoke test against a deployed control plane

[micahtyong]

Summary

scripts/e2e_engine_definition.py is the deep per-definition E2E: it spins up its own ephemeral app, refuses CATALOG_VISIBLE = False definitions and expects FFT-style constants the Miles LoRA definitions don't have. There was no quick way to check that a freshly redeployed shared lilo actually serves a given definition end to end.

scripts/definition_smoke.py is that check. It runs against an existing control plane (TINKER_BASE_URL / TINKER_API_KEY) and accepts any id from lilo.providers.modal.definitions; default is qwen3_8_27b_miles_lora_64k.

definition = module_for(id)
full : create_full_training_client(service, id)
lora : create_lora_training_client(base_model=id, rank=min(16, MAX_LORA_RANK),
                                   train_attn/mlp/unembed = lora_target_flags(TARGET_MODULES))
forward_backward(cross_entropy) + optim_step
save_weights_and_get_sampling_client() -> sample
forward_backward(importance_sampling on the sample) + optim_step
save_weights_and_get_sampling_client() -> sample
POST /api/v1/unload_model                      # release the trainer / slot

The definition id is passed as base_model, which definition_for in control_plane/http.py matches explicitly, so non-cataloged definitions (27B 64k/128k/256k) are reachable. --definition-id is repeatable, --parallel runs them concurrently, --list prints all definitions. Writes a timestamped JSON report to scripts/results/ (gitignored) and exits non-zero if any definition fails.

Two things learned from live runs that Tinker clients hitting Miles LoRA definitions need to know:

• Miles fixes LoRA targets per deployment, so the model's train_attn/train_mlp/train_unembed must equal what TARGET_MODULES implies ((True, True, False) for the current definitions). The Tinker SDK default train_unembed=True is rejected — and only after the trainer cold-starts. The leaf→flag mapping is extracted from MilesCommandBackend._validate_job into miles_config.lora_target_flags() so the script and backend can't drift.
• importance_sampling datums must have target_tokens == model_input[1:] + [next] over the whole sequence (prompt positions included, with zero advantages); miles_runtime.data validates the shift strictly.

export TINKER_BASE_URL=... TINKER_API_KEY=...
uv run scripts/definition_smoke.py                                   # 27B/64k
uv run scripts/definition_smoke.py --parallel \
    --definition-id qwen3_5_9b_miles_lora_16k --definition-id qwen3_5_4b_full_64k

Verified live against lilo in lilo-deploy at 784dc95: qwen3_5_9b_miles_lora_16k (449s) and qwen3_8_27b_miles_lora_64k (698s) pass all phases.

Link to Devin session: https://modal.devinenterprise.com/sessions/6463b80692584829b19bba8d9bd1d11d
Open in Devin Desktop: https://modal.devinenterprise.com/desktop/session/6463b80692584829b19bba8d9bd1d11d?variant=devin
Requested by: @micahtyong

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.

• Disable automatic comment, CI, and merge conflict monitoring

[micahtyong]

/devin review

[devin-ai-integration]

Starting Devin Review.

[devin-ai-integration]

/devin review

[devin-ai-integration]

Devin Review found 4 potential issues.