Persist Triton/Inductor kernel caches on a shared lilo-kernel-cache Volume

[kevintli]

Summary

Every lilo trainer is single_use_containers=True with min_containers=0, so each new trainer (autoscale, replacement, scoped run) cold-compiles every Triton/FLA kernel and Inductor graph — with TORCHINDUCTOR_COMPILE_THREADS=1 this was observed at ~30 min of startup. Mirrors modal-projects/stitch#251: point both compiler caches at a shared Volume v2 so kernels compile once per (arch × torch/triton version × source) and are reused by every subsequent container.

# providers/modal/kernel_cache.py
KERNEL_CACHE_ROOT = "/root/.cache/kernel-cache"
KERNEL_CACHE_ENV = {"TRITON_CACHE_DIR": f"{ROOT}/triton", "TORCHINDUCTOR_CACHE_DIR": f"{ROOT}/inductor"}
kernel_cache_volume = modal.Volume.from_name("lilo-kernel-cache", create_if_missing=True, version=2)

• All 14 definitions/*.py: TRAINER_VOLUMES[KERNEL_CACHE_ROOT] = kernel_cache_volume, and every trainer decorator now passes env=trainer_deployment_env() (6 of them previously passed no env=), which now always includes KERNEL_CACHE_ENV. Function-level env (rather than backend_env) is deliberate: it's inherited by torchrun ranks, by ray start on clustered worker nodes (which never run run_engine_with_backend), and by Ray actors, so no changes to _WORKER_ENV_VARS.
• Scoped trainer in scoped.py: same mount + env (covers custom engine.trainer_image).
• No explicit commit(); relies on Volume v2 background commits, as in stitch.

Operational notes: cache keys hash triton/ptxas/torch versions + arch + source, so runtime bumps are misses (no stale reuse) but the volume grows monotonically (~0.3 MB/Triton kernel; the equivalent stitch miles volume is 1.5 GB / 40k files) — prune manually. If an entry is ever bad, modal volume rm -r lilo-kernel-cache and trainers fall back to a cold compile.

Tests: tests/providers/test_kernel_cache.py asserts every definition and the scoped trainer mount the volume and set the env; test_modal_deployment.py updated for trainer_deployment_env() now including the cache env.

Benchmark

Pending — baseline (main) vs. cold (empty volume) vs. warm (separate container, populated volume) on qwen3_5_4b_full_64k in kevinli-lilo; numbers to follow.

Link to Devin session: https://modal.devinenterprise.com/sessions/467320372a264381b9c015c4dd9bf804
Open in Devin Desktop: https://modal.devinenterprise.com/desktop/session/467320372a264381b9c015c4dd9bf804?variant=devin
Requested by: @kevintli

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

Superseded by #65, which takes the same approach (shared v2 Volume at /root/.cache/kernel-cache, TRITON_CACHE_DIR/TORCHINDUCTOR_CACHE_DIR via backend_env, forwarded to Ray workers via _WORKER_ENV_VARS).

[kevintli]

#65 will handle this