Add native AutoML search and GEPA prompt optimization

[ramanathan831]

DO NOT MERGE independently. Companion to #277: review, validate and land the required changes as a coordinated change. Merge #277 first only when both changes and their dependencies are ready; do not merge it ahead of AutoML readiness.

Stack and review scope

This draft is based directly on #277's current head (e4481fc3bc18994ff63b6ab62e9eb331d5e77e1d), not on the broad workflow/skills migration. Its GitHub base is main because #277's source branch exists only in the contributor fork. Until #277 lands, this PR's Files changed also includes the parent change.

AutoML-only comparison: #277 head to this branch.

Supersedes the fork-only AutoML draft #4. The broader fork runtime/skills PRs are not merge prerequisites for this focused change.

Included

• Native Framework SFT adapter and one cosmos-automl CLI/skill, including random, Bayesian (GP/expected improvement), BFBO, Hyperband, Hyperband-ES, BOHB, ASHA, DEHB, PBT, LLM, hybrid and autoresearch; GEPA prompt optimization is a separate evaluation workflow under the same entrypoint.
• Native reasoner evaluation, decoder and provenance-checked DCP/LoRA export dependencies required for trial scoring. Training/checkpoint restoration builds on Add reproducible Cosmos3 Nano and Edge fine-tuning #277.
• Only the shared SLURM job-record/redaction/schema, Pyxis template and NCCL probe needed by AutoML. No annotation, serving, quantization, dataset-conversion or other-platform skill migration.
• Optional reasoner-eval dependencies instead of the broad workflow extra; Apache attribution/provenance retained only for included material. Consumers require no TAO checkout or SDK.
• Preserved v1 development package identity (nvidia-automl-core==1.0.0.dev0), explicit image/wheel identities, bounded controller/recovery policies and truthful validation limits.

Validation

• Focused branch: 14 existing Framework adapter tests and 103 existing shared-helper checks passed.
• All 32 retained AutoML/evaluation/checkpoint/inference modules passed Ruff lint/format. Skill validation, local documentation links/import closure and whitespace checks passed. All 41 retained Python files have unchanged syntax trees versus the validated broader stack. The unchanged core's 28 existing tests also passed. No new unit tests added.
• The fresh focused image passed actual parallel SLURM regression across all 12 SFT modes plus GEPA: 37 SFT stages + 4 GEPA evaluation batches, all 41 GPU allocations native COMPLETED, exit 0:0. Same WTS 64-training/16-validation subset, at most five cumulative optimizer steps, eight GPUs per SFT stage and one GPU per GEPA batch. See the scheduler reconciliation below; this was not a clean first-attempt matrix pass.

Before review/merge

• Keep draft and coordinated merge hold with Add reproducible Cosmos3 Nano and Edge fine-tuning #277.
• Independent non-author validation remains required. These completed smokes do not establish model quality, production convergence, live fault recovery or Hyperband-ES curve pruning. Address the pre-existing GEPA progress-publication gap and the remaining ambiguous-submission recovery limitation described below.
• Reusable core source remains in tao-automl/packages/automl-core; local wheels only. Core dependency changes require the repository's TAO Infra review before its core PR/publication can proceed. Do not bypass that gate.
• A distributable core artifact and refreshed dependency lock remain release/integration gates; the unpublished wheel is not silently fetched from a developer checkout.
• Agree the user-facing cookbook/discovery handoff with maintainers and preserve the single documented CLI/skill.

Refactor artifact identity

Focused commit: 14505ff0933dd303fc2793319cb6f8c6e95f2924. Framework wheel SHA256: c600ccb7bcff213fac5d0718eeead0bd1c126c6de0ceecec063bead0340eee34. Core wheel SHA256: ed279d7329de333380b9d649c541b6b75d30d9500ca42f3b3ee8c35d8ee3305f.

Validation image: nvcr.io/nvstaging/tao/cosmos-framework:automl-wts-cu128-focused-20261012, digest sha256:a4dd7851d3af16bcc89deb6729ae1417681ed128158a06f5629ea1550595f2a9. Exact installed-file verification passed (854 Framework files, 19 core package files), including absence of omitted workflow modules. Fresh Python 3.12 CPU controller imports passed without Torch. CPU-only SQSH conversion job 36427241 passed in 27m59s (native COMPLETED, exit 0:0), including in-SQSH package verification. SQSH SHA256: 0f162751b85e480a77d99e937ee7c093d7bb4958ff792814b94c0e2ce7e0293a. All 13 plans passed and parallel controller 36429196 was submitted with 41 sealed input files. Monitoring continued through terminal native accounting and artifact validation. Runtime code remains the exact commit above; follow-up commit 72f7e88 only records the results in documentation.

Completed regression and scheduler reconciliation

All 13 modes produced completed validated handoffs. SFT stage counts: random 1, Bayesian 4, BFBO 2, Hyperband 3, Hyperband-ES 3, BOHB 6, ASHA 3, DEHB 6, PBT 4, LLM 1, hybrid 2, autoresearch 2. GEPA completed four evaluation batches. Bayesian reached its GP/expected-improvement proposal; BOHB and DEHB recorded KDE and differential-evolution proposals. All 37 SFT stages had native training/evaluation success, complete validation records and valid full-state DCPs. Seven promotions/continuations produced 56 matching eight-rank restore receipts, including optimizer-configuration checks for both PBT continuations. All six API calls returned HTTP 200 and reported 5,997 total tokens.

The original matrix controller 36429196 ended FAILED, exit 1:0, after 38m16s: Bayesian, BFBO and PBT's initial sbatch --parsable calls exceeded the unchanged core's 30-second command timeout, leaving unknown outcomes and no job IDs. No blind retry was performed. Repeated queue/accounting checks over more than five minutes on the submission host and two reachable peers found no matching jobs; no worker receipts or native logs existed. A fourth configured login name did not resolve.

Fresh searches under controller 36429728 reused only the ten unspent SFT stages, with staggered starts and the original absolute deadline. It completed natively in 26m11s, exit 0:0. The image/core wheel and total 37-stage GPU budget were unchanged. Final accounting again found no late jobs for the three original names. Original errors and stale ledgers are preserved. This is manual reconciliation, not proof of automatic submission recovery.

All 53 sealed input files retained their hashes, credential sidecars were removed, and no tracked allocations remained running or queued. SFT scores all tied at 8/16; GEPA's best validation score was 8/16. This validates bounded execution, not quality or algorithm superiority.

The final audit also found an existing GEPA monitoring gap: evaluate_prompts calls the evaluator directly and does not emit the evaluation/status.json file declared by its plan. GEPA completion was verified from successful worker exits, metrics and exact per-example IDs, not that missing signal. Progress publication/no-progress monitoring remains a follow-up; this code is unchanged by the focused split.

Full evidence and artifact identities: focused regression results. No new unit tests, version bumps or release publication were introduced.