[Core] Parallelize OCI object storage model verification (MD5)

[fredericsun]

What this PR does

• Parallelizes OCI model file integrity checks across model files.
• Adds configurable --model-verification-concurrency.
• Enforces one pod-wide concurrency limit shared by all model tasks.
• Keeps the default at 1, preserving serial verification for a single model.
• Preserves existing MD5/size mismatch handling and verification metrics.
• Adds tests for shared concurrency limits and validation failures.

Why we need it

Final MD5 verification is a significant bottleneck for large, sharded models because files were previously validated sequentially.

Parallel verification significantly reduces this phase. The pod-wide limiter prevents simultaneous model tasks from multiplying CPU and NVMe read pressure beyond the configured limit.

With the default value of 1, concurrent model tasks also share one verification slot. Deployments must explicitly configure a higher value, such as 8, to enable the optimization.

Validation

Validated with gpt-oss-120b (195.8 GB) and a pod-wide verification concurrency of 8. Across 72 downloads, single-model final verification decreased from the serial baseline of ~324–329s to ~113–117s, while two concurrent models averaged ~125–127s per model. All 72 MD5 verifications passed, with no checksum mismatch, model-agent restart, or probe failure observed

How to test

go test ./pkg/modelagent ./cmd/model-agent
helm lint charts/ome-resources

The test verify that:

• The default concurrency is 1.
• Invalid concurrency values fall back to 1.
• Multiple model tasks share the same pod-wide limiter.
• MD5/size mismatches and validation errors are still reported.
• The configured concurrency limit is never exceeded.

Checklist

• Tests added/updated (if applicable)
• Docs updated (if applicable)
• make test passes locally

Summary by CodeRabbit

• New Features

  • Added configurable concurrency for model-file integrity verification.
  • Verification concurrency can be set through deployment values or a command-line option, defaulting to one concurrent check.
  • Shared limits now apply across simultaneous model verifications.

• Bug Fixes

  • Improved reporting of file-validation mismatches and verification errors.

• Documentation

  • Clarified hardware pool terminology in scoring documentation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6eaa2cc4-151d-4398-86f4-be331246d10c

📥 Commits

Reviewing files that changed from the base of the PR and between e3857d7 and 86560f5.

📒 Files selected for processing (1)

• pkg/alfred/policy/defrag/scoring.go

Included review availability: Your plan provides up to 2 included reviews per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The model-agent adds a configurable pod-wide limit for concurrent OCI model file integrity checks. CLI and Helm settings pass the value to Gopher, which uses a shared limiter and concurrent verification workers.

Changes

Model verification concurrency

Layer / File(s)	Summary
Concurrent verification and shared limiter pkg/modelagent/gopher.go, pkg/modelagent/verification_limiter.go, pkg/modelagent/gopher_test.go	Gopher uses a shared limiter and worker pool for model file verification. Values below one are clamped to one. Tests cover limiter sharing, concurrency bounds, mismatches, and validation errors.
Configuration and constructor wiring cmd/model-agent/main.go, cmd/model-agent/main_test.go, charts/ome-resources/values.yaml, charts/ome-resources/templates/model-agent-daemonset/daemonset.yaml	The CLI and Helm chart define modelVerificationConcurrency with a default of 1. The value is passed to Gopher through WithModelVerificationConcurrency.
GPU pool score terminology pkg/alfred/policy/defrag/scoring.go	The PoolScore comment identifies Node.GPUPool as the scored hardware pool.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to 86560

The PR adds configurable parallel model verification while preserving serial defaults and existing mismatch handling; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: slin1237

Sequence Diagram(s)

sequenceDiagram
  participant Gopher
  participant VerificationWorkers
  participant verificationLimiter
  participant Validator
  Gopher->>VerificationWorkers: verifyDownloadedFiles
  VerificationWorkers->>verificationLimiter: acquire permit
  verificationLimiter-->>VerificationWorkers: permit granted
  VerificationWorkers->>Validator: validate model file
  Validator-->>VerificationWorkers: validation result
  VerificationWorkers->>verificationLimiter: release permit
  VerificationWorkers-->>Gopher: validation errors and metrics

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 6 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: parallelized OCI object storage model verification with MD5 checks.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch model-verification-improvement

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

⚠️ Pre-commit checks failed

Please run the following locally and commit the fixes:

pre-commit run --all-files
git add -u && git commit

See CONTRIBUTING.md for setup instructions.