Skip to content

Track estimated Jev spend in resumable benchmarks - #47

Merged
CompleteDotTech merged 1 commit into
mainfrom
feat/benchmark-pricing
Oct 17, 2026
Merged

CompleteDotTech merged 1 commit into
mainfrom
feat/benchmark-pricing

Conversation

@CompleteDotTech

Copy link
Copy Markdown
Contributor

Local benchmark runs recorded provider tokens without a dollar total. Add resumable non-email benchmark helpers that show estimated USD spend in live status and reports using an explicitly configured, sourced Jev list rate ($0.042 per million input tokens; output free, checked October 17, 2026).

New attempts snapshot pricing and retain per-attempt costs. Earlier unpriced receipts receive identified retrospective estimates without rewriting the original receipts. Unknown usage and pending attempts remain visible as partial coverage. The helpers preserve the existing run and completed cases, record source-label limitations, and keep datasets, receipts and credentials outside Git.

Add offline pricing/resume checks to CI and document the setup, source provenance, request/token limits and invoice limitations in docs/COSTS.md.

Validation:

  • Fresh npm ci, npm run check and npm run build passed.
  • npm test: 348 passed, zero failures.
  • Offline benchmark runner/resume test passed.
  • Python pricing tests: 5 passed, covering backfill, rounding/retries, missing and pending usage, preserved snapshots and model mismatch.
  • git diff --check passed.

No dataset downloads or paid API calls run in CI. Hosted checks must pass before merge.

@CompleteDotTech
CompleteDotTech merged commit d701480 into main Oct 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant