feat: support the DeepSeek official endpoint - #42
Open
minixalpha wants to merge 3 commits into
Open
minixalpha wants to merge 3 commits into
minixalpha wants to merge 3 commits into
Conversation
DeepSeek's official Anthropic-compatible endpoint returns token usage but no per-request cost, no generation id, and no usage or billing endpoint, so the existing OpenRouter usage/generation reconciliation cannot resolve a cost. Add estimated_cost(), which prices the returned input, output, and cache token counts with a static DeepSeek price list. It is used only after usage_cost() fails, so a provider-reported cost always wins, and it is marked kind: estimated to stay distinct from a reported or reconciled cost.
Pinned task images occasionally resolve a package index that still points at removed .deb versions, so apt-get install fails with a 404 during agent setup before any model call. Retry the system-dependency transaction a bounded number of times; non-404 failures and the final attempt still propagate.
Add the structured result and report for running the 20-task pilot set on the DeepSeek official endpoint with deepseek-flash (16/18 scored, two infrastructure exceptions, token-estimated cost), plus the changelog entries for the cost estimator and the setup retry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Switches the benchmark path to DeepSeek's official Anthropic-compatible endpoint and adds the two pieces it needs: a token-based cost estimator (DeepSeek reports no per-request cost) and a setup retry for transient package-index 404s. Records the resulting pilot20.
estimated_cost(model, usage)with a static DeepSeek price list. Used only afterusage_cost()fails, markedkind: estimated.apt-gettransaction a bounded number of times on a transient HTTP 404.Result
20-task pilot set,
deepseek-flash, endpointhttps://api.deepseek.com/anthropic, max_turns 100, max_tokens 65536, concurrency 2.Reward 0:
torch-tensor-parallelism,pytorch-model-recovery. Exceptions:qemu-alpine-ssh(apt 404 during setup, no model call) andtorch-pipeline-parallelism(verifier timeout).Token-estimated cost: $1.356802 (input 0.3, cache read 0.006, output 1.2 USD per million).
Notes
bullseye-securityindex references removed.debversions, so the 404 is deterministic. The historical fix preloads a verified.debcache; that remains a follow-up.Validation
uv run pytest— 333 passed, 4 skippeduv run --project benchmarks/harbor pytest -c benchmarks/harbor/pyproject.toml benchmarks/harbor/tests— 28 passed