Skip to content

docs: record the deepseek-v4.1-flash rerun of six failed tasks - #40

Merged
minixalpha merged 1 commit into
mainfrom
docs/record-v4.1-flash-6task
Sep 26, 2026
Merged

minixalpha merged 1 commit into
mainfrom
docs/record-v4.1-flash-6task

Conversation

@minixalpha

Copy link
Copy Markdown
Owner

What

Record a benchmark experiment: re-run the six tasks that scored reward 0 in
tb21-stream-recovery-pilot20-20260921, using deepseek/deepseek-v4.1-flash
with every other limit unchanged (agent 1386df48e, max_tokens=65536,
max_turns=50, 3600 s per task, concurrency 2, one attempt, no retries).

Result

  • 4 / 6 passed: kv-store-grpc, dna-assembly, log-summary-date-ranges,
    circuit-fibsqrt.
  • 2 / 6 still failed by 50-turn exhaustion: caffe-cifar-10 (training only
    reached Iteration 200), mteb-leaderboard (never wrote /app/result.txt).
  • dna-assembly and circuit-fibsqrt passed cleanly for the first time.
  • 206 / 206 model replies returned complete; no transport interruptions;
    total cost $0.278759041.

Changes

  • docs/dev_notes/zh-CN/0.8.x.md — new “更换模型复跑未通过的 6 题” section (Chinese source).
  • docs/dev_notes/en/0.8.x.md — English version generated from the Chinese source.
  • benchmarks/harbor/results/tb21-v41flash-6task-20260925.json — structured result with per-task outcome, tokens, and cost.

Notes

  • No changelog entry: this records a benchmark experiment, not a user-facing change.
  • Not a controlled comparison — the model changed while routing, sampling, cache,
    and backend stayed uncontrolled, with a single attempt per task.
  • Raw trials and the experiment record live in Git-ignored jobs/ and are not uploaded.

Re-run the six tasks that scored reward 0 in the stream-recovery retest
with deepseek/deepseek-v4.1-flash, all other limits unchanged. Four
passed (kv-store-grpc, dna-assembly, log-summary-date-ranges,
circuit-fibsqrt) and two still failed by turn exhaustion (caffe-cifar-10,
mteb-leaderboard). dna-assembly and circuit-fibsqrt passed cleanly for
the first time.

Add the structured result under benchmarks/harbor/results/ and the full
account to both dev-notes languages. No changelog entry: this records a
benchmark experiment, not a user-facing change.
@minixalpha
minixalpha merged commit 41476d4 into main Sep 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant