test(agent-prompt): gate the documented prompt budget against the enforced one (#585) - #586
Merged
Merged
Conversation
…orced one Rebase of #586 onto main after v0.88.0's budget raise (62_000 -> 70_000, #629) and the keboola-expert.md trim (#634). Main's own doc prose (CONTRIBUTING.md, kbagent-pr-reviewer.md) already carries the correct 70 000 B figure and the enforcing-test cross-reference, so nothing there needs to change -- only the drift-guard test itself was still missing. Add test_documented_budget_matches_enforced_budget, which scans BUDGET_DOC_SITES for a "<number> B budget" mention and asserts the number matches PROMPT_BYTE_BUDGET, derived from the constant rather than a second hardcoded literal. That makes a legitimate future budget bump (constant and docs moving together) a no-op for this test, while a doc site left behind still fails it. The regex also recognizes the deprecated "KB" form so a site that regresses to kilobytes is caught, and tolerates space/comma grouping in the digits ("70 000", "70,000", "70000").
padak
force-pushed
the
claude/issue-585-enforce-prompt-budget-docs
branch
from
August 22, 2026 20:45
2a4412c to
09a879c
Compare
padak
marked this pull request as ready for review
August 22, 2026 20:58
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
padak
added a commit
that referenced
this pull request
Aug 22, 2026
pyproject/plugin.json/marketplace.json were already renumbered to 0.89.0 by v0.88.0. Changelog: adds 0.89.0 entries for #645 (describe-batch --from-file shape validation, issue #640), #642 (table-detail human column descriptions), #620 (sync-action forwards root authorization/runtime), #517 (stable metavar contract, issue #513), #586 (documented prompt budget gated against the enforced one, issue #585) and #641 (docs-only), and decorates the existing Silent-drift surfaces: * gotchas.md -- resolves both "(Release step: ... tag this sentence)" placeholders. Both were left by commits AFTER the v0.88.0 tag (#642 and #645), so both are tagged (since v0.89.0), not 0.88.0. Adds the #620 gotcha: below 0.89.0 a sync action on an OAuth / Service-Account component died with an opaque empty-body 400 because the broker reference was never forwarded. * #620 shipped with no doc surfaces at all -- CLAUDE.md, AGENT_CONTEXT and commands-reference.md now carry the forwarding rule (root only, never row-overridden, only when non-empty) with its version gate. * #645 never reached CLAUDE.md -- the describe-batch shape check and its behaviour change are recorded there now; commands-reference gains the version tag. * #642's human Description column is version-tagged in CLAUDE.md, commands-reference.md, AGENT_CONTEXT and storage-describe-workflow.md. * #643 was otherwise complete; adds the two surfaces it did not touch -- safe-write-workflow.md (delete is reversible; never blind-retry on <= 0.88.x) and a keboola-expert.md matrix row for delete/restore/trash-list. keboola-expert.md is 49 774 B, well inside the 70 000 B budget. make check green: 5934 passed, 12 skipped. version-gate-check resolves all 438 markers across 72 versions.
padak
added a commit
that referenced
this pull request
Aug 23, 2026
pyproject/plugin.json/marketplace.json were already renumbered to 0.89.0 by v0.88.0. Changelog: adds 0.89.0 entries for #645 (describe-batch --from-file shape validation, issue #640), #642 (table-detail human column descriptions), #620 (sync-action forwards root authorization/runtime), #517 (stable metavar contract, issue #513), #586 (documented prompt budget gated against the enforced one, issue #585) and #641 (docs-only), and decorates the existing Silent-drift surfaces: * gotchas.md -- resolves both "(Release step: ... tag this sentence)" placeholders. Both were left by commits AFTER the v0.88.0 tag (#642 and #645), so both are tagged (since v0.89.0), not 0.88.0. Adds the #620 gotcha: below 0.89.0 a sync action on an OAuth / Service-Account component died with an opaque empty-body 400 because the broker reference was never forwarded. * #620 shipped with no doc surfaces at all -- CLAUDE.md, AGENT_CONTEXT and commands-reference.md now carry the forwarding rule (root only, never row-overridden, only when non-empty) with its version gate. * #645 never reached CLAUDE.md -- the describe-batch shape check and its behaviour change are recorded there now; commands-reference gains the version tag. * #642's human Description column is version-tagged in CLAUDE.md, commands-reference.md, AGENT_CONTEXT and storage-describe-workflow.md. * #643 was otherwise complete; adds the two surfaces it did not touch -- safe-write-workflow.md (delete is reversible; never blind-retry on <= 0.88.x) and a keboola-expert.md matrix row for delete/restore/trash-list. keboola-expert.md is 49 774 B, well inside the 70 000 B budget. make check green: 5934 passed, 12 skipped. version-gate-check resolves all 438 markers across 72 versions.
padak
added a commit
that referenced
this pull request
Aug 23, 2026
…es (#651) Release prep for 0.89.0: adds the changelog entries for everything merged since v0.88.0 (#620, #642, #643, #644, #645, #646, #647, #648, #649, #650, #517, #586, #641), resolves every vNEXT placeholder left by feature PRs to v0.89.0 per the new #648 release process, closes the 10 gaps a full doc-surface audit found across the kbagent plugin (SKILL.md triggers, commands-reference, gotchas, workflow files, keboola-expert.md, AGENT_CONTEXT, CLAUDE.md), and records the live e2e verification evidence. Version files were already at 0.89.0 (bumped by #643); make version-sync is a no-op.
claude Bot
pushed a commit
that referenced
this pull request
Aug 23, 2026
Resolves the sync-map conflict in CONTRIBUTING.md. main moved from 0.86.0 to 0.89.0 while this branch sat, and two adjacent table rows collided: - `keboola-expert.md` row: keep main's "70 000 B prompt budget". v0.88.0 raised the ceiling from 62 000 B, and tests/test_agent_prompt.py (test_documented_budget_matches_enforced_budget, added in #586) now gates CONTRIBUTING.md against PROMPT_BYTE_BUDGET, so this branch's stale "60 KB" would fail CI. - `commands/*.md` row: keep this branch's widened row plus its follow-through list for adding a new slash-command file. No version or changelog change: main's #648 moved version bumps into dedicated release PRs, and this branch bumps nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
test_documented_budget_matches_enforced_budgetand corrects the four doc sites that quoted a stale prompt-budget figure.Why — a correction to the issue's premise first
The issue reports that the 62 000 B budget on
plugins/kbagent/agents/keboola-expert.mdis unenforced. It is enforced, and has been since v0.48.0:That test is in the per-PR CI suite (
ci.yml,pull_requesttrigger), andactions/checkoutchecks out the merge commit — so it already fails the merge, which is the scenario the issue is about. The proposed new test would have duplicated it.The reason
grep -rn 62000 scripts/ tests/ .github/ Makefilefound nothing: the constant is spelledPROMPT_BYTE_BUDGET = 62_000with a PEP 515 underscore separator, which that pattern cannot match.Why — the real defect underneath
Chasing that miss surfaced a genuine drift. Four places state the budget in prose, and all four were stale:
CONTRIBUTING.md:381CONTRIBUTING.md:464CONTRIBUTING.md:594plugins/kbagent/agents/kbagent-pr-reviewer.md:138v0.48.0 raised the ceiling 60 kB → 62 kB in the test and left every doc site behind. The file sits at 61 990 B today — over the documented budget, under the enforced one. So the prose was actively misleading in both directions: an author trimming to 60 KB cut ~2 kB of signal for nothing, and
/kbagent:review(which readskbagent-pr-reviewer.md) was telling reviewers the wrong number.Changes
tests/test_agent_prompt.py— newtest_documented_budget_matches_enforced_budget: every path inBUDGET_DOC_SITESmust quotePROMPT_BYTE_BUDGETas62 000 Band must not carry a60 KB prompt budgetclaim. Raising the cap now forces the docs to move with it.tests/test_agent_prompt.py— comment spells the literal in plain digits sogrep 62000hits, and names the constant as the single source of truth.CONTRIBUTING.md— three sites corrected; the first now also names the enforcing test, notes that CI builds the merge commit, and giveswc -cas the pre-flight check.plugins/kbagent/agents/kbagent-pr-reviewer.md— corrected and pointed at the test.No behavior change; docs and test only. No version bump, no changelog entry.
How it was tested
make check— clean: lint, format, changelog-check, 5490 passed, 12 skipped.CONTRIBUTING.mdsite to60 KBmakes it fail with the drifted path named; restoring makes it pass. A drift gate that cannot fail is worthless, so this was verified rather than assumed.wc -c), within budget.Note on headroom
The issue flags ~15 bytes free. Real headroom on this branch is 10 bytes (61 990 of 61 999 usable — the assertion is strict
<). The next addition to that file hits the ceiling immediately; this PR does not change that, it only makes the number people trim against the correct one. Splittingkeboola-expert.mdinto per-domain specialists is the standing recommendation in the test's own comment and is out of scope here.Fixes #585