Repository navigation
fix: optimize 27 web-security skills - #22
Closed
GangGreenTemperTatum wants to merge 4 commits into
Closed
GangGreenTemperTatum wants to merge 4 commits into
GangGreenTemperTatum wants to merge 4 commits into
Conversation
Rewrote lowest-scoring skills (data-exfil 18%→90%, scorer-reference 61%→86%, dom-vulnerability-* 71%→97%, hackerone-recon 74%→97%, vuln-kb 75%→100%). Common fixes: allowed-tools array→string, added Use-when clauses, removed verbose explanations of concepts the agent already knows, replaced pseudocode with executable rg/curl/grep commands, added validation checkpoints to workflows, cut monolithic files to essentials. 28 skills improved, average score 86%→92% across all 58 skills.
Targeted fixes based on judge feedback for each skill: - jxscout-static-analysis: 84% -> 96% (added trigger terms, trimmed HTTP context/limitations) - mcp-auth-exploitation: 86% -> 90% (added validation checkpoints between phases, trimmed explanatory text) - write-path-to-rce: 86% -> 97% (enriched description with specific frameworks and actions) - burp-suite: 85% -> 97% (added verification steps, common workflows section) - subdomain-takeover-check: 85% -> 94% (added executable examples, error handling guidance) - exploit-verifier: 86% -> 98% (added references to bundle files, consolidated inline guidance) - jxscout-relationships: 88% -> 93% (consolidated use cases into workflow, expanded checkpoint) - agent-browser: 89% -> 93% (trimmed auth/viewport/iOS/eval sections, reduced from 586 to 393 lines) - jxscout-security-research: 89% -> 99% (improved description trigger terms) - ssrf-redirect-loop: 89% -> 94% (added concrete curl examples, validation checkpoint) Skipped 3 skills already >=90%: scorer-reference (93%), oauth-flow-hijack (92%), dompurify-mxss-bypass (90%) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Port skills that existed in project-obsidian-md but were missing from the web-security capability: custom-sanitizer-audit, archive-path-traversal, type-confusion-testing, self-xss-escalation, saas-provider-url-ssrf, inline-script-breakout-exfil, config-file-parsing-bugs, insecure-defaults, and caido-proxy (adapted from caido-skill to match existing MCP server). All trimmed from source (avg 280→100 lines), large skills split into reference files, tessl review scores 92-100%.
67 skills (was 58), 9 ported from project-obsidian-md, 27 optimized via tessl review. Update description to reflect 60+ playbooks.
GangGreenTemperTatum
added a commit
that referenced
this pull request
Aug 13, 2026
…o-ts) Vendor the upstream caido/skills caido-mode skill (merged PR #22, @caido/sdk-client / caido-ts v0.4.0) into web-security as a third, independent Caido surface alongside the Python caido-sdk skill and the caido-proxy / caido-go MCP servers. - Vendor skills/caido-mode/ (tsx CLI + lib + tests) from caido/skills@5bb27a9 - Adapt SKILL.md for the dreadnode runtime: bootstrap from the skill dir, document non-interference with caido-sdk/caido-proxy (separate auth cache), and note runtime-provided CAIDO_URL/CAIDO_PAT - install_tools.sh: npm install the skill's node_modules at provision time - Dockerfile.runtime: pre-install tsx globally; document the caido-mode dep - capability.yaml: add caido-mode check, update description, bump to 1.7.0 Node deps (node_modules) are gitignored and installed at provision time. All 68 skill unit tests pass; just validate is green for web-security.
bjcpu01
pushed a commit
to bjcpu01/dn-capabilities
that referenced
this pull request
Oct 15, 2026
The vendored caido-mode skill already matches caido/skills@41697d8 (PR dreadnode#22) byte-for-byte, so there was no content to sync. What PR dreadnode#22 surfaced is that its central fix -- "restore Caido 0.57 compatibility (SDK 0.2.0 -> 0.4.0)" -- landed on only one of the four Caido surfaces this capability ships. Caido 0.57 changed the replay schema: ReplaySession became an interface, createReplaySession requires `kind: ReplaySessionKind!`, and sending moved to a task model. Bring the rest of the stack onto that side of the break. Version contract - caido-cli pin 0.45.0 -> 0.57.1 (both linux arches verified downloadable). The server was two years behind the client the skill runs. - caido-sdk-client floored at >=0.3.0 in capability.yaml, mcp/caido.py's PEP-723 header, and Dockerfile.runtime. A bare spec resolves 0.2.0, which hardcodes the pre-0.57 documents and fails with "Unknown field collection on type ReplaySession". 0.3.0 added the transport/latest vs transport/v0_56 split that negotiates per instance. Latent bug: caido_replay_request never worked mcp/caido.py passed ReplaySendOptions(host=, port=, tls=) and read result.task_status. The real dataclass is (raw, connection, settings) with connection as ConnectionInfoInput(host, port, is_tls), and the result field is `status`. This raised TypeError on every call, on 0.2.0 and 0.3.0 alike. It survived because the existing tests stub the SDK with MagicMock, which accepts any kwargs. Same wrong call was copied into the caido-sdk skill's example, which agents run verbatim. Skills - caido-mode: Dreadnode skill format -- drop upstream's `tags:` (not read by the loader; carried as metadata.role), add compatibility + provenance metadata, and a re-sync note listing the local fork points. Replace the three-surface blurb with a four-surface routing table and an auth-isolation table showing why the credential stores don't collide. - caido-sdk: correct the replay example, add an "API shapes that bite" table and the version floor. - caido-proxy: the MCP Tools section documented `mcp__caido__*` names that exist on neither server (Automate et al are caido-go only). Replaced with the real 9-tool Python surface taken from source, plus a two-server routing table, the access-token-vs-Cloud-PAT distinction, and redaction notes. Tests (+377 total, all green) - test_caido_mode_skill.py: skill tree, Dreadnode frontmatter, the version contract across all declaration sites, provisioning, and cross-surface coexistence. Also validates the caido-sdk example's constructor kwargs and the caido-proxy tool names against mcp/caido.py by AST, so docs can't drift from code. - test_caido_mcp.py: static TestCaidoSdkApiContract that catches the class of bug the mocks hid. Each guard was mutation-tested by reintroducing the defect. Also drop tools/caido_proxy.py (a `CaidoTools = None` stub referenced only by its own test), clearing the sole real --strict validate warning; true up the skill count to 81; bump to 1.10.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrote lowest-scoring skills (data-exfil 18%→90%, scorer-reference 61%→86%, dom-vulnerability-* 71%→97%, hackerone-recon 74%→97%, vuln-kb 75%→100%).
Common fixes: allowed-tools array→string, added Use-when clauses, removed verbose explanations of concepts the agent already knows, replaced pseudocode with executable rg/curl/grep commands, added validation checkpoints to workflows, cut monolithic files to essentials.
28 skills improved, average score 86%→92% across all 58 skills.