feat(agents): cross-model validation and strict majority consensus (P0)

[Mahajan-Sachin]

Closes #34

Summary

This PR implements P0 from SCANNER_AUGMENTATION_PLAN.md to eliminate precision inflation caused by self-validation ("model grading its own work") and ties defaulting to confirmed findings.

Changes

1. Cross-Model Validation Resolution:

   • In openhack/config.py, added resolve_validator_model and default cross-model mappings (e.g. hunter gpt-5.6-sol / glm-5.2 -> validator kimi-k2.5).
   • Added check_self_validation(hunter_model, validator_model) issuing startup warning log and session trace event when validator model matches hunter model.
   • Updated ValidatorSwarmAgent and CoordinatorAgent to resolve independent validator models by default.

2. Hardened Exploitability Prompts:

   • Ported hardened validator system prompt rules to VALIDATOR_PROMPT in openhack/prompts/validator.py:
     • CSRF: Confirm only if auth cookie is NOT SameSite=Lax/Strict.
     • XXE: Confirm only if XML parser resolves external entities (xml2js/sax do not -> false_positive).
     • XSS: Confirm only if input reaches a raw-HTML sink (JSX text is auto-escaped -> false_positive).
     • Framework Defaults & Blank Rationale: Dev configs (Nuxt DevTools, Vite HMR) and blank rationale -> false_positive.
     • Strict FP Definition: false_positive strictly means mitigated or unexploitable.
   • In openhack/agents/validator.py, framed candidate findings as an unverified hypothesis for independent audit.

3. Strict Majority Consensus Voting:

   • Added validator_votes: int = 1 in Settings (can be set to 3 for strict 2-of-3 voting).
   • In ValidatorSwarmAgent, confirmed status requires a strict majority (>= (votes // 2) + 1). Ties (e.g. 1-of-2) are strictly rejected.

Tests

• Added tests/test_cross_model_validation.py (12/12 passing).
• 41/41 related regression tests passing cleanly.

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge, with no outstanding findings.

Summary

The PR adds cross-model review, stricter validator prompts, and strict-majority voting.

• Hosted scans pick a different validator model by default.
• The validator checks concrete exploit conditions before confirming findings.
• Findings need a strict majority of validator votes to be confirmed.

No new actionable issues were found in the changes since the previous review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Collect validator votes] --> B{Confirmation majority?}
  B -->|Yes| C[Confirmed]
  B -->|No| D{Execution failures without rejection majority?}
  D -->|Yes| E[Unreviewed]
  D -->|No| F[Existing rejection path]

Loading

_{Reviews (7) · Last reviewed commit: "fix(validator): treat unfinished reviews..." · Reviewed by Greptile}

[Mahajan-Sachin]

@greptile-apps Please re-review the latest changes and regenerate the confidence score. All reported issues have been addressed:

1. Issue 1 (Prompt formatting crash): Doubled literal braces {{userInput}} and {{{{ }}}} in VALIDATOR_PROMPT. Added unit test verifying ValidatorAgent.get_system_prompt() formats without errors.
2. Issue 2 & Feature Hunter Review: Updated _get_models() in ValidatorSwarmAgent to preserve automatic cross-model selection when caller passes an unresolved client, while preventing double-inversion when hunter_model is explicitly passed.
3. Issue 3 (XSS scope): Expanded XSS prompt checks to allow executable URL attributes (such as href with javascript:) while keeping JSX text auto-escaping exclusion.
4. Issue 4 (Failed review handling): Distinguished unreviewed / failed validation reviews (needs_more_info, confidence: low) from actual false_positive rejections.
5. Issue 5 (Unit test isolation): Mocked ValidatorAgent.run in offline tests so they never touch the hosted API. All 16/16 unit tests passing.

[Mahajan-Sachin]

@greptile-apps any review, left?

[Mahajan-Sachin]

Hi @ananay,

This PR is ready for final review and merge! 🚀

Summary of What's Included (P0 from Augmentation Plan)

1. Cross-Model Validation & Anti-Self-Grading:

   • The scanner automatically selects an independent validator model (e.g., hunter gpt-5.6-sol / glm-5.2 ➔ validator kimi-k2.5) so hunters never grade their own findings.
   • Preserves automatic selection across all entry points (coordinator, swarms, and standalone scripts like run_feature_hunt.py).
   • Emits a warning and session trace if an identical model is forced by explicit overrides.

2. Hardened Exploitability Prompts:

   • Requires concrete proof of exploitability before confirmation (validates cookie SameSite flags for CSRF, verifies external entity parsing for XXE, and checks for raw-HTML sinks for XSS).
   • Rejects framework dev defaults (e.g., Nuxt DevTools, Vite HMR) and blank hunter rationales.

3. Strict Majority Consensus Voting:

   • Supports multi-vote consensus (validator_votes), requiring a strict majority (>= (votes // 2) + 1) to confirm.
   • Ties are strictly rejected, and execution/API failures are recorded separately as unreviewed rather than being misattributed as rejections.

Review & Test Status

• Bot Review: All Greptile review comments have been addressed and resolved. Greptile rated the PR Confidence Score: 5/5 ("The PR appears safe to merge, with no outstanding findings").
• Test Suite: Added 18 unit tests in tests/test_cross_model_validation.py covering model resolution, strict majority voting, failure isolation, and prompt checks. All tests pass cleanly.

Looking forward to your review and feedback!