You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Execution directive from North Star #216, 2026-10-15. This is the highest-priority customer-value gate. Preserve this issue as the canonical pilot task; do not create a parallel benchmark that masquerades as external adoption.
Pilot hypothesis: a security-sensitive FastAPI backend team using AI coding agents will find a bounded, honest authorization/Protected Effect advisory report useful enough to install and repeatedly review. This is a hypothesis to test with prospective design partners, not a claim of market demand. If the target team's actual change surface is CI secrets or self-protection, use an already supported advisory lane as an initial integration check, while retaining the authorization guarantee as the strategic differentiation to test.
Qualification before recruiting: identify responsible repo operator, PR volume, use of coding agents, reviewer, permitted integration mechanism, real source-profile coverage, privacy/security restrictions, and exact released OVK pin. Avoid unqualified v1.3.0-rc.1 claims. Do not seek installation in another project's CI without maintainer consent.
Advisory field protocol: observe a 14-day interval of the organization's natural PR stream, retaining all eligible/ineligible records. Target at least five genuine PRs as a minimum feasibility sample; extend the window if there are fewer, and report attrition. Store raw versioned artifacts bound to the exact PR SHA, exact configuration, verifier version and source profile. Run in non-blocking/advisory mode with unknown/unsupported explicitly reported. Obtain reviewer annotation for every consequential finding, including false alarms and materially missed issues discovered independently.
Required per-PR fields: source/target commit, changed surface, eligibility and coverage, checker invoked, claim class, decisive evidence and assumptions, result incl. unknown/abstention, runtime & CI billed cost, reviewer time, override, fix/recheck actions, and value to reviewer. Separate source-model and checker semantics. Never translate "PASS on IR" into "the source program is verified" without a qualified refinement.
Initial exit decision: independent reviewer finds the artifact intelligible and relevant in real engineering decisions; report all denominator/false-alarm cases and setup cost. Five PRs cannot establish a low false-negative rate. If no external partner accepts or source coverage is too narrow, record the NO-GO and revise market/source profile rather than claim a successful pilot.
Linked dependencies:#178/#179 for honest guarantee continuity; #25/#21 for attributable package/release state. #195 and #177 remain parallel scientific research, not pilot installation prerequisites in advisory mode.
Next human-owned action: shortlist ten qualified external teams, identify buyer/reviewer/operator for each, complete five problem interviews, secure one explicit pilot agreement. No organization or individual has yet agreed as of this update.
First candidate-repository reconnaissance (2026-10-15; discovery only, zero consent/partnership). Candidate work must be screened for the supported FastAPI source shape and real reviewer demand before offering installation. GitHub search confirms these public, unarchived repos and existing open PRs:
langflow-ai/langflow — candidate priority B: substantial application code, active backend risk changes; examples upload rejection PR #15788, audit-identity PR #15755. Prior OVK auth-shape evaluation used Langflow examples, which are development exposure, not independent accuracy evidence. Unknown: reviewer interest, FastAPI profile eligibility, deployment friction.
polarsource/polar — candidate priority C for a commercially meaningful financial/workflow domain; example recurring interval API validation PR #15539. Higher likely integration and governance burden; unknown whether current OVK auth profile covers an important route/effect.
yezz123/authx — possible smaller security-oriented maintainer interview, but current open PR list is dominated by dependency maintenance; it may be a poor first pilot for meaningful Protected Effect changes.
No message has been sent and no pilot was installed. Public PR existence establishes a possible active workstream only. The next action is to inspect exact source-profile eligibility on 1–2 public, development-only candidate PRs, record unknowns, and then solicit maintainer permission for an advisory trial. Do not publish test results as independent holdout measurements: Aegra/Langflow have been used in earlier OVK research. A distinct prospective cohort needs precommitment and reviewer agreement.
Selection criterion: prioritize expected verified customer utility per unit of integration/triage cost, not project popularity. A working reviewer relationship and adequately supported claim beat a technically impressive but unsupported large repository.
Initial technical triage of two real open PRs (development screening only, 2026-10-15):
Aegra #687 changes libs/aegra-api/src/aegra_api/api/store.py inside namespace scoping/encoding (namespace_label, _scope). This is a meaningful tenant/identity boundary to examine with maintainers, but the diff is mainly helper-level path construction. Do not assume the current FastAPI route authorization extractor proves the relevant namespace noncollision or cross-tenant isolation property. That property needs a precise machine claim and source semantics; current OVK support is unverified for it.
Langflow #15755 changes API audit retrieval, authentication context and representation/issuer metadata across 24 files. It contains actual identity and audit-access logic, including an owner_visibility branch. This is an attractive high-value reviewer problem, but its composed plugin/DB/auth semantics are likely outside a narrow static FastAPI route-guard profile. Treat as customer-problem discovery and a coverage-gap candidate, not an expected PASS.
Screening conclusion: Neither of these examples is presently demonstrated to be inside OVK's supported Protected Effect proof envelope. The next engineering action is a bounded dry run with full supported / unknown / unsupported logging on a separate development record, then a reviewer discussion about which specific guarantee matters. No CI installation, contact, or claim of sound source-level verification has occurred; these PRs must remain excluded from later blinded confirmatory data because we have inspected them.
Select the initial pilot based on admitted claim coverage and buyer utility, not superficial similarity to security topics.
Prospective natural-PR pilot screening — external maintainers NOT contacted
After final-head #228 CI/Repro/native green, a GitHub search of recent workflow-editing PRs identified the following real code-change candidates; none is an independent OVK evaluation, none has consented, and no coverage or vulnerability-detection number is established.
A. PrefectHQ/prefect #23289 — high-value product discovery, semantic eligibility UNPROVEN. Real PR edits .github/workflows/agents-md-update.yml and docs-update.yml, migrates documentation-update agents from Claude Code to Devin, reduces workflow permissions from contents:write/pull-requests:write to contents:read, and introduces DEVIN_API_KEY/DEVIN_ORG_ID on trigger-to-remote-agent boundaries. The question is whether a bounded CI-secrets and tool-authority property provides reviewer utility. The existing OVK workflow extractor reconstructs post-images from unified diff hunks, and explicitly skips invalid partial YAML. A touched workflow does not imply complete construct support; obtain the full post-change workflow at the exact head and qualify the supported claim before any pilot. This is a large, actively maintained outside ecosystem and is a more meaningful distribution hypothesis than a tiny fixture-only repository.
B. PrefectHQ/prefect #23360 — workflow secret boundary, eligibility UNPROVEN. Modifies DockerHub login conditional: login is enabled on pushes to main and on same-repository PRs; credentials are secrets.DOCKERHUB_USERNAME and secrets.DOCKERHUB_TOKEN. Full-event, exact-head condition and provenance analysis would be needed to distinguish intended trusted contexts from untrusted forks. The patch is a short hunk in a much larger YAML workflow, so source coverage must be reported unknown unless the complete post-image is actually loaded/checked.
C. flagos-ai/Torch-FL #563 — AI reviewer trust/approval semantics, source-family mismatch expected. Adds a pull_request_target AI-review job with secret AI_REVIEW_API_KEY plus a separately privileged machine-user PAT that submits an approval; changes CI tests. The workflow explicitly avoids checking out untrusted PR code. OVK's current workflow YAML extractor marks pull_request_target_with_pr_head_checkout by looking for PR-head-related strings anywhere in jobs, not by proving they configure an actual checkout. That heuristic could yield incorrect conclusions if event-head references occur only in metadata/API calls. Do not promote an unreviewed red/green here; this PR suggests a potential cross-project, agent-reviewer-authority claim family, contingent on a substantially stronger exact-step/effect extractor. Stars/traffic are not evidence of fit.
D. ogabasseyy/awoof #55 — explicit advisory AI-review CI with pull_request_target, fork restrictions, review-posting code and external API credentials. Source profile relevance plausible, but the repo has a small audience. More useful as a boundary case for guard-to-effect extraction and prompt-injection/secret permissions than as a virality target.
Method/gates for a real pilot: request maintainer consent (read-only, non-blocking pull_request workflow with minimum contents:read, no PR code exposed to secrets, source pinned to an attributable release), declare claim semantics and exact YAML coverage, capture all naturally occurring PRs and unsupported/unknown cases over a fixed window, preserve exact base/head/source identities, separately adjudicate false alerts/misses and reviewer minutes. A second natural PR and independent teammate reuse are the strongest product distribution milestones. No cold contact, code push to third-party repositories, check publication, merge gate, or security guarantee without explicit authorization.
Recommendation: source-profile eligibility screen first for Prefect #23289/#23360; decline unsupported PRs explicitly. Pursue live pilot only after release/ref governance #25, portable candidate review #228, and actual maintainer consent. Do not count research screening as an adoption success.
Release/pilot source-profile eligibility — partial GitHub Actions YAML is not a full source model (2026-10-17)
Confirmed mechanism, not a demonstrated exploitable false-PASS: At frozen development candidate 837c931d059f74dcda06432d83f5bde759f53d0a, ovk/core/diff_parser.py::extract_post_images constructs an approximate post-change file image using only unified-diff context/addition lines, omitting non-hunk material. ovk/adapters/workflow/diff_extract.py::workflow_inputs_from_diff skips YAML parse failures (except (yaml.YAMLError, ValueError): continue). ovk/adapters/workflow/yaml_extract.py also accepts any YAML mapping and defaults missing jobs to zero steps, missing triggers/permissions to empty structures. ovk/core/lane_compiler.py compiles only the resulting workflow inputs; ovk/core/kernel.py emits explicit compilation_failure_evidence if no obligations exist at all.
Precise unresolved case: A diff with multiple changed workflows where one parses and yields an obligation, while another workflow is silently skipped or partially reconstructed, might produce a result whose coverage is misread as encompassing both workflows. End-to-end real PASS behavior for this mixed case is not established by this inspection, so test the actual kernel result before asserting a vulnerability. A zero-job/unknown/partial file must never be reported as fully checked.
Mandatory source eligibility regression before choosing workflow-secret PRs as the first adoption wedge
Build a real two-file PR at pinned base/head/merge SHAs. Workflow A has a complete admissible post-image and a supported bounded secrets property. Workflow B is changed through a small hunk in the middle of a larger, valid workflow; separately test partial parse, syntactically valid incomplete mapping, and an unsupported job form.
Capture the complete post-change blob and Git tree identity from the checked-out, source-bound merge preview for each changed .github/workflows/*.{yml,yaml} path. Compare per-file digest to the exact checkout blob. Diff fragments alone cannot establish complete YAML semantics.
Record explicit per-workflow source_coverage = complete | partial | unsupported | error (or the versioned equivalent), provenance and excluded constructs. Assert an absent or unparsable workflow remains unresolved in the overall claim-specific decision, even if another workflow passed a different supported check.
Test cases for renamed, deleted, reusable/composite-action, pull_request_target, nested triggers, job/step conditional expressions, secret contexts and permissions; classify supported claim semantics explicitly. Do not make unsupported expressions look like checked authorization flows.
Regression must inspect the final EvidenceBundle and reviewer-readable report, not only extractor unit output. Validate no misleading allow for an entire workflow/PR due to skipped or partial source; ensure typed scope and unknown survive into downloadable Markdown and Action outputs.
Independently adjudicate a real PR only after eligibility test passes. Pilot prospect screening for Prefect #23289/#23360 remains analysis, not consent, verification coverage, or empirical accuracy.
Decision: no claim that OVK currently verifies arbitrary GitHub Actions secrets exposure or approval semantics. Keep advisory-only; explicitly describe the uncertainty to any design partner. This is a narrowly grounded trust boundary and a release-review requirement for a workflow-centric first pilot. Related independent-review launch gate #230.
Activity
fraware commentedon Oct 15, 2026
Execution directive from North Star #216, 2026-10-15. This is the highest-priority customer-value gate. Preserve this issue as the canonical pilot task; do not create a parallel benchmark that masquerades as external adoption.
Pilot hypothesis: a security-sensitive FastAPI backend team using AI coding agents will find a bounded, honest authorization/Protected Effect advisory report useful enough to install and repeatedly review. This is a hypothesis to test with prospective design partners, not a claim of market demand. If the target team's actual change surface is CI secrets or self-protection, use an already supported advisory lane as an initial integration check, while retaining the authorization guarantee as the strategic differentiation to test.
Qualification before recruiting: identify responsible repo operator, PR volume, use of coding agents, reviewer, permitted integration mechanism, real source-profile coverage, privacy/security restrictions, and exact released OVK pin. Avoid unqualified
v1.3.0-rc.1claims. Do not seek installation in another project's CI without maintainer consent.Advisory field protocol: observe a 14-day interval of the organization's natural PR stream, retaining all eligible/ineligible records. Target at least five genuine PRs as a minimum feasibility sample; extend the window if there are fewer, and report attrition. Store raw versioned artifacts bound to the exact PR SHA, exact configuration, verifier version and source profile. Run in non-blocking/advisory mode with unknown/unsupported explicitly reported. Obtain reviewer annotation for every consequential finding, including false alarms and materially missed issues discovered independently.
Required per-PR fields: source/target commit, changed surface, eligibility and coverage, checker invoked, claim class, decisive evidence and assumptions, result incl. unknown/abstention, runtime & CI billed cost, reviewer time, override, fix/recheck actions, and value to reviewer. Separate source-model and checker semantics. Never translate "PASS on IR" into "the source program is verified" without a qualified refinement.
Initial exit decision: independent reviewer finds the artifact intelligible and relevant in real engineering decisions; report all denominator/false-alarm cases and setup cost. Five PRs cannot establish a low false-negative rate. If no external partner accepts or source coverage is too narrow, record the NO-GO and revise market/source profile rather than claim a successful pilot.
Linked dependencies: #178/#179 for honest guarantee continuity; #25/#21 for attributable package/release state. #195 and #177 remain parallel scientific research, not pilot installation prerequisites in advisory mode.
Next human-owned action: shortlist ten qualified external teams, identify buyer/reviewer/operator for each, complete five problem interviews, secure one explicit pilot agreement. No organization or individual has yet agreed as of this update.
fraware commentedon Oct 15, 2026
First candidate-repository reconnaissance (2026-10-15; discovery only, zero consent/partnership). Candidate work must be screened for the supported FastAPI source shape and real reviewer demand before offering installation. GitHub search confirms these public, unarchived repos and existing open PRs:
No message has been sent and no pilot was installed. Public PR existence establishes a possible active workstream only. The next action is to inspect exact source-profile eligibility on 1–2 public, development-only candidate PRs, record unknowns, and then solicit maintainer permission for an advisory trial. Do not publish test results as independent holdout measurements: Aegra/Langflow have been used in earlier OVK research. A distinct prospective cohort needs precommitment and reviewer agreement.
Selection criterion: prioritize expected verified customer utility per unit of integration/triage cost, not project popularity. A working reviewer relationship and adequately supported claim beat a technically impressive but unsupported large repository.
fraware commentedon Oct 15, 2026
Initial technical triage of two real open PRs (development screening only, 2026-10-15):
libs/aegra-api/src/aegra_api/api/store.pyinside namespace scoping/encoding (namespace_label,_scope). This is a meaningful tenant/identity boundary to examine with maintainers, but the diff is mainly helper-level path construction. Do not assume the current FastAPI route authorization extractor proves the relevant namespace noncollision or cross-tenant isolation property. That property needs a precise machine claim and source semantics; current OVK support is unverified for it.owner_visibilitybranch. This is an attractive high-value reviewer problem, but its composed plugin/DB/auth semantics are likely outside a narrow static FastAPI route-guard profile. Treat as customer-problem discovery and a coverage-gap candidate, not an expected PASS.Screening conclusion: Neither of these examples is presently demonstrated to be inside OVK's supported Protected Effect proof envelope. The next engineering action is a bounded dry run with full
supported / unknown / unsupportedlogging on a separate development record, then a reviewer discussion about which specific guarantee matters. No CI installation, contact, or claim of sound source-level verification has occurred; these PRs must remain excluded from later blinded confirmatory data because we have inspected them.Select the initial pilot based on admitted claim coverage and buyer utility, not superficial similarity to security topics.
fraware commentedon Oct 16, 2026
Prospective natural-PR pilot screening — external maintainers NOT contacted
After final-head #228 CI/Repro/native green, a GitHub search of recent workflow-editing PRs identified the following real code-change candidates; none is an independent OVK evaluation, none has consented, and no coverage or vulnerability-detection number is established.
A. PrefectHQ/prefect #23289 — high-value product discovery, semantic eligibility UNPROVEN. Real PR edits
.github/workflows/agents-md-update.ymlanddocs-update.yml, migrates documentation-update agents from Claude Code to Devin, reduces workflow permissions fromcontents:write/pull-requests:writetocontents:read, and introducesDEVIN_API_KEY/DEVIN_ORG_IDon trigger-to-remote-agent boundaries. The question is whether a bounded CI-secrets and tool-authority property provides reviewer utility. The existing OVK workflow extractor reconstructs post-images from unified diff hunks, and explicitly skips invalid partial YAML. A touched workflow does not imply complete construct support; obtain the full post-change workflow at the exact head and qualify the supported claim before any pilot. This is a large, actively maintained outside ecosystem and is a more meaningful distribution hypothesis than a tiny fixture-only repository.B. PrefectHQ/prefect #23360 — workflow secret boundary, eligibility UNPROVEN. Modifies DockerHub login conditional: login is enabled on pushes to main and on same-repository PRs; credentials are
secrets.DOCKERHUB_USERNAMEandsecrets.DOCKERHUB_TOKEN. Full-event, exact-head condition and provenance analysis would be needed to distinguish intended trusted contexts from untrusted forks. The patch is a short hunk in a much larger YAML workflow, so source coverage must be reported unknown unless the complete post-image is actually loaded/checked.C. flagos-ai/Torch-FL #563 — AI reviewer trust/approval semantics, source-family mismatch expected. Adds a
pull_request_targetAI-review job with secretAI_REVIEW_API_KEYplus a separately privileged machine-user PAT that submits an approval; changes CI tests. The workflow explicitly avoids checking out untrusted PR code. OVK's current workflow YAML extractor markspull_request_target_with_pr_head_checkoutby looking for PR-head-related strings anywhere in jobs, not by proving they configure an actual checkout. That heuristic could yield incorrect conclusions if event-head references occur only in metadata/API calls. Do not promote an unreviewed red/green here; this PR suggests a potential cross-project, agent-reviewer-authority claim family, contingent on a substantially stronger exact-step/effect extractor. Stars/traffic are not evidence of fit.D. ogabasseyy/awoof #55 — explicit advisory AI-review CI with
pull_request_target, fork restrictions, review-posting code and external API credentials. Source profile relevance plausible, but the repo has a small audience. More useful as a boundary case for guard-to-effect extraction and prompt-injection/secret permissions than as a virality target.Method/gates for a real pilot: request maintainer consent (read-only, non-blocking
pull_requestworkflow with minimumcontents:read, no PR code exposed to secrets, source pinned to an attributable release), declare claim semantics and exact YAML coverage, capture all naturally occurring PRs and unsupported/unknown cases over a fixed window, preserve exact base/head/source identities, separately adjudicate false alerts/misses and reviewer minutes. A second natural PR and independent teammate reuse are the strongest product distribution milestones. No cold contact, code push to third-party repositories, check publication, merge gate, or security guarantee without explicit authorization.Recommendation: source-profile eligibility screen first for Prefect #23289/#23360; decline unsupported PRs explicitly. Pursue live pilot only after release/ref governance #25, portable candidate review #228, and actual maintainer consent. Do not count research screening as an adoption success.
fraware commentedon Oct 17, 2026
Release/pilot source-profile eligibility — partial GitHub Actions YAML is not a full source model (2026-10-17)
Confirmed mechanism, not a demonstrated exploitable false-PASS: At frozen development candidate
837c931d059f74dcda06432d83f5bde759f53d0a,ovk/core/diff_parser.py::extract_post_imagesconstructs an approximate post-change file image using only unified-diff context/addition lines, omitting non-hunk material.ovk/adapters/workflow/diff_extract.py::workflow_inputs_from_diffskips YAML parse failures (except (yaml.YAMLError, ValueError): continue).ovk/adapters/workflow/yaml_extract.pyalso accepts any YAML mapping and defaults missingjobsto zero steps, missing triggers/permissions to empty structures.ovk/core/lane_compiler.pycompiles only the resulting workflow inputs;ovk/core/kernel.pyemits explicitcompilation_failure_evidenceif no obligations exist at all.Precise unresolved case: A diff with multiple changed workflows where one parses and yields an obligation, while another workflow is silently skipped or partially reconstructed, might produce a result whose coverage is misread as encompassing both workflows. End-to-end real
PASSbehavior for this mixed case is not established by this inspection, so test the actual kernel result before asserting a vulnerability. A zero-job/unknown/partial file must never be reported as fully checked.Mandatory source eligibility regression before choosing workflow-secret PRs as the first adoption wedge
.github/workflows/*.{yml,yaml}path. Compare per-file digest to the exact checkout blob. Diff fragments alone cannot establish complete YAML semantics.source_coverage = complete | partial | unsupported | error(or the versioned equivalent), provenance and excluded constructs. Assert an absent or unparsable workflow remains unresolved in the overall claim-specific decision, even if another workflow passed a different supported check.pull_request_target, nested triggers, job/step conditional expressions, secret contexts and permissions; classify supported claim semantics explicitly. Do not make unsupported expressions look like checked authorization flows.allowfor an entire workflow/PR due to skipped or partial source; ensure typed scope and unknown survive into downloadable Markdown and Action outputs.Decision: no claim that OVK currently verifies arbitrary GitHub Actions secrets exposure or approval semantics. Keep advisory-only; explicitly describe the uncertainty to any design partner. This is a narrowly grounded trust boundary and a release-review requirement for a workflow-centric first pilot. Related independent-review launch gate #230.