Skip to content

question(ci): audit required checks and merge-queue coverage before expanding enforcement #1736

Closed

Description

@dnlrsls

Question

Which existing and proposed deterministic checks should be required for Engram, and do their current event coverage, reliability and exact check names support that policy on both PRs and the merge queue?

Why does this need issue tracking?

This needs inspection of repository settings, workflow behavior and existing policy, followed by a durable maintainer decision. Workflow presence alone does not prove a check is required. Unstable or missing merge-group checks can block contributors or leave enforcement gaps.

Affected Area

Project workflow or policy.

Additional Context

Acceptance criteria:

  • Record the actual current required contexts/rulesets and any inaccessible settings.
  • Map each current required context to its workflow/job and relevant PR/merge-group events.
  • Compare actual enforcement with CONTRIBUTING.md; distinguish existing behavior from proposals.
  • Record failures/instability before recommending any new mandatory check.
  • Evaluate the proposed dependency-review check separately from advisory PR-size and AI-review output.
  • Produce a scoped recommendation with migration/rollback considerations and explicit owner approval.
  • Do not mutate rulesets, add required checks or bypass existing gates as part of this audit.

Non-goals: reimplementing completed merge-queue compatibility, forcing unstable checks, or making AI verdicts authoritative.
Related prior work: #926 (merge-queue compatibility) and #1140 (required type-label check migration).
Companion configuration improvements should use focused PRs and remain separately reviewable.

Activity

  1. added
    type:questionTracked question requiring maintainer investigation or a durable decision
    on Oct 9, 2026
  2. dnlrsls commented on Oct 9, 2026

    @dnlrsls
    MemberAuthor

    Read-only audit: preserve the six current required contexts

    The effective rules and workflow mapping support keeping the existing gates unchanged. This is a bounded diagnosis, not approval of new gates or completion of every acceptance criterion. Source was pinned to b737f4a86ad0498727735fa376dc31f823a7607c; settings and history were observed live.

    Confirmed enforcement

    • Both active repository rulesets were readable: pr (13932028) and merge-queue (23758931). The effective main rules independently confirmed the six contexts below, all bound to GitHub Actions integration 15368, with strict status-check policy false.
    • Queue settings: ALLGREEN, SQUASH, 60-minute timeout, max_entries_to_build=5, min_entries_to_merge=1, max_entries_to_merge=5, and five-minute minimum-group wait. Ordinary required approving-review count is zero; code-owner, last-push approval and thread-resolution requirements are false.
    Required context Workflow/job PR and queue coverage
    Unit Tests ci.yml:57–89, unit-tests pull_request; merge_group: checks_requested
    E2E Tests ci.yml:91–106, e2e-tests Same
    Plugin Tests ci.yml:170–199, plugin-tests Same
    Check Issue Reference pr-check.yml:43–87, check-issue-reference pull_request; merge_group: checks_requested
    Check Issue Has status:approved pr-check.yml:89–151, check-issue-approved Same; depends on reference job
    Check PR Has type:* Label pr-label-check.yml:13–63, check-label-policy pull_request_target; merge_group: checks_requested

    Metadata validation uses trusted event-base helpers and current queued PR metadata. That does not make every inline workflow definition immutable. PR Validation omits reopened; changing the approval label on a linked issue alone does not refresh the checks.

    Observed execution and limits

    • feat(review): add advisory pull request size notice #1750 entered the queue and merged; all six required queue jobs passed: CI, Validation, Labels. Lint, Windows jobs and Performance Ratchet skipped there by design.
    • Bounded samples: latest 20 CI runs had 17 successes, one failure and two running; latest 20 PR Validation runs passed; latest 12 label runs had ten successes and two cancellations. CI failure 37979438310 was only non-required Lint (unchecked resp.Body.Close); a later revision passed. Validation 37897717217 correctly rejected fix(mcp): validate updates against the expected owner #1725's missing closing reference, then skipped dependent approval. One label cancellation was explicitly superseded by successful same-SHA runs; the other cause was not inspected. These samples are not a reliability guarantee.
    • Classic branch-protection GET returned 404; absent/null bypass actors do not establish no bypass. Those boundaries remain unknown. An accidentally queried wrong REST route was discarded as evidence.

    Proposed follow-ups, not approved changes

    1. Correct the stale Merge Queue Activation documentation, which proposes one build/one PR despite the active five-entry limits, and align test descriptions.
    2. Scope validation-refresh fixes separately, covering reopened PRs and linked-issue approval changes without replacing existing queue compatibility.
    3. Keep PR-size and AI output advisory. A successful size-notice job does not enforce the size budget. CodeRabbit's observed review uses Repository UI configuration; the file proposed in feat(review): version advisory CodeRabbit policy #1741 is absent from the pinned root. Copilot automatic review is also configured in the effective rules (review_on_push=true, drafts excluded), distinct from a required status gate. Resolve that duplication explicitly under the one-advisory-reviewer policy.
    4. Keep feat(ci): review introduced dependency vulnerabilities on pull requests #1734 separate: its inherited SBOM404/compare403 prerequisites, representative clean/vulnerable results and queue coverage remain unproven. Stage any future dependency review as non-required before considering enforcement.
    5. Any settings migration needs explicit owner/admin approval and an authorized admin executor. Rollback should remove only a newly added requirement, preserving the existing six gates.

    No rulesets, required checks, bypasses, source files, labels or issue state were changed by this audit. No builds/tests or forced workflow runs were performed. The issue remains open for owner decisions and missing evidence.

  3. dnlrsls commented on Oct 12, 2026

    @dnlrsls
    MemberAuthor

    Audit completion: retain the six existing required contexts

    The recommendation is to keep current enforcement unchanged. This closes the audit-only scope, not approval of new gates, implementation or administrator settings changes. The approval-event refresh is tracked separately in #1802.

    Sources are pinned to d134a555; effective main rules were freshly read on October 12, 2026.

    Required context Workflow / job Coverage
    Unit Tests ci.yml / unit-tests pull_request, merge_group: checks_requested
    E2E Tests ci.yml / e2e-tests Same
    Plugin Tests ci.yml / plugin-tests Same
    Check Issue Reference pr-check.yml / check-issue-reference pull_request, merge_group: checks_requested
    Check Issue Has status:approved pr-check.yml / check-issue-approved Same; depends on reference validation
    Check PR Has type:* Label pr-label-check.yml / check-label-policy pull_request_target, merge_group: checks_requested

    No rulesets, bypasses, workflows, required checks, repository permissions or app settings were changed. No local tests/builds or forced workflow runs were needed for this read-only refresh. Closing the audit does not close those separate follow-ups or claim missing rollout proof.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status:needs-reviewNeeds maintainer reviewtype:questionTracked question requiring maintainer investigation or a durable decision

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions