Skip to content

ci(models): add deduplicated maintenance alerts and release triage - #163

Open
patlux wants to merge 2 commits into
mainfrom
catalog-maintenance-alerts
Open

patlux wants to merge 2 commits into
mainfrom
catalog-maintenance-alerts

Conversation

@patlux

@patlux patlux commented Oct 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • Create/reopen one bot-owned maintenance issue after two consecutive failed scheduled/manual default-branch catalog runs; update rather than spam, and close after recovery. Also supports the weekly latest-host workflow from fix(core): preserve OMP reasoning and add host protocol contracts #162.
  • Guard fork/PR/non-default/stale runs; use only trusted default-branch code and workflow metadata, never run artifacts/log contents. No dependency installation in the privileged alert job.
  • Preserve parser/pricing stderr in separate reports, avoid missing-report masking and limit formatter scope to generated catalog files.
  • Document weekly triage, release-readiness evidence, migration notices and the distinction between merged and released. Publication remains separately authorized through the existing tag/OIDC process.

Verification

  • 8 mocked GitHub API tests covering threshold, dedup/reopen, recovery, stale runs, untrusted runs and issue ownership: PASS.
  • Full test suite with real Pi 1.1.0 and OMP 18.11.1 against mock APIs: PASS.
  • Typecheck, format, diff checks, actionlint 1.7.7: PASS.

No real alert issue created during tests; no release, permission/settings changes or merge performed.

@patlux patlux self-assigned this Oct 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 2 package(s) with unknown licenses.
See the Details below.

License Issues

.github/workflows/maintenance-alert.yml

PackageVersionLicenseIssue Type
actions/checkout4.*.*NullUnknown License
actions/github-script7.*.*NullUnknown License
Allowed Licenses: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, 0BSD

OpenSSF Scorecard

PackageVersionScoreDetails
actions/actions/checkout 4.*.* 🟢 6.3
Details
CheckScoreReason
Code-Review🟢 10all changesets reviewed
Maintained🟢 45 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 4
Binary-Artifacts🟢 10no binaries found in the repo
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Pinned-Dependencies🟢 3dependency not pinned by hash detected -- score normalized to 3
Fuzzing⚠️ 0project is not fuzzed
Packaging⚠️ -1packaging workflow not detected
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 9security policy file detected
SAST🟢 10SAST tool is run on all commits
Branch-Protection🟢 6branch protection is not maximal on development and all release branches
actions/actions/github-script 7.*.* 🟢 6.6
Details
CheckScoreReason
Binary-Artifacts🟢 10no binaries found in the repo
Maintained⚠️ 00 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0
Code-Review🟢 10all changesets reviewed
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Token-Permissions🟢 9detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies⚠️ 1dependency not pinned by hash detected -- score normalized to 1
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
License🟢 10license file detected
Fuzzing⚠️ 0project is not fuzzed
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 9security policy file detected
SAST🟢 10SAST tool is run on all commits
Branch-Protection🟢 5branch protection is not maximal on development and all release branches

Scanned Files

  • .github/workflows/maintenance-alert.yml

@github-actions

Copy link
Copy Markdown
Contributor

Memory benchmark

Compared base 3e43a7c with PR head bcb6e99 on the same GitHub-hosted darwin runner. Lower values are better.

Metric Base PR PR − Base Change
Stable RSS 120.6 MiB ± 0.7 MiB 120.2 MiB ± 0.3 MiB -0.5 MiB ± 1.1 MiB -0.4%
Physical footprint 66.2 MiB ± 0.6 MiB 66.3 MiB ± 0.3 MiB +0.5 MiB ± 0.8 MiB +0.7%
Physical peak 77.5 MiB ± 0.5 MiB 77.7 MiB ± 0.7 MiB +0.0 MiB ± 1.0 MiB +0.0%

Interpretation: No metric shows a clear Base-to-PR difference beyond its measured run-to-run variation.

Extension overhead above pi baseline

Metric Base overhead PR overhead Difference
Stable RSS +7.3 MiB ± 0.6 MiB +7.0 MiB ± 1.3 MiB -0.5 MiB ± 1.1 MiB
Physical footprint +3.5 MiB ± 0.5 MiB +3.7 MiB ± 1.2 MiB +0.5 MiB ± 0.8 MiB
Physical peak +3.4 MiB ± 0.6 MiB +4.0 MiB ± 1.2 MiB +0.0 MiB ± 1.0 MiB

Values are medians of 6 alternating, paired runs. The value after ± is the median absolute deviation (MAD). Each process was sampled 12 times after a 2500 ms warm-up.

Environment: pi 0.84.4, Bun 1.4.0, darwin arm64. RSS comes from ps; physical footprint and peak come from macOS footprint.

This is a comparative signal, not a pass/fail threshold. GitHub-hosted runner noise can affect absolute values.

@patlux patlux left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The trust model of the alert workflow (default-branch code only, no artifacts/logs, job-level if + script-level checks) is right. Comments inline are about robustness of the threshold logic, API volume, and two nits.

Two cross-PR notes:

  • workflows: [Command Code catalog sync, Latest host compatibility] — the second only exists after #162. Harmless (GitHub just never triggers), but worth mentioning in the merge order.
  • The tracked map in the script and the workflows: list in the yml are both keyed by the workflow name: strings. A rename in either model-metadata.yml or latest-hosts.yml silently disables alerts. A comment at all three places would help.

status: "completed",
per_page: 100,
})
const eligible = recent

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threshold relies on listWorkflowRuns already containing the triggering run as completed. workflow_run.completed can fire before the listing API reflects that (eventual consistency). If the current run is missing, eligible[0]/eligible[1] are the two previous runs and the decision is off by one in either direction.

Make it explicit: treat run as the head and take the newest eligible item with run_number < run.run_number as the predecessor, independent of whether the API already lists run.

)
)
return "stale"
const issues = await github.paginate(github.rest.issues.listForRepo, {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

state: "all" + creator paginates every bot-created issue forever. Add a dedicated label (e.g. maintenance-alert) on create and filter by labels here — cheaper and visible to humans in the issue list.

Same idea for listWorkflowRuns above: you only need runs up to the predecessor; github.paginate supports an early done() callback.

issue_number: issue.number,
state: "closed",
state_reason: "completed",
body: `${marker}\n\nRecovered in [run ${run.run_number}](https://github.com/${repo.owner}/${repo.repo}/actions/runs/${run.id}).\n\n${config.action}`,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recovery overwrites the body, so the failing-run links disappear from the issue once it is closed. A short comment ("Recovered in run N") plus close keeps the history — that is one comment per incident, not spam.

- name: Compare with the latest Command Code CLI
shell: bash
run: npm run check:commandcode-catalog | tee commandcode-catalog-report.md
run: npm run check:commandcode-catalog 2>&1 | tee commandcode-catalog-report.md

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2>&1 into the Markdown report also captures npm's own stderr (npm warn, the npm error banner on failure) into the step summary. If the intent is to keep the script's stderr, run tsx directly or use npm run --silent.

Comment thread CHANGELOG.md

## Unreleased

- Open one actionable maintenance issue after repeated scheduled catalog/latest-host failures, update it without comment spam and close it after recovery. Keep report stderr, avoid broad formatting in catalog automation and document weekly release triage without automating publication.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI/maintainer tooling is not user-facing; CONTRIBUTING limits the changelog to user-facing behavior. I'd drop the entry and let RELEASE.md be the record.

@pierreraby pierreraby left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Verified locally at the exact head in an isolated worktree (npm ci): tsc --noEmit PASS, test-maintenance-alert 8/8, full test:unit exit 0 on re-run (401 checks; first run hit a timing flake in the unrelated test-abort mid-stream test, which passes in isolation and is untouched by this diff), Prettier + git diff --check PASS. I also had this head independently reviewed in a fresh context (APPROVE, no P0/P1): threshold/dedup/reopen/recovery logic correct (2 consecutive failures, update-not-spam, close-on-recovery), fork/PR/stale guards double-enforced (workflow if + script re-checks), trusted default-branch code only, minimal permissions, tests non-vacuous.

Non-blocking follow-ups from that review: (1) per-item head_branch re-check in eligible/stale guard (currently relies on the list query param, which the test fixture ignores); (2) assert the created body starts with the dedup marker; (3) pin the tracked workflow names against .github/workflows/*.yml so a rename cannot silently disable alerts. None block merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants