Skip to content

chore(models): sync Command Code catalog - #166

Closed
github-actions[bot] wants to merge 1 commit into
mainfrom
automation/commandcode-catalog
Closed

github-actions[bot] wants to merge 1 commit into
mainfrom
automation/commandcode-catalog

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Automated synchronization with the latest published command-code CLI package.

This updates only machine-readable compatibility metadata:

  • CLI version used in the x-command-code-version header
  • image-input capabilities
  • reasoning capability and selectable effort levels
  • model-specific maximum output limits
  • documented catalog snapshot version

Pricing remains review-only because CLI documentation does not represent every pricing tier and temporary promotion used by the provider.

@karaaslanz karaaslanz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent source-to-official-package review at HEAD 47ba19cb629e2b47cdfe2e01deab10e44b930f59.

I downloaded the published command-code@1.82.5 npm tarball with npm pack --ignore-scripts (no package scripts or downloaded CLI code executed), verified its SHA-512 integrity, read its model reference and dist/cli.mjs as text, and independently rederived the four metadata maps used by the catalog sync tool. Compared against the pinned GitHub src/commandcode-catalog.ts blob d92effa0e33cf06e4dc83abfe823a9992c514e5d.

PASS: 92 model IDs, zero mismatches: image input 70/70, reasoning 79/79, reasoning efforts and value arrays 64/64, max output tokens 10/10. The PR changes only README.md and the generated catalog; pricing files were not touched. Reproducible probe SHA-256: 22abb7572208f28138782082980310c6d348e1d229bd7b1b90e5b74c79f71b4a (pi-166-official-npm-metadata-consistency-probe.cjs). Scope is published CLI metadata consistency, not live endpoint, pricing, or full runtime tests.

CI gate: the PR workflows currently report action_required rather than success. I am recording this as a focused COMMENTED verification, not a full PR approval or merge recommendation until CI is resolved. No upstream source changes or lifecycle script execution.

@github-actions

Copy link
Copy Markdown
Contributor Author

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@github-actions

Copy link
Copy Markdown
Contributor Author

Memory benchmark

Compared base 3e43a7c with PR head 47ba19c on the same GitHub-hosted darwin runner. Lower values are better.

Metric Base PR PR − Base Change
Stable RSS 121.1 MiB ± 0.4 MiB 119.8 MiB ± 0.5 MiB -1.2 MiB ± 0.5 MiB -1.0%
Physical footprint 67.2 MiB ± 0.5 MiB 66.4 MiB ± 0.5 MiB -0.5 MiB ± 0.6 MiB -0.7%
Physical peak 78.4 MiB ± 1.2 MiB 76.9 MiB ± 0.5 MiB -1.5 MiB ± 0.4 MiB -1.9%

Interpretation: Differences larger than their measured MAD: Stable RSS -1.2 MiB ± 0.5 MiB, Physical peak -1.5 MiB ± 0.4 MiB.

Extension overhead above pi baseline

Metric Base overhead PR overhead Difference
Stable RSS +6.1 MiB ± 1.2 MiB +5.1 MiB ± 1.3 MiB -1.2 MiB ± 0.5 MiB
Physical footprint +3.6 MiB ± 0.7 MiB +3.5 MiB ± 0.6 MiB -0.5 MiB ± 0.6 MiB
Physical peak +4.8 MiB ± 1.0 MiB +3.1 MiB ± 1.9 MiB -1.5 MiB ± 0.4 MiB

Values are medians of 6 alternating, paired runs. The value after ± is the median absolute deviation (MAD). Each process was sampled 12 times after a 2500 ms warm-up.

Environment: pi 0.84.4, Bun 1.4.0, darwin arm64. RSS comes from ps; physical footprint and peak come from macOS footprint.

This is a comparative signal, not a pass/fail threshold. GitHub-hosted runner noise can affect absolute values.

@pierreraby pierreraby left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — head 47ba19cb (automated catalog sync, 2 files)

Verified independently at this exact head.

Live re-derivation with the repository's own tool. npm run check:commandcode-catalog at this head inspects the published command-code@1.82.5 package and compares it against the committed snapshot: no drift on any map — CLI version current, image-capable models 70/70, reasoning 79/79, selectable efforts 64/64, model-specific output limits 10/10, and None for every New/Removed/Changed row.

Local suites. npm ci --ignore-scripts, npm run typecheck, npm run format:check and git diff --check clean; full npm test green, including the real-host suites ([pi-local] PASS on pi 1.3.0, [omp-compat] PASS).

CI on this head is green after the pending runs were approved: 14 success, 1 skipped (CodeQL, Semgrep OSS + pi extension audit, Gitleaks, npm audit, dependency review, lifecycle-script check, typecheck, format, catalog sync, memory benchmark, OMP compatibility), mergeable_state: clean.

Scope of the change — only README.md and the generated src/commandcode-catalog.ts; pricing is intentionally untouched:

  • CLI version 1.82.4 → 1.82.5
  • new image modality: meta/muse-spark-open, nvidia/nemotron-3.5-super-vl
  • new reasoning models: the two above plus stealth/cat-jam:free; removed: inclusionai/ling-3.0-flash-sante:free
  • new reasoning efforts: low/medium/high for the two models, low/medium/high/xhigh/max for stealth/cat-jam:free
  • output limits: stealth/cat-jam:free 131,072 added, inclusionai/ling-3.0-flash-sante:free 32,768 removed

I also read @karaaslanz's independent source-to-official-package verification on this same head (npm tarball fetched with --ignore-scripts, SHA-512 verified, metadata rederived from the package as text: 92 model IDs, zero mismatches, reproducible probe SHA-256 22abb7572208f28138782082980310c6d348e1d229bd7b1b90e5b74c79f71b4a) — it agrees with the live re-derivation above, so the snapshot is confirmed by two independent methods.

Approving. Noted for the record, none of these block:

  1. No CHANGELOG.md entry. CONTRIBUTING reserves the changelog for user-facing behavior; a dropped reasoning flag for a free model and a new 128k-output free model (stealth/cat-jam:free) are arguably user-visible, so a line either here or in the release PR would be welcome.
  2. The verification scope is published-CLI metadata consistency, not live model availability or endpoint behavior — stealth/cat-jam:free still has to be reachable for the accounts allowed to use it.
  3. The automation branch is force-pushed by the daily job, so any future head invalidates this review and needs a fresh check.

Independent live re-derivation plus the full local suite at the exact head; struck at 47ba19cb629e2b47cdfe2e01deab10e44b930f59.

@pierreraby

Copy link
Copy Markdown
Collaborator

Thanks for the independent source-to-official-package check, @karaaslanz.

The three runs that were sitting in action_required have now been approved, and CI is green on this head (47ba19cb, 14 success / 1 skipped, mergeable_state: clean). I also re-derived the four metadata maps live with the repository's own check:commandcode-catalog tool against the published command-code@1.82.5 package at this exact head: zero drift everywhere (image 70/70, reasoning 79/79, efforts 64/64, output limits 10/10), which matches your probe result. Full local npm test is green too, including the real-host pi 1.3.0 and OMP suites.

Since your review was scoped to CI being unresolved, would you like to convert it into an approval if you are satisfied with the head? No merge is pending from my side — that stays with patlux.

@karaaslanz karaaslanz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up to my source-to-official-npm-package review on unchanged head 47ba19cb629e2b47cdfe2e01deab10e44b930f59.

I have now rechecked the live GitHub PR status after the previously blocked Actions jobs were approved. HEAD is unchanged, mergeable=MERGEABLE, mergeStateStatus=CLEAN, and current checks show 14 SUCCESS / 1 expected SKIPPED, including typecheck, formatting, dependency review, lifecycle-script checks, CodeQL, Semgrep, secrets scan, memory benchmark, OMP compatibility and catalog sync reporting.

My own previously recorded reproducible verification remains valid at this exact head: published command-code@1.82.5 tarball SHA-512 verified, 92 model IDs, zero divergence in the generated image (70/70), reasoning (79/79), effort values (64/64) and max-output (10/10) metadata; no pricing changes. @pierreraby additionally reports an independently successful npm test run, including Pi 1.3.0/OMP real-host suites, and independently ran the repository's check:commandcode-catalog without drift. Their full-suite run is their evidence; my source-to-package test is my own.

Given these new results, the limited earlier CI caveat is resolved and I approve the current head as a two-file, machine-generated model catalog snapshot update. This is not a claim to have live-tested availability/pricing for every model. Merge and release remain with @patlux, and a future automation force-push should trigger fresh validation.

@pierreraby

Copy link
Copy Markdown
Collaborator

Closing as superseded by #167, now merged (db1781c).

#167 carries exactly this PR's catalog metadata (CLI 1.82.5 → 1.83.0 entry set is identical — only the pinned version string differed) plus the reviewed pricing snapshot and its tests, so we merged that one instead of landing both.

Credit where it's due: the 1.82.5 metadata delta was independently verified by @karaaslanz (tarball SHA-512, 92 model ids, zero mismatch) and approved in this PR — that review stands as the validation of the catalog content that shipped via #167. Thanks Ömer!

@pierreraby pierreraby closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants