Skip to content

feat: TEM-27 remove FileBrowser, patch HIGH/CRITICAL vulns, and harden build/release CI - #154

Merged
Chmokachka merged 18 commits into
mainfrom
feat/TEM-27-upgrade-filebrowser
Aug 19, 2026
Merged

feat: TEM-27 remove FileBrowser, patch HIGH/CRITICAL vulns, and harden build/release CI#154
Chmokachka merged 18 commits into
mainfrom
feat/TEM-27-upgrade-filebrowser

Conversation

@Chmokachka

@Chmokachka Chmokachka commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

TEM-27. Removes the unused FileBrowser binary from runpod/base, patches the outstanding CRITICAL/HIGH vulnerabilities across our image families, turns Grype into a hard gate on the families we've cleaned up, and hardens the build/release pipeline so a flaky build can no longer silently skip a release or mis-bump the version.

FileBrowser is being archived on Sept 1st by upstream and ships with known, will-not-fix security advisories. We installed the binary in base but never started it (only comfyui-base runs it), so it was pure attack surface with no benefit — dropped it.

Changes

FileBrowser

  • base: remove the FileBrowser install step entirely (binary was never launched by start.sh, so no runtime behavior changes).
  • pytorch-cluster: drop the stale comment that port 8080 / FileBrowser is inherited from base.

Vulnerability fixes

  • pytorch: re-install a patched Pillow from PyPI over the CVE-affected copy torchvision drags in from the PyTorch wheel index. Version is centralized as PILLOW_VERSION in docker-bake.hcl and only applied when Pillow is actually present (torch-only builds are untouched).
  • pytorch-cluster: bump node_exporter 1.11.1 → 1.12.1 and Grafana 13.1.0 → 13.1.1 (Go stdlib / dependency CVEs in the embedded binaries).
  • autoresearch: bump runpodctl v2.3.0 → v2.7.2 (Go stdlib / x-crypto CVEs).
  • nvidia-pytorch: jupyterlab 4.5.9 → 4.5.10, mistune 3.2.1 → 3.3.0, add msgpack==1.2.1, add soupsieve==2.8.4, pillow 12.2.0 → 12.3.0, aiohttp 3.14.1 → 3.14.3.
  • rocm: jupyterlab 4.5.9 → 4.5.10, add msgpack==1.2.1, pillow 12.2.0 → 12.3.0.
  • base: jupyterlab 4.5.9 → 4.5.10.

CI - vulnerability gating

  • grype: add a fail-on-findings input (default false, report-only). Enabled (true) for the cleaned-up families: base, pytorch, nvidia. Other families stay report-only until their remaining findings are resolved.

CI - build cache

  • docker-setup: upgrade useblacksmith/setup-docker-builder v1 → v2.0.1 and add a required cache-key input. Each image family uses a distinct BuildKit sticky-disk lineage; parallel pytorch/cluster CUDA shards each get their own per-shard key (…/pytorch-<cuda>, …/cluster-<cuda>). This also fixes ROCm "no space left on device" failures.
  • PR vs release lineages: pull-request and off-main jobs append /pr to the cache key so they cannot commit into the sticky-disk lineage used by main / release builds. Enabling Blacksmith sticky-disk Branch Protection in the dashboard is still recommended.

CI - release correctness

  • compute-version: feat/fix/perf/none come from the squash subject (PR title) only, so a ci:/chore: squash whose body lists * feat: branch commits no longer phantom-bumps. A major bump is type!: in the title, or a git-trailer BREAKING CHANGE: / BREAKING-CHANGE: footer (own line, with a colon) in the description. Convention: mark breaking PRs with ! in the title and preferably add the footer too; the footer alone is still major. Documented in docs/RELEASE.md; detect-bump.sh is covered by .github/workflows/compute-version.yml.
  • manual-release (break-glass): workflow_dispatch that creates the git tag + GitHub Release for an already-built-and-pushed version, without rebuilding. Recovers the case where a flaky build left release stuck in skipped. Requires tag and source_run_id (the original Build and Release run). The tag is created on that run's commit, not on current main. Guards: dispatch from main only; the source run must be release.yml + push to main; every job except release/changes/cleanup must have succeeded and at least one family build ran; the SHA must still be on main; compute-version at that SHA must equal the requested tag; one canary image per family must resolve in the registry (docker buildx imagetools inspect); the git tag must not already exist. Shares the push-release concurrency lane.
  • base.yml job graph: build-pytorch, build-autoresearch, and build-cluster run only when their parent job is success. A skipped pytorch job (e.g. because build-base failed) no longer starts cluster against a missing or stale pytorch.

Follow-up

OCI image labels (org.opencontainers.image.version, org.opencontainers.image.revision, …) land in #147 (TEM-11). After a labeled version is in the registry, the manual-release inspect step can also assert image.version against the computed version and image.revision against the source-run SHA.

Notes

  • pytorch-cluster (dcgm-exporter), rocm (base OpenSSL/Python), and autoresearch (pillow/urllib3 via its own uv.lock) still have findings that depend on upstream releases — those families remain report-only for now.

@Chmokachka Chmokachka changed the title feat: TEM-27 upgrade filebrowser feat: TEM-27 upgrade filebrowser, jupyterlab Jul 27, 2026
@Chmokachka Chmokachka changed the title feat: TEM-27 upgrade filebrowser, jupyterlab feat: TEM-27 get rid of filebrowser and upgrade jupyterlab Jul 29, 2026
@Chmokachka Chmokachka changed the title feat: TEM-27 get rid of filebrowser and upgrade jupyterlab feat: TEM-27 remove filebrowser + patch HIGH/CRITICAL vulns + harden CI Jul 29, 2026
@Chmokachka Chmokachka changed the title feat: TEM-27 remove filebrowser + patch HIGH/CRITICAL vulns + harden CI feat: TEM-27 remove FileBrowser, patch HIGH/CRITICAL vulns, and harden build/release CI Jul 30, 2026
@Chmokachka
Chmokachka marked this pull request as ready for review July 31, 2026 11:21
@kodxana

kodxana commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Thanks, I reviewed the current head (27b1d595) and the tested merge result (950aa0e). The normal image path has strong evidence: 33 hard-gated Grype image scans passed, and 39 functional smoke tests completed with 0 failures. The only runtime gap is ROCm, where all 3 images were skipped because MI300X capacity was unavailable.

I still see three issues that should be addressed before merge:

  1. The manual release workflow can tag a different commit from the one used to build the images.

    The [manual release workflow](https://github.com/runpod/containers/blob/27b1d595ec59cf5633725bc1f9c59096614259b5/.github/workflows/manual-release.yml) accepts only a tag and uses the current ${{ github.sha }} as target_commitish.

    If another change reaches main after the failed build, running the recovery workflow will tag the newer main commit even though the images were built from the older commit. The generated release notes can also include unrelated changes. There is currently no verification that the required image tags or digests actually exist.

    Please accept an original workflow run ID or commit SHA, verify that it belongs to main, verify the expected image manifests or digests, and create the tag against that exact commit.

  2. Subject-only version detection silently drops valid BREAKING CHANGE footers.

    The updated [compute-version action](https://github.com/runpod/containers/blob/27b1d595ec59cf5633725bc1f9c59096614259b5/.github/actions/compute-version/action.yml) passes only the commit subject to detect_bump. A commit such as:

    feat: replace the image interface
    
    BREAKING CHANGE: remove the previous interface
    

    will now produce a minor bump instead of a major bump. The [Conventional Commits specification](https://www.conventionalcommits.org/en/v1.0.0/) explicitly supports both a BREAKING CHANGE: footer and ! in the subject.

    Please either parse only a properly formed breaking-change footer, or explicitly require ! in the squash title and add tests documenting that requirement. The current PR run exercised the pull_request title path, not this changed push path.

  3. Pull request jobs are writing to the same sticky-disk cache lineages used by trusted release builds.

    The [exact merge-result run](https://github.com/runpod/containers/actions/runs/30608012364) shows the pull request jobs successfully committing the new runpod/containers/... sticky disks. [Blacksmith documents](https://docs.blacksmith.sh/blacksmith-caching/dependencies-sticky-disks) that pull request jobs can modify shared sticky disks by default and recommends enabling Branch Protection to prevent cache poisoning.

    Please enable and verify Blacksmith sticky-disk Branch Protection, or isolate pull request cache writes from the cache keys used by main and release builds.

Test gap: ROCm built successfully, but its functional result was 0 PASS, 0 FAIL, 3 SKIP. Since this PR changes the ROCm requirements, please rerun that smoke test when MI300X capacity is available, or explicitly record that the gap is accepted.

Minor cleanup: the [pytorch-cluster Dockerfile](https://github.com/runpod/containers/blob/27b1d595ec59cf5633725bc1f9c59096614259b5/official-templates/pytorch-cluster/Dockerfile) still says that port 8080 and FileBrowser are inherited from the base image. That comment should be removed or corrected with this change.

Keep the break-glass tag on the commit that built the images, honour a
BREAKING CHANGE footer without letting squash-body feat/fix lines bump,
and stop pull_request jobs committing into the release sticky-disk lineage.

Co-authored-by: Cursor <cursoragent@cursor.com>
@runpod runpod deleted a comment from mchekmasova-cbrands Aug 13, 2026
@kodxana

kodxana commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Thanks, I rechecked the latest head. The version detection, PR cache isolation, source-run commit handling, and FileBrowser cleanup are all fixed, and the current tests completed with 39 PASS and 0 FAIL.

One release-safety issue still remains. The manual release workflow accepts any semver tag, but it does not verify that the tag matches the version and images produced by the selected source run. A typo or mismatched tag could create a GitHub release for images that do not exist. Please derive or validate the expected tag from the source run and verify the corresponding image refs or manifests before creating the release.

@Madiator2011Work Madiator2011Work left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Chmokachka
Chmokachka merged commit aa6a729 into main Aug 19, 2026
102 of 105 checks passed
@Chmokachka
Chmokachka deleted the feat/TEM-27-upgrade-filebrowser branch August 19, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants