Skip to content

Add THIRD_PARTY_NOTICES.md and tooling to generate it - #2723

Open
abrarshivani wants to merge 1 commit into
NVIDIA:mainfrom
abrarshivani:third-party-notices
Open

Add THIRD_PARTY_NOTICES.md and tooling to generate it#2723
abrarshivani wants to merge 1 commit into
NVIDIA:mainfrom
abrarshivani:third-party-notices

Conversation

@abrarshivani

@abrarshivani abrarshivani commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Adds THIRD_PARTY_NOTICES.md and tools/generate-notices.sh, the script that
produces it, so the Go deps behind the released artifacts have their licenses
listed in the repo.

  • make notices regenerates it
  • make notices-check regenerates and diffs it, and runs in CI on every build

What to review

98% of the diff is generated. Hand-written, 715 lines across 8 files:

File Lines
tools/generate-notices.sh 523
.github/workflows/release-third-party-notices.yaml 94
.github/workflows/notices-check.yaml 62
Makefile 19
tools/go.mod 7
.github/workflows/ci.yaml 6
.gitignore 3
tools/tools.go 1

tools/generate-notices.sh is the one to read: named functions behind a main(),
about 120 lines of it comments. tools/go.mod pins go-licenses v2.0.1, and
.gitignore covers /.licenses-cache, the generator's scratch tree. Generated:
THIRD_PARTY_NOTICES.md (27,778 lines) and tools/go.sum (97). The two index
tables at the top of the notices file are the summary; the rest is verbatim
license text.

How it works

Needs Go (from versions.mk) and go-licenses, which make install-tools puts
in ./bin from the pin in tools/go.mod. The runtime pass reads vendor/ and
needs no network; the toolchain pass uses the module cache.

  1. Use ./bin/go-licenses ahead of $PATH so the pinned version runs
  2. Check PLATFORMS against DOCKER_BUILD_PLATFORM_OPTIONS in multi-arch.mk,
    fail if out of sync
  3. Run go-licenses save and csv once per platform over ./cmd/... with
    GOFLAGS=-mod=vendor and CGO_ENABLED=0, then merge. Platforms are
    linux/amd64 and linux/arm64.
  4. Same for the toolchain in tools/go.mod over four platforms including
    darwin, reading the tool list from tools/tools.go so it stays in step with
    make install-tools
  5. Sort whole lines, then join all licenses per package instead of picking one
  6. Add module@version by longest-prefix match against vendor/modules.txt
  7. Render the header, an index table per surface, then each license verbatim,
    fenced wide enough to survive licenses that are themselves Markdown
  8. Write to a temp file, then move it into place

To add a dep: change go.mod, go mod vendor, make notices, commit. CI fails
the PR if you forget.

Implementation notes

  • Run per platform and merged. go-licenses resolves only the host's
    graph, and build-tagged sources pull different deps per platform.
  • Only the local module goes to --ignore. It matches raw string prefixes,
    not path segments, so a stdlib list from go list std | cut -d/ -f1 yields
    the bare token go and drops golang.org/x/*, google.golang.org/*,
    gopkg.in/* and go.uber.org/*: 18 runtime and 14 toolchain modules.
    go-licenses already excludes the stdlib.
  • Licenses are joined, not picked. go-licenses emits one row per license,
    so collapsing on the package field loses the rest: filepath-securejoin would
    show as BSD-3-Clause without its MPL-2.0. Which one survived was unstable too,
    since a keyed sort -u compares only that key and BSD and GNU sort break ties
    differently.
  • Output must match on every machine, since CI diffs it. Whole-line
    sort -u, not sort -t, -k1,1 -u. LC_ALL=C on every sort and grep, or the
    filename match misses LICENSE under a Turkish locale. The awk counts instead
    of testing in, which mawk and gawk evaluate differently, and mawk is
    /usr/bin/awk on Debian and Ubuntu. grep -a so a NUL byte cannot change the
    fence width.
  • Vendored deps use module@version from vendor/modules.txt, since
    go-licenses in vendor mode reports a HEAD URL into this repo that stops
    describing released content once main moves. No resolved version fails the
    run.
  • mv, not cp. cp truncates the target first, so an interrupted run
    leaves a half-written file.

Scope

Covers ./cmd/..., not ./.... CMDS is a wildcard over ./cmd/* and the
Dockerfile builds by running make cmds, so this tracks the build rather than a
hand-kept list. Against ./... the only extras are ginkgo and gomega from
the never-shipped e2e suite. gpuop-cfg is scanned even though it is a build
helper not copied into the image; listing slightly more than ships is the safe
direction.

Not covered: the non-Go contents of the image, meaning the base image packages,
the BusyBox tree, the CUDA sample and the CUDA compat libraries. Base image deps
go through that image's own compliance process, the rest through separate flows
including any source-distribution obligations. The generated file says so in its
header.

Shipping

Not copied into the runtime image, so image size is unchanged. It ships as a
GitHub Release asset: release-third-party-notices.yaml triggers on
release: published, checks out the tagged commit and uploads the committed
file, failing if it is missing or empty. Nothing is regenerated at release time,
since notices-check already proves the file matches the tree. It follows
release-image-list.yaml, and also takes workflow_dispatch with a tag for
backfilling an existing release.

notices-check runs on every build, with no changed-paths filter. The runtime
list is the set of packages ./cmd/... imports, so it goes stale when a .go
file changes its imports, not only when go.mod or vendor/ move. A filter
would pass green while the file rotted, and the failure would surface on main
instead of on the PR that caused it.

Testing

Current output: 124 runtime packages, 74 build toolchain packages, 1.44 MB.
Index rows are per package; one module can own several.

  • Same sha256 on macOS (BSD userland) and in golang:1.26.5 (GNU coreutils,
    gawk). Two runs on each host also match.
  • Completeness against go list -deps ./cmd/... for both released platforms,
    mapped to license-owning dirs: 124 expected, 124 present, 0 missing, 0 extra.
  • Every index row has a license-text section. 0 Unknown licenses, 0 "License
    text unavailable", 0 unresolved module versions.
  • Multi-licensed deps checked against the actual license files:
    filepath-securejoin (BSD-3-Clause / MPL-2.0), klauspost/compress,
    sigs.k8s.io/yaml, sigs.k8s.io/json.
  • make notices-check passes on the committed file, fails when stale, and fails
    when untracked. git diff reports nothing for an untracked path, so that gate
    would otherwise pass silently.
  • The CI job has run and passed (1m32s), which also confirms a file generated on
    macOS reproduces byte-identically on the Linux runner.
  • Error paths exit 1 with useful messages: missing input file, out-of-sync
    platform matrix, a filesystem replace in vendor/modules.txt, and an
    unreadable module list. A versioned replace correctly uses the
    replacement.
  • shellcheck clean on the generator. actionlint and yamllint clean on both
    workflows.

Comment thread .github/workflows/release-third-party-notices.yaml

@rahulait rahulait left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread tools/generate-notices.sh
Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants