osquery extensions: verified, root-owned install with rollback and per-extension status

[javuto]

osquery extensions: verified, root-owned install with rollback and per-extension status

Companion to the osctrl change that adds an extension library, environment and host assignments, and the osctrld routes POST /{env}/osctrld-extensions (list) and POST /{env}/osctrld-extension (download). Builds on per-node flags overrides (#27).

Problem

osctrld's extension sync wasn't safe to run as root, and osquery couldn't load what it installed:

• It downloaded each extension from whatever URL the list contained, with a GET and no integrity check.
• It wrote files in place (it could corrupt a running binary) with no ownership change. osquery refuses to autoload extensions that aren't root-owned.
• It never wrote extensions.load, never removed extensions dropped from the list, and an empty list did nothing.
• It didn't tell osctrl the host's platform or architecture.
• A bad extension got no restart guard, rollback or status. The flags guard didn't cover extensions.

What changes

Protocol

• List request: {secret, uuid?, platform, arch, status?}. The UUID is the cached host UUID used by the flags step; platform/arch come from runtime.GOOS/GOARCH.
• List response: {name, sha256, size, url} per extension.
• Download: POST <osctrl base>/<env>/osctrld-extension with {secret, sha256}, built from osctrld's own configured base. The list's url is never used to choose where to connect.
• Compatibility: a 404 from an older osctrl still means "no extensions".

Safe install (extensions.go, owner_posix.go, owner_windows.go)

• Names: ^[A-Za-z0-9_-]{1,64}$. The file is <name>.ext, or <name>.exe on Windows.
• Download and verification:
  • Streamed into a temp file in the extensions directory, with a hard size cap (the listed size, at most 64 MB).
  • Size and lowercase SHA-256 are verified, then ownership is set, mode 0755, and the file is renamed over the target.
  • Nothing is ever written through a symlink or a non-regular file.
• Ownership:
  • POSIX: root-owned.
  • Windows: owner Administrators and a protected DACL (Administrators/SYSTEM full control, Users read and execute), set with SetNamedSecurityInfo.
  • The extensions directory gets the same treatment, and junctions are refused.
• Existing files: a file with the right hash is kept only if it's safely owned; otherwise it's downloaded again.

Extensions step (extguard.go, service.go) runs after the flags step, with its own restart and check.

• Autoload file:

  • osctrld manages only its own lines in osquery's autoload file and keeps lines added by hand.
  • The file is --extensions_autoload from the current flagfile if set, otherwise osquery's default. Either way it must be a *.load file directly in osquery's base directory, root-owned, and not the flagfile, secret, certificate or osctrld state files.
  • Refused paths are reported once. The file is reconciled every sync, so drift is repaired.

• Removal: only files osctrld installed are removed; operator files are never touched. Replaced or removed files are renamed aside (.prev) until verified, which also handles Windows file locks. Leftover temp files are swept.

• Verification: after the restart, within 60 s osqueryd must be running and each changed extension must have a running process from its installed path.

  • osqueryd down → the previous set is restored.
  • An extension not running → it's removed, or for an update the previous version is restored, and it's reported rejected.
  • Rejected content (by SHA-256) isn't re-applied while osctrl keeps serving it.
  • A removal that keeps osqueryd down is undone, remembered, and not retried until osctrl serves that name again.

• Interruptions: pending and restarted markers resume a check cut short by a crash, SIGTERM or a failed restart, restarting first when needed.

• Status per extension, sent on the next request:

  • ok;
  • rejected;
  • kept: a removal failed (osqueryd went down, the file couldn't be moved aside, or the autoload file couldn't be written). It is re-sent on every sync until osctrl serves that name again.
  • removed: removed successfully.
  • ok is also sent when osctrl serves a kept extension again at the version still installed.

  Errors are capped at 256 bytes, and the queue is deduplicated by name, newest kept, capped at 24 entries. The queue is dropped only on HTTP 400/413; it's kept on 401/403, 5xx and network errors.

• Disabled extensions: if the flags set --disable_extensions, files are still installed and the status says extensions are disabled.

Security and operational impact

• Integrity: osctrld runs only binaries whose size and SHA-256 match what osctrl recorded at upload, fetched from its own configured osctrl. They are root-owned, or admin-only writable on Windows.
• Autoload file: a flags edit can't make osctrld write to system files like /etc/sudoers or /etc/ld.so.preload.
• Bad extensions: a bad extension can't keep osquery down. It costs at most one extra restart and a rollback, and it's reported.
• Revoked extensions stay visible: a command-execution extension that can't be removed is reported kept on every sync, so osctrl keeps warning about it until it's gone.
• Timing: changes apply on the next service sync (default 60 minutes), up to two restarts when flags and extensions both change. The checks can take about 2 minutes in the worst case.
• State: <flagfile>.osctrld-extensions-state.json, mode 0600.
• Compatibility:
  • Against an older osctrl, the extensions route 404s, which still means "no extensions". An older osctrl that does serve the old route ignores kept and removed.
  • Older osctrld clients get an empty list from the new osctrl, and report failed removals as rejected, which osctrl doesn't flag.
• Known limits:
  • "Running" means a process with the installed path exists, which any local user could imitate; it affects the reported status only.
  • The Windows permissions and file-locking code compiles, is vetted and is unit-tested, but hasn't run on a real Windows host. Run an install once on Windows before releasing.

Validation

• go test ./... -count=1 passes, also with -race.
• go vet ./... is clean for GOOS=darwin, linux and windows; gofmt -l cmd prints nothing.
• golangci-lint run ./...: 0 issues, also with GOOS=windows.
• go build -o /dev/null ./cmd/osctrld: ok.
• New tests, each seen failing first:
  • Installer: hash/size mismatch, oversize, symlinks, unsafe matching files, the entry URL ignored, download 404, directory repair, Windows DACL construction.
  • Autoload: path rules (system files, outside the base, non-.load, protected files, symlinks), operator lines kept, drift repair, refusal reported once.
  • Removal of only osctrld's own files, including the empty-list case.
  • Process check: success, a failed extension, osqueryd down, update rollback keeps the previous version, rejected content skipped until it changes, rejected removals kept.
  • Resume with and without a restart, flags-then-extensions ordering.
  • Statuses: kept on every failure path and re-sent each sync, removed, ok when served again, caps, deduplication, drop only on 400/413.
  • FIFOs and non-regular files refused.
• Not exercised: osquery autoloading on a live host, and Windows on a real host.