Per-node flags overrides: host identity and a rollback guard for osquery restarts

[javuto]

Per-node flags overrides: host identity and a rollback guard for osquery restarts

Companion to the osctrl change that adds per-node osquery flags overrides. With it, an osctrl admin can give one host extra flags during an investigation: events, file integrity monitoring, process auditing and other startup-only settings. osctrl serves those flags only when osctrld says which host is asking. Because a bad flag stops osqueryd at startup, osctrld now also guards every restart it triggers.

Problem

• No host identity. osctrld's flags request carried only the environment secret, so osctrl had no way to pick flags for a specific host.
• No guard on restarts. osquery exits at startup on an unknown flag or an invalid value. The service sync wrote whatever flags it received and restarted osqueryd. One typo in an override would leave the host without osquery until someone fixed it in osctrl and the next sync ran, up to an hour later.

What changes

Host identity (service mode only)

• Lookup: the service sync reads the host's osquery UUID by running osqueryd -S --json "select uuid from system_info;". That is the identity osquery enrolls with (--host_identifier=uuid).
• Limits and caching: the lookup has a 15-second limit. The last good UUID is cached locally, so a failed lookup doesn't make syncs alternate between override and environment flags. With nothing cached, no UUID is sent and the host gets the environment flags.
• New request fields: the flags request gains two optional fields: uuid and apply (result = ok | rejected, sha256, error, at). apply reports the outcome of the last flags change.
• One-shot commands unchanged: install, verify and flags send neither field. Only the service loop can later remove an override, so only it may receive one.

Restart guard

• After a restart for changed flags: osctrld waits up to 60 s and needs to see osqueryd running on 3 consecutive polls. Requiring three polls catches a service manager respawning a crashing osqueryd.
• If osqueryd doesn't stay up:
  • osctrld asks for the flags again without uuid, which returns the environment's flags;
  • it writes them and restarts osqueryd;
  • it records the rejected content's SHA-256;
  • it reports rejected on the next sync.
• No restart loop:
  • While osctrl keeps serving the rejected content, osctrld applies the environment flags instead and never re-applies the rejected content.
  • As soon as the served flags change (override edited or removed, or environment flags changed), it tries again.
  • The rollback's own restart is not re-checked.
• Interrupted checks resume: a Pending hash is saved before new flags are written. If the check was cut short by a restart command error, SIGTERM, a crash or a reboot, the next sync runs it without restarting osqueryd again.
• Process check: osqueryd counts as running when a process is named osqueryd or osqueryd.exe. The old command-line substring check didn't work on Windows.

State file

• <flagfile>.osctrld-state.json holds the last apply status, the rejected hash, the pending hash and the cached host UUID. Mode 0600.
• A missing or corrupt file is treated as empty, and the sync carries on.

Security and operational impact

• Flags overrides need osctrld in service mode. Without this change, a host keeps getting its environment's flags.
• A bad override can't keep osquery down for long. The worst case is about a minute without osqueryd, then the environment flags are back.
• Rollback needs osctrl. If osctrl can't be reached at that moment, the rollback waits for the next successful sync.
• Compatibility both ways:
  • Against an older osctrl, the new fields are ignored and nothing changes.
  • A newer osctrl serves an older osctrld the environment flags, as before.
• The UUID and apply status are sent with the environment secret, like today's flags request. osctrl validates them and uses them only to choose the flags and to show the status.
• Known limits:
  • "Running" doesn't prove osqueryd loaded the new flags; for example, a failed restart command may leave the old process running.
  • An environment flags change that breaks osqueryd while an override is active is reported as the override's rejection.

Validation

• go test ./... -count=1 passes, also with -race.
• go vet ./... is clean for GOOS=darwin, linux and windows; gofmt -l cmd prints nothing.
• golangci-lint run ./...: 0 issues.
• go build -o /dev/null ./cmd/osctrld: ok.
• New tests, each seen failing first:
  • UUID lookup: success; command error, empty result, bad JSON or invalid value returning an error; timeout; cached fallback.
  • Request body: uuid and apply sent in service mode, and absent otherwise.
  • Guard: ok; rollback; restart error still checked; interrupted check resumed without a restart; rejected content skipped; retry when the served content changes; environment flags failing too.
  • State file: corrupt, missing, a directory in its place, 0600 mode.
  • Process-name matching.
  • End-to-end syncOnce cycles, including "a second cycle does nothing".
  • Error text follows the configured timeout.
• Not exercised: a real osqueryd rejecting a flag on a live host.