From 134fd43ae8c90055e5b134d509c385a5449744b4 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 14 Jul 2026 13:49:36 +1000 Subject: [PATCH 01/10] feat(cli): anonymous opt-out usage analytics + `stash telemetry` Add anonymous, opt-out usage analytics to the stash CLI, plus a `stash telemetry [status|enable|disable]` command to manage it. - Coarse events only (command, version, OS/arch, success, duration); a property allowlist at the emitter boundary makes it impossible for plaintext, schema, table/column names, connection strings, or argument values to leave the machine. - Opt-out by default, Homebrew-style: nothing is sent until the one-time first-run notice has been shown once. Four opt-out gates in precedence order: DO_NOT_TRACK, STASH_TELEMETRY_DISABLED, CI auto-detect, and the persisted `stash telemetry disable` flag in ~/.cipherstash/telemetry.json. - Events go through a first-party Cloudflare proxy (infra/telemetry-proxy), so a future US->EU PostHog move is a proxy-target change with no CLI re-release. posthog-node, events-only (no person profiles), lazy client, flush bounded by a timeout so it never blocks or slows the CLI. - Ships dormant: no events are sent until a real PostHog key is embedded at release (STASH_POSTHOG_KEY overrides for testing). Updates the stash-cli skill and adds a changeset. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- .changeset/cli-anonymous-telemetry.md | 21 ++ infra/telemetry-proxy/README.md | 49 ++++ infra/telemetry-proxy/worker.js | 46 ++++ infra/telemetry-proxy/wrangler.toml | 15 ++ packages/cli/src/bin/main.ts | 45 ++++ packages/cli/src/cli/registry.ts | 14 ++ packages/cli/src/commands/index.ts | 1 + packages/cli/src/commands/telemetry/index.ts | 64 +++++ packages/cli/src/messages.ts | 15 ++ .../cli/src/telemetry/__tests__/state.test.ts | 59 +++++ .../src/telemetry/__tests__/telemetry.test.ts | 160 ++++++++++++ packages/cli/src/telemetry/index.ts | 234 ++++++++++++++++++ packages/cli/src/telemetry/state.ts | 74 ++++++ skills/stash-cli/SKILL.md | 26 +- 14 files changed, 822 insertions(+), 1 deletion(-) create mode 100644 .changeset/cli-anonymous-telemetry.md create mode 100644 infra/telemetry-proxy/README.md create mode 100644 infra/telemetry-proxy/worker.js create mode 100644 infra/telemetry-proxy/wrangler.toml create mode 100644 packages/cli/src/commands/telemetry/index.ts create mode 100644 packages/cli/src/telemetry/__tests__/state.test.ts create mode 100644 packages/cli/src/telemetry/__tests__/telemetry.test.ts create mode 100644 packages/cli/src/telemetry/index.ts create mode 100644 packages/cli/src/telemetry/state.ts diff --git a/.changeset/cli-anonymous-telemetry.md b/.changeset/cli-anonymous-telemetry.md new file mode 100644 index 00000000..b3898aa9 --- /dev/null +++ b/.changeset/cli-anonymous-telemetry.md @@ -0,0 +1,21 @@ +--- +'stash': minor +--- + +Add anonymous, opt-out usage analytics to the `stash` CLI, plus a +`stash telemetry [status|enable|disable]` command to manage it. + +Only coarse events are collected — command name, CLI version, OS/arch, +success/failure, and duration. Plaintext, schema, table/column names, connection +strings, and argument values are never collected (enforced by a property +allowlist at the emitter boundary). A one-time notice is shown on first run, and +nothing is sent on that run. + +Telemetry is off by default in CI and can be disabled with `DO_NOT_TRACK=1` +(the cross-tool standard), `STASH_TELEMETRY_DISABLED=1`, or +`stash telemetry disable` (persisted to `~/.cipherstash/telemetry.json`). + +Events are sent via a first-party proxy and never block or slow the CLI. The +feature ships dormant — no events are sent until a PostHog project key is +embedded at release. Updates the `stash-cli` skill to document the command and +opt-out controls. diff --git a/infra/telemetry-proxy/README.md b/infra/telemetry-proxy/README.md new file mode 100644 index 00000000..b3dd7e1a --- /dev/null +++ b/infra/telemetry-proxy/README.md @@ -0,0 +1,49 @@ +# Stash telemetry proxy + +A Cloudflare Worker that receives the `stash` CLI's anonymous analytics at +`telemetry.cipherstash.com` and forwards them to PostHog. See `worker.js` for why +the CLI proxies instead of calling PostHog directly (first-party requests, and +so the US→EU migration needs no CLI re-release). + +## Deploy + +Requires [`wrangler`](https://developers.cloudflare.com/workers/wrangler/) and +access to the Cloudflare account that holds the `cipherstash.com` zone. + +```bash +cd infra/telemetry-proxy +wrangler deploy +``` + +The `custom_domain` route provisions `telemetry.cipherstash.com` and its TLS +certificate automatically on first deploy. + +## Verify + +```bash +# 404 for an unknown path (the Worker is not an open proxy) +curl -sS -o /dev/null -w '%{http_code}\n' https://telemetry.cipherstash.com/ + +# A capture POST is accepted and forwarded to PostHog +curl -sS -X POST https://telemetry.cipherstash.com/batch/ \ + -H 'content-type: application/json' \ + -d '{"api_key":"","batch":[]}' +``` + +## Moving to PostHog Cloud EU (future) + +The forward target is the `POSTHOG_HOST` var, not anything baked into the CLI. +To migrate the entire installed CLI fleet: + +1. Edit `POSTHOG_HOST` in `wrangler.toml` to `eu.i.posthog.com`. +2. `wrangler deploy`. + +No CLI change, no release. Every already-installed `stash` follows the proxy. + +## Wiring the CLI + +The CLI targets `https://telemetry.cipherstash.com` and carries the public, +write-only PostHog project key (embedded at release in +`packages/cli/src/telemetry/index.ts`; overridable with `STASH_POSTHOG_KEY` for +testing). Until a real key is embedded, CLI telemetry is dormant — no events are +sent regardless of this proxy. diff --git a/infra/telemetry-proxy/worker.js b/infra/telemetry-proxy/worker.js new file mode 100644 index 00000000..25676240 --- /dev/null +++ b/infra/telemetry-proxy/worker.js @@ -0,0 +1,46 @@ +/** + * CipherStash telemetry reverse proxy (Cloudflare Worker). + * + * The `stash` CLI sends anonymous analytics to `telemetry.cipherstash.com`, and + * this Worker forwards them to PostHog. Two reasons the CLI never talks to + * PostHog directly: + * + * 1. First-party + transparent — requests stay on our domain and sail through + * corporate firewall allowlists. + * 2. Indirection — a released CLI hard-codes whatever host it shipped with, + * forever. Because the target lives in the `POSTHOG_HOST` var, a future + * US→EU migration is a var change + redeploy here, with NO CLI re-release: + * every already-installed `stash` follows automatically. + * + * A bare DNS CNAME to PostHog can't do this — PostHog serves its own TLS cert + * and routes by Host header — so the domain must be terminated and rewritten, + * which is exactly what this Worker does. + */ + +// PostHog ingestion path prefixes. Restricting to these keeps the Worker from +// being a general-purpose open proxy while still covering posthog-node's +// batch/capture endpoints. +const ALLOWED_PREFIXES = ['/batch', '/capture', '/e', '/i/', '/flags', '/array'] + +export default { + async fetch(request, env) { + const url = new URL(request.url) + + if (request.method !== 'POST' && request.method !== 'GET') { + return new Response('Method Not Allowed', { status: 405 }) + } + if (!ALLOWED_PREFIXES.some((prefix) => url.pathname.startsWith(prefix))) { + return new Response('Not Found', { status: 404 }) + } + + const target = env.POSTHOG_HOST // "us.i.posthog.com" now; "eu.i.posthog.com" later + url.hostname = target + url.port = '' + url.protocol = 'https:' + + const forwarded = new Request(url, request) + forwarded.headers.set('Host', target) + + return fetch(forwarded) + }, +} diff --git a/infra/telemetry-proxy/wrangler.toml b/infra/telemetry-proxy/wrangler.toml new file mode 100644 index 00000000..c313e47e --- /dev/null +++ b/infra/telemetry-proxy/wrangler.toml @@ -0,0 +1,15 @@ +name = "stash-telemetry-proxy" +main = "worker.js" +compatibility_date = "2026-01-01" + +# Binds the Worker to our own hostname. Cloudflare provisions and renews the TLS +# certificate for the custom domain automatically. The zone (cipherstash.com) +# must be on this Cloudflare account. +routes = [ + { pattern = "telemetry.cipherstash.com", custom_domain = true } +] + +# The forward target. Change to "eu.i.posthog.com" and redeploy to move the whole +# CLI fleet to PostHog Cloud EU — no CLI release required. +[vars] +POSTHOG_HOST = "us.i.posthog.com" diff --git a/packages/cli/src/bin/main.ts b/packages/cli/src/bin/main.ts index c71794a8..52db4918 100644 --- a/packages/cli/src/bin/main.ts +++ b/packages/cli/src/bin/main.ts @@ -31,11 +31,18 @@ import { manifestCommand, planCommand, statusCommand, + telemetryCommand, testConnectionCommand, upgradeCommand, wizardCommand, } from '../commands/index.js' import { messages } from '../messages.js' +import { + initTelemetry, + maybeShowFirstRunNotice, + shutdownTelemetry, + trackCommand, +} from '../telemetry/index.js' function isModuleNotFound(err: unknown): boolean { return ( @@ -92,6 +99,7 @@ Commands: wizard AI-guided encryption setup (reads your codebase) doctor Diagnose install problems (native binaries, runtime) manifest Print the structured, versioned command surface (--json for docs/agents) + telemetry Manage anonymous usage analytics (status, enable, disable) eql install Scaffold stash.config.ts (if missing) and install EQL extensions eql upgrade Upgrade EQL extensions to the latest version @@ -431,6 +439,40 @@ export async function run() { return } + // Anonymous, opt-out usage analytics. The notice shows once (to stderr) and + // the run that shows it sends nothing; both are no-ops when telemetry is off. + initTelemetry(pkg.version) + maybeShowFirstRunNotice() + + const startedAt = Date.now() + let succeeded = true + let errorType: string | undefined + + try { + await dispatch(command, subcommand, commandArgs, flags, values) + } catch (err) { + succeeded = false + errorType = err instanceof Error ? err.constructor.name : 'Unknown' + throw err + } finally { + trackCommand({ + command, + subcommand, + success: succeeded, + durationMs: Date.now() - startedAt, + errorType, + }) + await shutdownTelemetry() + } +} + +async function dispatch( + command: string, + subcommand: string | undefined, + commandArgs: string[], + flags: Record, + values: Record, +) { switch (command) { case 'init': await initCommand(flags, values) @@ -473,6 +515,9 @@ export async function run() { // the native binary is missing. manifestCommand({ json: flags.json, version: pkg.version }) break + case 'telemetry': + await telemetryCommand(subcommand) + break case 'wizard': { // Forward everything after `stash wizard` verbatim. The wizard package // owns its own flag parsing; we don't try to interpret its surface diff --git a/packages/cli/src/cli/registry.ts b/packages/cli/src/cli/registry.ts index c388d5ed..66f527f5 100644 --- a/packages/cli/src/cli/registry.ts +++ b/packages/cli/src/cli/registry.ts @@ -231,6 +231,20 @@ export const registry: CommandGroup[] = [ }, ], }, + { + name: 'telemetry', + summary: 'Manage anonymous usage analytics', + long: [ + 'Manage the anonymous, opt-out usage analytics the CLI collects to', + 'improve the tool. `status` (the default) reports whether telemetry is', + 'on and which setting governs it; `enable` / `disable` write your saved', + 'preference. Telemetry is also disabled by the DO_NOT_TRACK or', + 'STASH_TELEMETRY_DISABLED environment variables and automatically in CI.', + 'No plaintext, schema, table/column names, or connection details are', + 'ever collected. See https://cipherstash.com/docs/reference/cli.', + ].join('\n'), + examples: ['telemetry status', 'telemetry disable', 'telemetry enable'], + }, ], }, { diff --git a/packages/cli/src/commands/index.ts b/packages/cli/src/commands/index.ts index 9069fe67..914dbfe3 100644 --- a/packages/cli/src/commands/index.ts +++ b/packages/cli/src/commands/index.ts @@ -9,4 +9,5 @@ export { initCommand } from './init/index.js' export { manifestCommand } from './manifest/index.js' export { planCommand } from './plan/index.js' export { statusCommand } from './status/index.js' +export { telemetryCommand } from './telemetry/index.js' export { wizardCommand } from './wizard/index.js' diff --git a/packages/cli/src/commands/telemetry/index.ts b/packages/cli/src/commands/telemetry/index.ts new file mode 100644 index 00000000..a794d0e3 --- /dev/null +++ b/packages/cli/src/commands/telemetry/index.ts @@ -0,0 +1,64 @@ +import * as p from '@clack/prompts' +import { messages } from '../../messages.js' +import { + setTelemetryDisabled, + type TelemetryDisabledReason, + telemetryStatus, +} from '../../telemetry/index.js' + +/** Human-readable explanation for why telemetry is currently off. */ +function reasonText(reason: TelemetryDisabledReason): string { + switch (reason) { + case 'do-not-track': + return 'disabled by the DO_NOT_TRACK environment variable' + case 'stash-disabled': + return 'disabled by the STASH_TELEMETRY_DISABLED environment variable' + case 'ci': + return 'disabled automatically in CI' + case 'config': + return 'disabled (run `stash telemetry enable` to turn it back on)' + case 'unconfigured': + return 'not active in this build' + } +} + +function printStatus(): void { + const status = telemetryStatus() + if (status.enabled) { + p.log.info('Telemetry is enabled. Anonymous usage analytics are collected.') + return + } + p.log.info(`Telemetry is ${reasonText(status.reason)}.`) + // An env override wins over the persisted flag, so flag `enable` won't help. + if (status.reason === 'do-not-track' || status.reason === 'stash-disabled') { + p.log.info( + 'An environment variable is overriding your saved preference; unset it to re-enable.', + ) + } +} + +/** + * `stash telemetry [status|enable|disable]` — manage anonymous CLI analytics. + * `enable`/`disable` write the persisted opt-out flag; `status` (the default) + * reports the current state and which gate governs it. + */ +export async function telemetryCommand(sub: string | undefined): Promise { + switch (sub) { + case undefined: + case 'status': + printStatus() + break + case 'enable': + setTelemetryDisabled(false) + p.log.success(messages.telemetry.enabled) + printStatus() + break + case 'disable': + setTelemetryDisabled(true) + p.log.success(messages.telemetry.disabled) + break + default: + p.log.error(`${messages.telemetry.unknownSubcommand}: ${sub}`) + process.exit(1) + } +} diff --git a/packages/cli/src/messages.ts b/packages/cli/src/messages.ts index 0e7e418a..08ce9371 100644 --- a/packages/cli/src/messages.ts +++ b/packages/cli/src/messages.ts @@ -109,4 +109,19 @@ export const messages = { ) => `\`${pkg}\` is not installed in this project.\n\nInstall the CipherStash packages, then re-run:\n ${installCommands}\n\nOr run \`${stash} init\` to set everything up.`, }, + telemetry: { + /** + * The one-time first-run notice. Printed to stderr so it never pollutes + * piped or `--json` stdout. Nothing is sent on the run that shows it. + */ + notice: [ + 'CipherStash collects anonymous CLI usage analytics to improve the tool.', + 'No plaintext, schema, table/column names, or connection details are ever collected.', + 'We honor the DO_NOT_TRACK standard. Opt out any time: stash telemetry disable', + 'Learn more: https://cipherstash.com/docs/reference/cli', + ].join('\n'), + enabled: 'Telemetry enabled.', + disabled: 'Telemetry disabled.', + unknownSubcommand: 'Unknown telemetry command', + }, } as const diff --git a/packages/cli/src/telemetry/__tests__/state.test.ts b/packages/cli/src/telemetry/__tests__/state.test.ts new file mode 100644 index 00000000..3a474759 --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/state.test.ts @@ -0,0 +1,59 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { afterEach, beforeEach, describe, expect, it } from 'vitest' +import { readState, writeState } from '../state.js' + +let home: string + +beforeEach(() => { + home = fs.mkdtempSync(path.join(os.tmpdir(), 'stash-telemetry-')) + process.env.HOME = home +}) + +afterEach(() => { + fs.rmSync(home, { recursive: true, force: true }) +}) + +const file = () => path.join(home, '.cipherstash', 'telemetry.json') + +describe('telemetry state', () => { + it('returns a fresh default (with an id) when no file exists', () => { + const state = readState() + expect(state.telemetryDisabled).toBe(false) + expect(state.noticeShownAt).toBeUndefined() + expect(state.anonymousId).toMatch(/[0-9a-f-]{36}/) + }) + + it('persists 0600 and round-trips', () => { + const written = writeState({ + anonymousId: 'anon-x', + telemetryDisabled: true, + noticeShownAt: '2026-07-14T00:00:00.000Z', + }) + expect(readState()).toEqual(written) + // Private to the user — never group/world readable. + expect(fs.statSync(file()).mode & 0o077).toBe(0) + }) + + it('recovers from a corrupt file with a fresh id, never throwing', () => { + fs.mkdirSync(path.dirname(file()), { recursive: true }) + fs.writeFileSync(file(), '{ this is not json') + const state = readState() + expect(state.anonymousId).toMatch(/[0-9a-f-]{36}/) + expect(state.telemetryDisabled).toBe(false) + }) + + it('coerces a partial/garbage object to valid defaults', () => { + fs.mkdirSync(path.dirname(file()), { recursive: true }) + fs.writeFileSync( + file(), + JSON.stringify({ telemetryDisabled: 'yes', noticeShownAt: 42 }), + ) + const state = readState() + // A non-boolean flag is not treated as opted out; a non-string ts is dropped. + expect(state.telemetryDisabled).toBe(false) + expect(state.noticeShownAt).toBeUndefined() + expect(state.anonymousId).toMatch(/[0-9a-f-]{36}/) + }) +}) diff --git a/packages/cli/src/telemetry/__tests__/telemetry.test.ts b/packages/cli/src/telemetry/__tests__/telemetry.test.ts new file mode 100644 index 00000000..92305f05 --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/telemetry.test.ts @@ -0,0 +1,160 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + ALLOWED_PROP_KEYS, + resolveStatus, + sanitize, + type TelemetryStatus, +} from '../index.js' +import type { TelemetryState } from '../state.js' + +const enabledState: TelemetryState = { + anonymousId: 'anon-1', + telemetryDisabled: false, +} + +/** The gates only run once a project key is present; otherwise it's dormant. */ +function withKey(): void { + vi.stubEnv('STASH_POSTHOG_KEY', 'phc_test_key') +} + +/** Clear every env var that resolveStatus / isCI consult. */ +function clearGateEnv(): void { + for (const name of [ + 'DO_NOT_TRACK', + 'STASH_TELEMETRY_DISABLED', + 'CI', + 'CONTINUOUS_INTEGRATION', + 'BUILD_NUMBER', + 'GITHUB_ACTIONS', + 'GITLAB_CI', + 'CIRCLECI', + 'TRAVIS', + 'BUILDKITE', + 'JENKINS_URL', + 'TEAMCITY_VERSION', + ]) { + vi.stubEnv(name, '') + } +} + +function reasonOf(status: TelemetryStatus): string { + return status.enabled ? 'enabled' : status.reason +} + +describe('resolveStatus gates', () => { + beforeEach(() => { + clearGateEnv() + }) + afterEach(() => { + vi.unstubAllEnvs() + }) + + it('is dormant (unconfigured) when no project key is set', () => { + // No STASH_POSTHOG_KEY and the embedded literal is still the placeholder. + expect(resolveStatus(enabledState)).toEqual({ + enabled: false, + reason: 'unconfigured', + }) + }) + + it('is enabled with a key and no opt-out signals', () => { + withKey() + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + }) + + it('DO_NOT_TRACK disables and outranks the persisted flag', () => { + withKey() + vi.stubEnv('DO_NOT_TRACK', '1') + // Even with telemetry "enabled" in config, the env override wins. + expect(reasonOf(resolveStatus(enabledState))).toBe('do-not-track') + }) + + it('STASH_TELEMETRY_DISABLED disables', () => { + withKey() + vi.stubEnv('STASH_TELEMETRY_DISABLED', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('stash-disabled') + }) + + it('DO_NOT_TRACK outranks STASH_TELEMETRY_DISABLED', () => { + withKey() + vi.stubEnv('DO_NOT_TRACK', '1') + vi.stubEnv('STASH_TELEMETRY_DISABLED', '1') + expect(reasonOf(resolveStatus(enabledState))).toBe('do-not-track') + }) + + it('auto-disables in CI', () => { + withKey() + vi.stubEnv('CI', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('ci') + }) + + it('honors a non-standard CI marker (GITHUB_ACTIONS)', () => { + withKey() + vi.stubEnv('GITHUB_ACTIONS', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('ci') + }) + + it('the persisted flag disables when no env override is present', () => { + withKey() + expect( + reasonOf(resolveStatus({ ...enabledState, telemetryDisabled: true })), + ).toBe('config') + }) + + it('treats DO_NOT_TRACK=0 as not opted out', () => { + withKey() + vi.stubEnv('DO_NOT_TRACK', '0') + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + }) +}) + +describe('sanitize (property allowlist)', () => { + it('drops any key not on the allowlist', () => { + const out = sanitize({ + command: 'eql', + subcommand: 'install', + // Everything below is exactly what must never leave the machine. + table: 'users', + column: 'email', + databaseUrl: 'postgres://secret@host/db', + plaintext: 'alice@example.com', + argv: ['--database-url', 'postgres://…'], + }) + expect(out).toEqual({ command: 'eql', subcommand: 'install' }) + }) + + it('keeps the full coarse event shape', () => { + const event = { + command: 'encrypt', + subcommand: 'backfill', + success: true, + durationMs: 1234, + errorType: undefined, + cliVersion: '0.17.1', + os: 'darwin', + arch: 'arm64', + nodeVersion: '22.0.0', + } + expect(sanitize(event)).toEqual(event) + for (const key of Object.keys(event)) { + expect(ALLOWED_PROP_KEYS.has(key)).toBe(true) + } + }) + + it('the allowlist contains no identifying or payload fields', () => { + for (const forbidden of [ + 'table', + 'column', + 'schema', + 'databaseUrl', + 'connectionString', + 'plaintext', + 'ciphertext', + 'argv', + 'args', + 'value', + ]) { + expect(ALLOWED_PROP_KEYS.has(forbidden)).toBe(false) + } + }) +}) diff --git a/packages/cli/src/telemetry/index.ts b/packages/cli/src/telemetry/index.ts new file mode 100644 index 00000000..1d31058c --- /dev/null +++ b/packages/cli/src/telemetry/index.ts @@ -0,0 +1,234 @@ +import { PostHog } from 'posthog-node' +import { messages } from '../messages.js' +import { readState, type TelemetryState, writeState } from './state.js' + +/** + * Anonymous, opt-out CLI usage analytics. + * + * Design (see the CLI analytics plan): + * - Opt-out by default, but **nothing is sent until the first-run notice has + * been shown once** — the run that shows it is always a freebie. + * - Four opt-out gates, any of which disables: `DO_NOT_TRACK`, + * `STASH_TELEMETRY_DISABLED`, CI auto-detection, and the persisted + * `stash telemetry disable` flag. + * - Events are anonymous (no PostHog "person" profiles) and carry only a fixed + * allowlist of coarse properties. Table/column/schema names, connection + * strings, plaintext, ciphertext, and raw argument values can never leave — + * see {@link sanitize}. + * - Sending never blocks or slows the CLI: the client is built lazily, flushing + * is bounded by a timeout, and every failure is swallowed. + */ + +/** The CLI talks to our Cloudflare proxy, not PostHog directly, so that a future + * US→EU migration is a proxy-target change with no CLI re-release. */ +const POSTHOG_HOST = 'https://telemetry.cipherstash.com' + +/** + * Public, write-only PostHog project key — safe to embed, exactly like a web + * SDK key. This literal is replaced with the real key at release (build-time + * define); `STASH_POSTHOG_KEY` overrides it for testing / self-hosting. Until a + * real key is present telemetry stays fully dormant: {@link resolveStatus} + * returns `unconfigured`, so no banner shows and nothing is ever sent. + */ +const EMBEDDED_KEY = '__STASH_POSTHOG_KEY__' + +function projectKey(): string { + return process.env.STASH_POSTHOG_KEY ?? EMBEDDED_KEY +} + +/** Flush is bounded so a slow or unreachable endpoint can't hang `stash`. */ +const FLUSH_TIMEOUT_MS = 1500 + +export type TelemetryDisabledReason = + | 'do-not-track' + | 'stash-disabled' + | 'ci' + | 'config' + | 'unconfigured' + +export type TelemetryStatus = + | { enabled: true } + | { enabled: false; reason: TelemetryDisabledReason } + +/** The only property keys allowed to leave the machine. Everything else is dropped. */ +export const ALLOWED_PROP_KEYS: ReadonlySet = new Set([ + 'command', + 'subcommand', + 'success', + 'durationMs', + 'errorType', + 'cliVersion', + 'os', + 'arch', + 'nodeVersion', +]) + +/** Interpret an env var as a boolean flag: set to `1`/`true`/`yes` (any case). */ +function envFlag(name: string): boolean { + const raw = process.env[name] + return raw != null && ['1', 'true', 'yes'].includes(raw.trim().toLowerCase()) +} + +/** An env var is present with a meaningful (non-empty) value. */ +function present(name: string): boolean { + const raw = process.env[name] + return raw != null && raw !== '' +} + +/** Best-effort CI detection — CI runs are noise and are auto-opted-out. */ +function isCI(): boolean { + return ( + envFlag('CI') || + present('CONTINUOUS_INTEGRATION') || + present('BUILD_NUMBER') || + [ + 'GITHUB_ACTIONS', + 'GITLAB_CI', + 'CIRCLECI', + 'TRAVIS', + 'BUILDKITE', + 'JENKINS_URL', + 'TEAMCITY_VERSION', + ].some(present) + ) +} + +/** + * Resolve whether telemetry is on, in precedence order. Env overrides win over + * the persisted flag, so a user can force telemetry off in a context where the + * config file says otherwise (and never the reverse). + */ +export function resolveStatus(state: TelemetryState): TelemetryStatus { + if (projectKey() === EMBEDDED_KEY) { + return { enabled: false, reason: 'unconfigured' } + } + if (envFlag('DO_NOT_TRACK')) return { enabled: false, reason: 'do-not-track' } + if (envFlag('STASH_TELEMETRY_DISABLED')) { + return { enabled: false, reason: 'stash-disabled' } + } + if (isCI()) return { enabled: false, reason: 'ci' } + if (state.telemetryDisabled) return { enabled: false, reason: 'config' } + return { enabled: true } +} + +/** Defence-in-depth: keep only allowlisted keys, whatever a caller passes. */ +export function sanitize( + properties: Record, +): Record { + const out: Record = {} + for (const [key, value] of Object.entries(properties)) { + if (ALLOWED_PROP_KEYS.has(key)) out[key] = value + } + return out +} + +// Module-level state, resolved once per process at import time. +let state = readState() +let status = resolveStatus(state) +/** True when this is the first-ever run (notice not yet shown). It stays true + * for the whole process even after we persist the notice, so the current run + * never sends — the freebie. */ +const firstRun = state.noticeShownAt === undefined +let cliVersion = '0.0.0' +let client: PostHog | null = null + +/** Provide the CLI version for event properties. Call once at startup. */ +export function initTelemetry(version: string): void { + cliVersion = version +} + +export function telemetryStatus(): TelemetryStatus { + return status +} + +function getClient(): PostHog { + if (client === null) { + client = new PostHog(projectKey(), { + host: POSTHOG_HOST, + flushAt: 1, + flushInterval: 0, + // Never resolve IP → geo; we don't want or store location. + disableGeoip: true, + }) + } + return client +} + +function baseProps(): Record { + return { + cliVersion, + os: process.platform, + arch: process.arch, + nodeVersion: process.versions.node, + } +} + +export interface CommandEvent { + command: string + subcommand?: string + success: boolean + durationMs: number + /** The error constructor name on failure — a class, never a message. */ + errorType?: string +} + +/** + * Record that a command ran. A no-op when telemetry is disabled or on the + * first run. Never throws. + */ +export function trackCommand(event: CommandEvent): void { + if (!status.enabled || firstRun) return + try { + getClient().capture({ + distinctId: state.anonymousId, + event: 'command_invoked', + properties: { + ...sanitize({ ...event }), + ...baseProps(), + // Keep events anonymous: no PostHog person profiles. + $process_person_profile: false, + }, + }) + } catch { + // Telemetry must never surface in the command path. + } +} + +/** + * Show the one-time first-run notice (to stderr, so it never pollutes piped or + * `--json` stdout), then mark it shown. No-op when telemetry is disabled or the + * notice was already shown. The run that shows it sends nothing (see {@link firstRun}). + */ +export function maybeShowFirstRunNotice(): void { + if (!status.enabled || !firstRun) return + if (process.stderr.isTTY) { + process.stderr.write(`${messages.telemetry.notice}\n`) + } + try { + state = writeState({ ...state, noticeShownAt: new Date().toISOString() }) + status = resolveStatus(state) + } catch { + // A write failure just means we may show the notice again next run. + } +} + +/** Persist the opt-out flag. Surfaces write errors (the command wants to know). */ +export function setTelemetryDisabled(disabled: boolean): void { + state = writeState({ ...state, telemetryDisabled: disabled }) + status = resolveStatus(state) +} + +/** Flush any buffered events, bounded by {@link FLUSH_TIMEOUT_MS}. Never throws. */ +export async function shutdownTelemetry(): Promise { + if (client === null) return + try { + await Promise.race([ + client.shutdown(), + new Promise((resolve) => setTimeout(resolve, FLUSH_TIMEOUT_MS)), + ]) + } catch { + // Swallow — a failed flush must not fail the process. + } finally { + client = null + } +} diff --git a/packages/cli/src/telemetry/state.ts b/packages/cli/src/telemetry/state.ts new file mode 100644 index 00000000..fab18b1b --- /dev/null +++ b/packages/cli/src/telemetry/state.ts @@ -0,0 +1,74 @@ +import { randomUUID } from 'node:crypto' +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' + +/** + * Machine-level telemetry state, persisted alongside the auth profile in + * `~/.cipherstash/` but in its own file. We only ever read/write `telemetry.json` + * here — never the auth secrets (`auth.json`, `secretkey.json`, `workspaces/`). + * + * The file holds three things: a random anonymous id (for de-duplicating events + * in aggregate, never derivable to a person), the persisted opt-out flag written + * by `stash telemetry disable`, and the timestamp of the first-run notice so the + * banner shows exactly once per install. + */ +export interface TelemetryState { + /** Random UUID; the PostHog `distinctId`. Not tied to any identity. */ + anonymousId: string + /** Set by `stash telemetry disable`. The lowest-precedence opt-out gate. */ + telemetryDisabled: boolean + /** ISO timestamp the first-run notice was shown; unset means "never shown". */ + noticeShownAt?: string +} + +/** Resolved at call time so it always tracks the current home directory. */ +function stateDir(): string { + return path.join(os.homedir(), '.cipherstash') +} +function stateFile(): string { + return path.join(stateDir(), 'telemetry.json') +} + +/** Coerce arbitrary parsed JSON into a valid state, filling gaps with defaults. */ +function normalize(value: unknown): TelemetryState { + const o = ( + typeof value === 'object' && value !== null ? value : {} + ) as Record + return { + anonymousId: + typeof o.anonymousId === 'string' && o.anonymousId.length > 0 + ? o.anonymousId + : randomUUID(), + telemetryDisabled: o.telemetryDisabled === true, + noticeShownAt: + typeof o.noticeShownAt === 'string' ? o.noticeShownAt : undefined, + } +} + +/** + * Read the state file. Never throws: a missing or corrupt file yields a fresh + * default (with a new anonymous id), so a bad file can never break a command. + * The fresh id is only ephemeral until something calls {@link writeState}. + */ +export function readState(): TelemetryState { + try { + return normalize(JSON.parse(fs.readFileSync(stateFile(), 'utf-8'))) + } catch { + return { anonymousId: randomUUID(), telemetryDisabled: false } + } +} + +/** + * Persist state (0600, private). Returns the normalized value actually written. + * Unlike {@link readState} this may throw — callers that must not fail a command + * (the emitter) wrap it; the `stash telemetry` command lets a write error surface. + */ +export function writeState(state: TelemetryState): TelemetryState { + const normalized = normalize(state) + fs.mkdirSync(stateDir(), { recursive: true }) + fs.writeFileSync(stateFile(), `${JSON.stringify(normalized, null, 2)}\n`, { + mode: 0o600, + }) + return normalized +} diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index fca5c955..2a94ccd3 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -16,7 +16,7 @@ Think Prisma Migrate or Drizzle Kit: a dev-time tool that prepares the database, Use this skill when: - The user wants to set up CipherStash or install EQL in a PostgreSQL database. -- Any `stash` command is being run: `init`, `plan`, `impl`, `status`, `auth`, `eql`, `db`, `encrypt`, `schema`, `manifest`, `doctor`, `wizard`, `env`. +- Any `stash` command is being run: `init`, `plan`, `impl`, `status`, `auth`, `eql`, `db`, `encrypt`, `schema`, `manifest`, `doctor`, `telemetry`, `wizard`, `env`. - A `stash.config.ts` file exists or needs to be created. - A `.cipherstash/` directory exists (`context.json`, `plan.md`, `migrations.json`, `setup-prompt.md`). - The user mentions "stash CLI", "EQL install", "encryption schema", or an encryption rollout/cutover. @@ -140,6 +140,29 @@ First hit wins: The resolved URL is returned in memory only. It is never written to disk or into `process.env`. +## Telemetry + +The CLI collects **anonymous, opt-out** usage analytics — coarse events only +(command name, CLI version, OS/arch, success/failure, duration). It **never** +collects plaintext, schema, table/column names, connection strings, or argument +values. A one-time notice is printed on first run, and nothing is sent on that +first run. + +Opt out in any of these ways (any one wins; env vars override the saved +preference): + +| Mechanism | Effect | +|---|---| +| `DO_NOT_TRACK=1` | Honors the cross-tool standard; disables telemetry | +| `STASH_TELEMETRY_DISABLED=1` | Disables telemetry | +| `CI=true` (or common CI markers) | Auto-disabled in CI | +| `npx stash telemetry disable` | Persists opt-out to `~/.cipherstash/telemetry.json` | + +`npx stash telemetry status` reports the current state and which setting governs +it; `npx stash telemetry enable` clears the saved opt-out (env overrides still +apply). State lives in `~/.cipherstash/telemetry.json` — a non-secret file +distinct from the auth credentials in that directory. + ## Configuration `stash.config.ts` in the project root: @@ -279,6 +302,7 @@ Flags below are the decision-relevant ones. Run `stash --help` for the | `status` | Rollout quest log (above) | | `manifest [--json]` | Print the structured, versioned command surface | | `doctor` | Diagnose install problems (native binaries, runtime). Runs before the CLI body loads, so it works when the native binary is broken. | +| `telemetry [status\|enable\|disable]` | Manage anonymous usage analytics (below) | | `wizard` | AI-guided encryption setup — thin wrapper over `@cipherstash/wizard` | ### Auth From ad776759dcfa11fc5ef08fc2fd96ed1cb2340af3 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 14 Jul 2026 14:20:10 +1000 Subject: [PATCH 02/10] fix(cli): address telemetry review + Copilot feedback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings: - shutdownTelemetry: clear the flush-timeout timer so a fast flush no longer leaves a dangling setTimeout that hangs the process for the full timeout; also set fetchRetryCount:0 so an unreachable endpoint fails fast instead of retrying with backoff (which kept internal timers alive and hung the CLI for seconds). Enabled commands now exit in ~140ms even when the endpoint is down. - First-run notice: persist noticeShownAt ONLY when the notice was actually displayed (stderr is a TTY), so a non-interactive first run no longer consumes the freebie and starts sending without the user ever seeing the disclosure. - Broaden DO_NOT_TRACK / STASH_TELEMETRY_DISABLED to opt out on any non-empty value except 0/false (matches the DO_NOT_TRACK convention). - Reuse the shared isCiEnv() from config/tty.ts instead of a second, divergent CI detector; widen isCiEnv() with provider markers (GITHUB_ACTIONS, GITLAB_CI, Azure TF_BUILD, Jenkins, TeamCity, …) so telemetry never fires in CI even when a provider doesn't set CI=true. Export CI_ENV_VARS and make the region / database-url tests hermetic against those ambient vars. - Drop the dead resolveStatus() re-resolve after writing the notice. Copilot: - Sanitize the combined event+base props so the allowlist is the single enforced boundary (a future baseProps field can't bypass it). - Make the first-run opt-out hint runner-aware (npx stash …) so it's actionable before stash is on PATH. - Restore process.env.HOME in the state test so it can't leak across test files. - Worker returns 500 when POSTHOG_HOST is unset instead of forwarding to an invalid host. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- infra/telemetry-proxy/worker.js | 9 +- .../cli/src/__tests__/database-url.test.ts | 27 ++++++ packages/cli/src/bin/main.ts | 2 +- .../commands/auth/__tests__/region.test.ts | 27 ++++++ packages/cli/src/config/tty.ts | 46 ++++++++-- packages/cli/src/messages.ts | 15 ++-- .../cli/src/telemetry/__tests__/state.test.ts | 7 ++ .../src/telemetry/__tests__/telemetry.test.ts | 37 +++++--- packages/cli/src/telemetry/index.ts | 87 ++++++++++--------- 9 files changed, 188 insertions(+), 69 deletions(-) diff --git a/infra/telemetry-proxy/worker.js b/infra/telemetry-proxy/worker.js index 25676240..d9e3add5 100644 --- a/infra/telemetry-proxy/worker.js +++ b/infra/telemetry-proxy/worker.js @@ -26,6 +26,14 @@ export default { async fetch(request, env) { const url = new URL(request.url) + const target = env.POSTHOG_HOST // "us.i.posthog.com" now; "eu.i.posthog.com" later + if (!target) { + // Misconfiguration — fail loudly rather than forward to an invalid host. + return new Response('Telemetry proxy misconfigured: POSTHOG_HOST unset', { + status: 500, + }) + } + if (request.method !== 'POST' && request.method !== 'GET') { return new Response('Method Not Allowed', { status: 405 }) } @@ -33,7 +41,6 @@ export default { return new Response('Not Found', { status: 404 }) } - const target = env.POSTHOG_HOST // "us.i.posthog.com" now; "eu.i.posthog.com" later url.hostname = target url.port = '' url.protocol = 'https:' diff --git a/packages/cli/src/__tests__/database-url.test.ts b/packages/cli/src/__tests__/database-url.test.ts index 885ce42d..3f03e784 100644 --- a/packages/cli/src/__tests__/database-url.test.ts +++ b/packages/cli/src/__tests__/database-url.test.ts @@ -2,6 +2,7 @@ import fs from 'node:fs' import os from 'node:os' import path from 'node:path' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { CI_ENV_VARS } from '../config/tty.js' import { messages } from '../messages.js' // Mock seams. Hoisted so the in-test reconfiguration touches the same fn @@ -39,6 +40,30 @@ let originalEnv: string | undefined let originalCi: string | undefined let originalIsTty: boolean | undefined let tmpDir: string +// isCiEnv() also consults provider vars (GITHUB_ACTIONS, GITLAB_CI, …); a real +// GITHUB_ACTIONS=true in this repo's own CI would otherwise flip "not CI" tests. +let savedProviderCi: Record = {} + +function neutralizeProviderCi(): void { + savedProviderCi = {} + for (const name of CI_ENV_VARS) { + if (name === 'CI') continue // handled by originalCi above/below + savedProviderCi[name] = process.env[name] + // biome-ignore lint/performance/noDelete: restore exact absence in afterEach. + delete process.env[name] + } +} + +function restoreProviderCi(): void { + for (const [name, value] of Object.entries(savedProviderCi)) { + if (value === undefined) { + // biome-ignore lint/performance/noDelete: restore exact absence. + delete process.env[name] + } else { + process.env[name] = value + } + } +} function noProject() { detect.detectSupabaseProject.mockReturnValue({ @@ -56,6 +81,7 @@ beforeEach(() => { delete process.env.DATABASE_URL // biome-ignore lint/performance/noDelete: ditto. delete process.env.CI + neutralizeProviderCi() tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'database-url-test-')) noProject() }) @@ -73,6 +99,7 @@ afterEach(() => { } else { process.env.CI = originalCi } + restoreProviderCi() Object.defineProperty(process.stdin, 'isTTY', { value: originalIsTty, configurable: true, diff --git a/packages/cli/src/bin/main.ts b/packages/cli/src/bin/main.ts index 52db4918..bb420e0a 100644 --- a/packages/cli/src/bin/main.ts +++ b/packages/cli/src/bin/main.ts @@ -442,7 +442,7 @@ export async function run() { // Anonymous, opt-out usage analytics. The notice shows once (to stderr) and // the run that shows it sends nothing; both are no-ops when telemetry is off. initTelemetry(pkg.version) - maybeShowFirstRunNotice() + maybeShowFirstRunNotice(STASH) const startedAt = Date.now() let succeeded = true diff --git a/packages/cli/src/commands/auth/__tests__/region.test.ts b/packages/cli/src/commands/auth/__tests__/region.test.ts index 838bfac6..819ec719 100644 --- a/packages/cli/src/commands/auth/__tests__/region.test.ts +++ b/packages/cli/src/commands/auth/__tests__/region.test.ts @@ -1,4 +1,5 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { CI_ENV_VARS } from '../../../config/tty.js' import { messages } from '../../../messages.js' // region.ts imports `* as p from '@clack/prompts'` (no native `@cipherstash/auth` @@ -29,6 +30,30 @@ const { let originalRegionEnv: string | undefined let originalCi: string | undefined let originalIsTty: boolean | undefined +// isCiEnv() also consults provider vars (GITHUB_ACTIONS, GITLAB_CI, …); a real +// GITHUB_ACTIONS=true in this repo's own CI would otherwise flip "not CI" tests. +let savedProviderCi: Record = {} + +function neutralizeProviderCi(): void { + savedProviderCi = {} + for (const name of CI_ENV_VARS) { + if (name === 'CI') continue // handled by originalCi above/below + savedProviderCi[name] = process.env[name] + // biome-ignore lint/performance/noDelete: restore exact absence in afterEach. + delete process.env[name] + } +} + +function restoreProviderCi(): void { + for (const [name, value] of Object.entries(savedProviderCi)) { + if (value === undefined) { + // biome-ignore lint/performance/noDelete: restore exact absence. + delete process.env[name] + } else { + process.env[name] = value + } + } +} function setTty(value: boolean | undefined) { Object.defineProperty(process.stdin, 'isTTY', { @@ -52,6 +77,7 @@ beforeEach(() => { delete process.env[REGION_ENV_VAR] // biome-ignore lint/performance/noDelete: ditto. delete process.env.CI + neutralizeProviderCi() }) afterEach(() => { @@ -67,6 +93,7 @@ afterEach(() => { } else { process.env.CI = originalCi } + restoreProviderCi() setTty(originalIsTty) vi.restoreAllMocks() }) diff --git a/packages/cli/src/config/tty.ts b/packages/cli/src/config/tty.ts index 09fd476a..1b97dd4e 100644 --- a/packages/cli/src/config/tty.ts +++ b/packages/cli/src/config/tty.ts @@ -4,14 +4,50 @@ */ /** - * True when `CI` is set to a common truthy spelling (`true`/`1`, - * case-insensitive) — not every CI provider sets `CI=true` exactly. Shared by - * the region resolver (`commands/auth/region.ts`) and the DATABASE_URL resolver - * (`config/database-url.ts`) so their non-interactive gating stays identical. + * Provider-specific markers for CI systems that don't reliably set `CI=true` + * (Jenkins, Azure Pipelines/`TF_BUILD`, TeamCity, …). GitHub Actions and GitLab + * both DO set `CI=true`, so their vars here are belt-and-suspenders. Detected by + * presence of any non-empty value. + */ +export const CI_PROVIDER_ENV_VARS = [ + 'GITHUB_ACTIONS', + 'GITLAB_CI', + 'CIRCLECI', + 'TRAVIS', + 'BUILDKITE', + 'JENKINS_URL', + 'TEAMCITY_VERSION', + 'TF_BUILD', // Azure Pipelines + 'APPVEYOR', + 'DRONE', + 'BITBUCKET_BUILD_NUMBER', + 'CODEBUILD_BUILD_ID', +] as const + +/** + * Every env var {@link isCiEnv} consults. Exported so tests can neutralize all + * CI signals hermetically (clearing only `CI` leaves ambient provider vars — a + * real `GITHUB_ACTIONS=true` in this repo's own CI — able to flip the result). + */ +export const CI_ENV_VARS = [ + 'CI', + 'CONTINUOUS_INTEGRATION', + ...CI_PROVIDER_ENV_VARS, +] as const + +/** + * True in a CI environment. `CI`/`CONTINUOUS_INTEGRATION` set to a truthy + * spelling (`true`/`1`) covers most providers — including GitHub Actions and + * GitLab — and {@link CI_PROVIDER_ENV_VARS} catches the ones that don't set + * `CI`. Shared by the region resolver (`commands/auth/region.ts`), the + * DATABASE_URL resolver (`config/database-url.ts`), and telemetry gating so they + * all decide "is this CI?" the same way. */ export function isCiEnv(): boolean { const ciVar = process.env.CI?.trim() - return ciVar !== undefined && /^(1|true)$/i.test(ciVar) + if (ciVar !== undefined && /^(1|true)$/i.test(ciVar)) return true + if (process.env.CONTINUOUS_INTEGRATION?.trim()) return true + return CI_PROVIDER_ENV_VARS.some((name) => Boolean(process.env[name]?.trim())) } /** diff --git a/packages/cli/src/messages.ts b/packages/cli/src/messages.ts index 08ce9371..9e671825 100644 --- a/packages/cli/src/messages.ts +++ b/packages/cli/src/messages.ts @@ -113,13 +113,16 @@ export const messages = { /** * The one-time first-run notice. Printed to stderr so it never pollutes * piped or `--json` stdout. Nothing is sent on the run that shows it. + * `stashRef` is the runner-aware invocation (e.g. `npx stash`) so the + * opt-out command is actionable even before `stash` is on PATH. */ - notice: [ - 'CipherStash collects anonymous CLI usage analytics to improve the tool.', - 'No plaintext, schema, table/column names, or connection details are ever collected.', - 'We honor the DO_NOT_TRACK standard. Opt out any time: stash telemetry disable', - 'Learn more: https://cipherstash.com/docs/reference/cli', - ].join('\n'), + notice: (stashRef: string) => + [ + 'CipherStash collects anonymous CLI usage analytics to improve the tool.', + 'No plaintext, schema, table/column names, or connection details are ever collected.', + `We honor the DO_NOT_TRACK standard. Opt out any time: set DO_NOT_TRACK=1, or run \`${stashRef} telemetry disable\`.`, + 'Learn more: https://cipherstash.com/docs/reference/cli', + ].join('\n'), enabled: 'Telemetry enabled.', disabled: 'Telemetry disabled.', unknownSubcommand: 'Unknown telemetry command', diff --git a/packages/cli/src/telemetry/__tests__/state.test.ts b/packages/cli/src/telemetry/__tests__/state.test.ts index 3a474759..91b52be4 100644 --- a/packages/cli/src/telemetry/__tests__/state.test.ts +++ b/packages/cli/src/telemetry/__tests__/state.test.ts @@ -5,13 +5,20 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest' import { readState, writeState } from '../state.js' let home: string +let originalHome: string | undefined beforeEach(() => { + originalHome = process.env.HOME home = fs.mkdtempSync(path.join(os.tmpdir(), 'stash-telemetry-')) process.env.HOME = home }) afterEach(() => { + if (originalHome === undefined) { + delete process.env.HOME + } else { + process.env.HOME = originalHome + } fs.rmSync(home, { recursive: true, force: true }) }) diff --git a/packages/cli/src/telemetry/__tests__/telemetry.test.ts b/packages/cli/src/telemetry/__tests__/telemetry.test.ts index 92305f05..96793b0a 100644 --- a/packages/cli/src/telemetry/__tests__/telemetry.test.ts +++ b/packages/cli/src/telemetry/__tests__/telemetry.test.ts @@ -1,4 +1,5 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { CI_ENV_VARS } from '../../config/tty.js' import { ALLOWED_PROP_KEYS, resolveStatus, @@ -17,21 +18,12 @@ function withKey(): void { vi.stubEnv('STASH_POSTHOG_KEY', 'phc_test_key') } -/** Clear every env var that resolveStatus / isCI consult. */ +/** Clear every env var that resolveStatus / isCiEnv consult. */ function clearGateEnv(): void { for (const name of [ 'DO_NOT_TRACK', 'STASH_TELEMETRY_DISABLED', - 'CI', - 'CONTINUOUS_INTEGRATION', - 'BUILD_NUMBER', - 'GITHUB_ACTIONS', - 'GITLAB_CI', - 'CIRCLECI', - 'TRAVIS', - 'BUILDKITE', - 'JENKINS_URL', - 'TEAMCITY_VERSION', + ...CI_ENV_VARS, ]) { vi.stubEnv(name, '') } @@ -101,10 +93,27 @@ describe('resolveStatus gates', () => { ).toBe('config') }) - it('treats DO_NOT_TRACK=0 as not opted out', () => { + it('treats DO_NOT_TRACK=0 / false / empty as not opted out', () => { withKey() - vi.stubEnv('DO_NOT_TRACK', '0') - expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + for (const value of ['0', 'false', '']) { + vi.stubEnv('DO_NOT_TRACK', value) + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + } + }) + + it('honors DO_NOT_TRACK set to any other non-empty value', () => { + withKey() + // The convention is that setting the variable at all signals opt-out. + for (const value of ['1', 'true', 'on', 'yes', 'please']) { + vi.stubEnv('DO_NOT_TRACK', value) + expect(reasonOf(resolveStatus(enabledState))).toBe('do-not-track') + } + }) + + it('auto-disables for a provider marker without CI (GitLab)', () => { + withKey() + vi.stubEnv('GITLAB_CI', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('ci') }) }) diff --git a/packages/cli/src/telemetry/index.ts b/packages/cli/src/telemetry/index.ts index 1d31058c..6848e33b 100644 --- a/packages/cli/src/telemetry/index.ts +++ b/packages/cli/src/telemetry/index.ts @@ -1,4 +1,5 @@ import { PostHog } from 'posthog-node' +import { isCiEnv } from '../config/tty.js' import { messages } from '../messages.js' import { readState, type TelemetryState, writeState } from './state.js' @@ -63,50 +64,34 @@ export const ALLOWED_PROP_KEYS: ReadonlySet = new Set([ 'nodeVersion', ]) -/** Interpret an env var as a boolean flag: set to `1`/`true`/`yes` (any case). */ -function envFlag(name: string): boolean { - const raw = process.env[name] - return raw != null && ['1', 'true', 'yes'].includes(raw.trim().toLowerCase()) -} - -/** An env var is present with a meaningful (non-empty) value. */ -function present(name: string): boolean { - const raw = process.env[name] - return raw != null && raw !== '' -} - -/** Best-effort CI detection — CI runs are noise and are auto-opted-out. */ -function isCI(): boolean { - return ( - envFlag('CI') || - present('CONTINUOUS_INTEGRATION') || - present('BUILD_NUMBER') || - [ - 'GITHUB_ACTIONS', - 'GITLAB_CI', - 'CIRCLECI', - 'TRAVIS', - 'BUILDKITE', - 'JENKINS_URL', - 'TEAMCITY_VERSION', - ].some(present) - ) +/** + * An opt-out env var is "set" when present and not an explicit off value. This + * is deliberately broad: `DO_NOT_TRACK=1`, `=true`, `=on`, or any other non-empty + * value opts out, matching the DO_NOT_TRACK convention that setting the variable + * at all signals intent. Only `''`, `'0'`, and `'false'` mean "not opted out". + */ +function envOptOut(name: string): boolean { + const raw = process.env[name]?.trim().toLowerCase() + return raw != null && raw !== '' && raw !== '0' && raw !== 'false' } /** * Resolve whether telemetry is on, in precedence order. Env overrides win over * the persisted flag, so a user can force telemetry off in a context where the - * config file says otherwise (and never the reverse). + * config file says otherwise (and never the reverse). CI detection is shared + * with the rest of the CLI via {@link isCiEnv}. */ export function resolveStatus(state: TelemetryState): TelemetryStatus { if (projectKey() === EMBEDDED_KEY) { return { enabled: false, reason: 'unconfigured' } } - if (envFlag('DO_NOT_TRACK')) return { enabled: false, reason: 'do-not-track' } - if (envFlag('STASH_TELEMETRY_DISABLED')) { + if (envOptOut('DO_NOT_TRACK')) { + return { enabled: false, reason: 'do-not-track' } + } + if (envOptOut('STASH_TELEMETRY_DISABLED')) { return { enabled: false, reason: 'stash-disabled' } } - if (isCI()) return { enabled: false, reason: 'ci' } + if (isCiEnv()) return { enabled: false, reason: 'ci' } if (state.telemetryDisabled) return { enabled: false, reason: 'config' } return { enabled: true } } @@ -147,6 +132,11 @@ function getClient(): PostHog { host: POSTHOG_HOST, flushAt: 1, flushInterval: 0, + // Fire-and-forget: a short-lived CLI must not retry a failed send. Retries + // (default 3, exponential backoff) would keep internal timers pending and + // hang the process for seconds when the endpoint is unreachable, defeating + // the bounded-flush guarantee. Drop the event instead. + fetchRetryCount: 0, // Never resolve IP → geo; we don't want or store location. disableGeoip: true, }) @@ -183,8 +173,10 @@ export function trackCommand(event: CommandEvent): void { distinctId: state.anonymousId, event: 'command_invoked', properties: { - ...sanitize({ ...event }), - ...baseProps(), + // Sanitize the FULL property set (event + base) so the allowlist is the + // single enforced boundary — a future prop added to baseProps() can't + // bypass it. Only the explicit PostHog control key is added afterward. + ...sanitize({ ...event, ...baseProps() }), // Keep events anonymous: no PostHog person profiles. $process_person_profile: false, }, @@ -196,17 +188,21 @@ export function trackCommand(event: CommandEvent): void { /** * Show the one-time first-run notice (to stderr, so it never pollutes piped or - * `--json` stdout), then mark it shown. No-op when telemetry is disabled or the + * `--json` stdout) and mark it shown. No-op when telemetry is disabled or the * notice was already shown. The run that shows it sends nothing (see {@link firstRun}). + * + * `noticeShownAt` is persisted ONLY when the notice was actually displayed — + * i.e. stderr is a TTY. A non-interactive first run therefore does not consume + * the freebie: telemetry stays dormant until a real run has shown the disclosure. + * `stashRef` is the runner-aware invocation (e.g. `npx stash`) so the opt-out + * hint is actionable before the CLI is on PATH. */ -export function maybeShowFirstRunNotice(): void { +export function maybeShowFirstRunNotice(stashRef: string): void { if (!status.enabled || !firstRun) return - if (process.stderr.isTTY) { - process.stderr.write(`${messages.telemetry.notice}\n`) - } + if (!process.stderr.isTTY) return + process.stderr.write(`${messages.telemetry.notice(stashRef)}\n`) try { - state = writeState({ ...state, noticeShownAt: new Date().toISOString() }) - status = resolveStatus(state) + writeState({ ...state, noticeShownAt: new Date().toISOString() }) } catch { // A write failure just means we may show the notice again next run. } @@ -221,14 +217,21 @@ export function setTelemetryDisabled(disabled: boolean): void { /** Flush any buffered events, bounded by {@link FLUSH_TIMEOUT_MS}. Never throws. */ export async function shutdownTelemetry(): Promise { if (client === null) return + // The timer MUST be cleared once shutdown() wins the race: an uncleared + // pending setTimeout keeps the Node event loop alive, so the process would + // hang for the full timeout after the flush already completed. + let timer: ReturnType | undefined try { await Promise.race([ client.shutdown(), - new Promise((resolve) => setTimeout(resolve, FLUSH_TIMEOUT_MS)), + new Promise((resolve) => { + timer = setTimeout(resolve, FLUSH_TIMEOUT_MS) + }), ]) } catch { // Swallow — a failed flush must not fail the process. } finally { + if (timer !== undefined) clearTimeout(timer) client = null } } From 9048fc64af977e6e50df07cf5c76110af3f0e5e1 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 14 Jul 2026 14:32:51 +1000 Subject: [PATCH 03/10] fix(cli): scrub ambient CI markers in the pty e2e harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Widening isCiEnv() with provider markers (GITHUB_ACTIONS, GITLAB_CI, …) made the pty e2e harness leak this repo's own CI env into the spawned CLI: the harness defaults CI=true and interactive tests override with `CI: ''`, but that no longer fully de-CIs the environment once GITHUB_ACTIONS=true is also present, so `auth login` skipped the region prompt and the test timed out. Strip every CI signal (CI_ENV_VARS) from the child env before applying the harness's CI default and per-test overrides, so CI detection is fully controlled by the harness — matching how the unit tests were made hermetic. Verified the full e2e suite passes with GITHUB_ACTIONS=true CI=true exported. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- packages/cli/tests/helpers/pty.ts | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/packages/cli/tests/helpers/pty.ts b/packages/cli/tests/helpers/pty.ts index 3d7b787e..b92cc1c9 100644 --- a/packages/cli/tests/helpers/pty.ts +++ b/packages/cli/tests/helpers/pty.ts @@ -4,6 +4,7 @@ import { dirname, join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' import { type IPty, spawn } from 'node-pty' import stripAnsi from 'strip-ansi' +import { CI_ENV_VARS } from '../../src/config/tty.js' const __dirname = dirname(fileURLToPath(import.meta.url)) @@ -98,11 +99,17 @@ export function render(args: string[], opts: RenderOptions = {}): Rendered { // debugging a failure. NO_COLOR: '1', FORCE_COLOR: '0', - // Match the convention the CLI itself uses (e.g. install.ts checks - // `process.env.CI !== 'true'`) so test runs hit the same code paths. - CI: 'true', - ...(opts.env ?? {}), } + // isCiEnv() consults provider markers (GITHUB_ACTIONS, GITLAB_CI, …) as well as + // CI. Those leak in from the ambient environment (this repo's own CI), so a + // test that de-CIs with `env: { CI: '' }` would otherwise STILL look like CI + // and skip interactive prompts. Strip every CI signal so the harness default + // below (and per-test overrides) fully control CI detection. + for (const name of CI_ENV_VARS) delete env[name] + // Match the convention the CLI itself uses (e.g. install.ts checks + // `process.env.CI !== 'true'`) so test runs hit the same code paths. + env.CI = 'true' + Object.assign(env, opts.env ?? {}) // Use the absolute path to the current node binary — node-pty's // `posix_spawnp` doesn't inherit PATH lookup reliably across all macOS / From 92bedb99578d1858f21c58195b80b71cddaa607c Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 15 Jul 2026 15:29:29 +1000 Subject: [PATCH 04/10] feat(cli): make telemetry PostHog host overridable (STASH_POSTHOG_HOST) Symmetric with STASH_POSTHOG_KEY: the host now defaults to the Cloudflare proxy (telemetry.cipherstash.com) but can be overridden for testing against a real PostHog ingestion endpoint before the proxy is deployed, or for self-hosting. Verified the CLI POSTs the event batch to the configured host. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- packages/cli/src/telemetry/index.ts | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/packages/cli/src/telemetry/index.ts b/packages/cli/src/telemetry/index.ts index 6848e33b..01478afe 100644 --- a/packages/cli/src/telemetry/index.ts +++ b/packages/cli/src/telemetry/index.ts @@ -20,9 +20,16 @@ import { readState, type TelemetryState, writeState } from './state.js' * is bounded by a timeout, and every failure is swallowed. */ -/** The CLI talks to our Cloudflare proxy, not PostHog directly, so that a future - * US→EU migration is a proxy-target change with no CLI re-release. */ -const POSTHOG_HOST = 'https://telemetry.cipherstash.com' +/** Default endpoint — our Cloudflare proxy, not PostHog directly, so a future + * US→EU migration is a proxy-target change with no CLI re-release. + * `STASH_POSTHOG_HOST` overrides it (testing against a real PostHog ingestion + * endpoint before the proxy is deployed, or self-hosting), symmetric with + * `STASH_POSTHOG_KEY`. */ +const DEFAULT_POSTHOG_HOST = 'https://telemetry.cipherstash.com' + +function posthogHost(): string { + return process.env.STASH_POSTHOG_HOST ?? DEFAULT_POSTHOG_HOST +} /** * Public, write-only PostHog project key — safe to embed, exactly like a web @@ -129,7 +136,7 @@ export function telemetryStatus(): TelemetryStatus { function getClient(): PostHog { if (client === null) { client = new PostHog(projectKey(), { - host: POSTHOG_HOST, + host: posthogHost(), flushAt: 1, flushInterval: 0, // Fire-and-forget: a short-lived CLI must not retry a failed send. Retries From ed08f4cbe67b4a614ccd24faa4c1756aeba124d2 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 15 Jul 2026 16:08:42 +1000 Subject: [PATCH 05/10] feat(cli): classify caller (agent harness vs interactive) in telemetry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a coarse, fixed-enum `caller` property to command telemetry so we can gauge how much stash usage is agent-driven — a lot of it is our own skills being run by a customer's coding agent. resolveCaller() recognises Claude Code (CLAUDECODE / CLAUDE_CODE_ENTRYPOINT), Cursor (CURSOR_TRACE_ID), and Codex (CODEX_SANDBOX) by the env var each sets, falls back to `editor` for a VS Code-family terminal (TERM_PROGRAM=vscode), then to `interactive`/`non-interactive` by stdin TTY. Only the fixed label is emitted, never the raw env value — some carry session/trace ids — so the anonymity contract holds. `caller` is added to the emitter allowlist. Updates the stash-cli skill disclosure and the changeset field list to match. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- .changeset/cli-anonymous-telemetry.md | 12 +-- packages/cli/src/config/__tests__/tty.test.ts | 84 +++++++++++++++++++ packages/cli/src/config/tty.ts | 63 ++++++++++++++ .../src/telemetry/__tests__/telemetry.test.ts | 1 + packages/cli/src/telemetry/index.ts | 6 +- skills/stash-cli/SKILL.md | 9 +- 6 files changed, 165 insertions(+), 10 deletions(-) create mode 100644 packages/cli/src/config/__tests__/tty.test.ts diff --git a/.changeset/cli-anonymous-telemetry.md b/.changeset/cli-anonymous-telemetry.md index b3898aa9..1cd88530 100644 --- a/.changeset/cli-anonymous-telemetry.md +++ b/.changeset/cli-anonymous-telemetry.md @@ -5,11 +5,13 @@ Add anonymous, opt-out usage analytics to the `stash` CLI, plus a `stash telemetry [status|enable|disable]` command to manage it. -Only coarse events are collected — command name, CLI version, OS/arch, -success/failure, and duration. Plaintext, schema, table/column names, connection -strings, and argument values are never collected (enforced by a property -allowlist at the emitter boundary). A one-time notice is shown on first run, and -nothing is sent on that run. +Only coarse events are collected — command name, CLI version, OS/arch, Node +version, success/failure, duration, and a coarse caller class (e.g. +`claude-code`, `cursor`, `interactive`) derived from environment markers so we +can gauge agent- vs human-driven usage. Plaintext, schema, table/column names, +connection strings, argument values, and any session/trace identifier are never +collected (enforced by a property allowlist at the emitter boundary). A one-time +notice is shown on first run, and nothing is sent on that run. Telemetry is off by default in CI and can be disabled with `DO_NOT_TRACK=1` (the cross-tool standard), `STASH_TELEMETRY_DISABLED=1`, or diff --git a/packages/cli/src/config/__tests__/tty.test.ts b/packages/cli/src/config/__tests__/tty.test.ts new file mode 100644 index 00000000..8efd07a5 --- /dev/null +++ b/packages/cli/src/config/__tests__/tty.test.ts @@ -0,0 +1,84 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { CALLER_ENV_VARS, resolveCaller } from '../tty.js' + +describe('resolveCaller', () => { + // This repo's own agent harness sets some caller markers ambiently, so clear + // every consulted var before each case; otherwise the fallback tests would + // pick up e.g. CLAUDECODE from the process running the suite. + beforeEach(() => { + for (const name of CALLER_ENV_VARS) vi.stubEnv(name, '') + }) + afterEach(() => { + vi.unstubAllEnvs() + }) + + /** Force process.stdin.isTTY for the fallback cases; restored after each test. */ + function withStdinTty(isTTY: boolean, run: () => void): void { + const original = Object.getOwnPropertyDescriptor(process.stdin, 'isTTY') + Object.defineProperty(process.stdin, 'isTTY', { + value: isTTY, + configurable: true, + }) + try { + run() + } finally { + if (original) Object.defineProperty(process.stdin, 'isTTY', original) + else delete (process.stdin as { isTTY?: boolean }).isTTY + } + } + + it('detects Claude Code via CLAUDECODE', () => { + vi.stubEnv('CLAUDECODE', '1') + expect(resolveCaller()).toBe('claude-code') + }) + + it('detects Claude Code via the CLAUDE_CODE_ENTRYPOINT backup marker', () => { + vi.stubEnv('CLAUDE_CODE_ENTRYPOINT', 'cli') + expect(resolveCaller()).toBe('claude-code') + }) + + it('detects Cursor via CURSOR_TRACE_ID', () => { + vi.stubEnv('CURSOR_TRACE_ID', 'abc-123') + expect(resolveCaller()).toBe('cursor') + }) + + it('detects Codex via CODEX_SANDBOX', () => { + vi.stubEnv('CODEX_SANDBOX', 'seatbelt') + expect(resolveCaller()).toBe('codex') + }) + + it('confirmed agents outrank the editor marker (Cursor also sets TERM_PROGRAM=vscode)', () => { + vi.stubEnv('CURSOR_TRACE_ID', 'abc-123') + vi.stubEnv('TERM_PROGRAM', 'vscode') + expect(resolveCaller()).toBe('cursor') + }) + + it('the first agent marker wins when several are present', () => { + vi.stubEnv('CLAUDECODE', '1') + vi.stubEnv('CURSOR_TRACE_ID', 'abc-123') + expect(resolveCaller()).toBe('claude-code') + }) + + it('classifies a VS Code-family editor terminal as editor, not an agent', () => { + vi.stubEnv('TERM_PROGRAM', 'vscode') + expect(resolveCaller()).toBe('editor') + }) + + it('is case-insensitive on TERM_PROGRAM', () => { + vi.stubEnv('TERM_PROGRAM', 'vsCode') + expect(resolveCaller()).toBe('editor') + }) + + it('ignores whitespace-only marker values', () => { + vi.stubEnv('CLAUDECODE', ' ') + withStdinTty(true, () => expect(resolveCaller()).toBe('interactive')) + }) + + it('falls back to interactive when stdin is a TTY and no marker is set', () => { + withStdinTty(true, () => expect(resolveCaller()).toBe('interactive')) + }) + + it('falls back to non-interactive when stdin is not a TTY', () => { + withStdinTty(false, () => expect(resolveCaller()).toBe('non-interactive')) + }) +}) diff --git a/packages/cli/src/config/tty.ts b/packages/cli/src/config/tty.ts index 1b97dd4e..0ea35fbd 100644 --- a/packages/cli/src/config/tty.ts +++ b/packages/cli/src/config/tty.ts @@ -50,6 +50,69 @@ export function isCiEnv(): boolean { return CI_PROVIDER_ENV_VARS.some((name) => Boolean(process.env[name]?.trim())) } +/** + * Coarse classification of who invoked the CLI, for anonymous telemetry. A + * heuristic, not a guarantee: it recognises the major coding-agent harnesses by + * an env var each one sets, and otherwise falls back to whether stdin is a TTY. + * Only the fixed label ever leaves the machine (see the telemetry allowlist) — + * never the underlying env value, some of which carry session/trace IDs. Read + * the numbers as a lower bound on agent usage: a harness without a known marker + * lands in `interactive`/`non-interactive`. + */ +export type CallerKind = + | 'claude-code' + | 'cursor' + | 'codex' + | 'editor' + | 'interactive' + | 'non-interactive' + +/** + * Known coding-agent harnesses, each recognised by an env var it sets in the + * shell it drives. Ordered most-specific first; the first match wins. Extend + * this as new harnesses appear — every `kind` must stay a fixed, non-identifying + * string, because that label (not the env value) is what telemetry emits. + */ +const AGENT_MARKERS: ReadonlyArray<{ + readonly kind: CallerKind + readonly envVars: readonly string[] +}> = [ + // Claude Code sets CLAUDECODE=1; CLAUDE_CODE_ENTRYPOINT is a backup signal. + { kind: 'claude-code', envVars: ['CLAUDECODE', 'CLAUDE_CODE_ENTRYPOINT'] }, + // Cursor's agent/terminal exports a per-session trace id. + { kind: 'cursor', envVars: ['CURSOR_TRACE_ID'] }, + // OpenAI Codex CLI runs commands in a sandbox that exports CODEX_SANDBOX. + { kind: 'codex', envVars: ['CODEX_SANDBOX'] }, +] + +/** + * Every env var {@link resolveCaller} consults. Exported so tests can neutralize + * all caller signals hermetically — this repo's own agent harness sets some of + * these ambiently, which would otherwise flip the result. + */ +export const CALLER_ENV_VARS = [ + ...AGENT_MARKERS.flatMap((m) => m.envVars), + 'TERM_PROGRAM', +] as const + +/** + * Classify the caller. Confirmed agent harnesses win first; then a VS Code-family + * editor terminal (`TERM_PROGRAM=vscode`, which Cursor also sets, so it is checked + * after Cursor and kept distinct because it may be a human in an editor); then the + * plain TTY heuristic. + */ +export function resolveCaller(): CallerKind { + for (const marker of AGENT_MARKERS) { + if (marker.envVars.some((name) => Boolean(process.env[name]?.trim()))) { + return marker.kind + } + } + if (process.env.TERM_PROGRAM?.trim().toLowerCase() === 'vscode') { + return 'editor' + } + return process.stdin.isTTY ? 'interactive' : 'non-interactive' +} + /** * True when it's safe to show an interactive clack prompt: stdin is a TTY and * we're not in CI. Every prompt gate (the DATABASE_URL resolver, the config diff --git a/packages/cli/src/telemetry/__tests__/telemetry.test.ts b/packages/cli/src/telemetry/__tests__/telemetry.test.ts index 96793b0a..5ff30f56 100644 --- a/packages/cli/src/telemetry/__tests__/telemetry.test.ts +++ b/packages/cli/src/telemetry/__tests__/telemetry.test.ts @@ -143,6 +143,7 @@ describe('sanitize (property allowlist)', () => { os: 'darwin', arch: 'arm64', nodeVersion: '22.0.0', + caller: 'claude-code', } expect(sanitize(event)).toEqual(event) for (const key of Object.keys(event)) { diff --git a/packages/cli/src/telemetry/index.ts b/packages/cli/src/telemetry/index.ts index 01478afe..353ecae5 100644 --- a/packages/cli/src/telemetry/index.ts +++ b/packages/cli/src/telemetry/index.ts @@ -1,5 +1,5 @@ import { PostHog } from 'posthog-node' -import { isCiEnv } from '../config/tty.js' +import { isCiEnv, resolveCaller } from '../config/tty.js' import { messages } from '../messages.js' import { readState, type TelemetryState, writeState } from './state.js' @@ -69,6 +69,7 @@ export const ALLOWED_PROP_KEYS: ReadonlySet = new Set([ 'os', 'arch', 'nodeVersion', + 'caller', ]) /** @@ -157,6 +158,9 @@ function baseProps(): Record { os: process.platform, arch: process.arch, nodeVersion: process.versions.node, + // Coarse, fixed-enum classification of the caller (agent harness vs + // interactive shell). Never the raw env value — see resolveCaller. + caller: resolveCaller(), } } diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 2a94ccd3..ffd82575 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -143,10 +143,11 @@ The resolved URL is returned in memory only. It is never written to disk or into ## Telemetry The CLI collects **anonymous, opt-out** usage analytics — coarse events only -(command name, CLI version, OS/arch, success/failure, duration). It **never** -collects plaintext, schema, table/column names, connection strings, or argument -values. A one-time notice is printed on first run, and nothing is sent on that -first run. +(command name, CLI version, OS/arch, Node version, success/failure, duration, +and a coarse caller class such as `claude-code`/`cursor`/`interactive` derived +from environment markers). It **never** collects plaintext, schema, table/column +names, connection strings, argument values, or any session/trace identifier. A +one-time notice is printed on first run, and nothing is sent on that first run. Opt out in any of these ways (any one wins; env vars override the saved preference): From 931b2b646d47500cd27d51c0d3db58bc9d65f46a Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 15 Jul 2026 16:18:18 +1000 Subject: [PATCH 06/10] chore(cli): move telemetry proxy to the platform (cipherstash-suite) repo The Cloudflare Worker that terminates telemetry.cipherstash.com is deployed platform infrastructure, not part of the published stash package, so it belongs with the rest of our infra in cipherstash-suite (infra/telemetry-proxy) rather than shipping in this repo. No CLI behaviour changes: the CLI still targets https://telemetry.cipherstash.com and stays dormant until a key is embedded. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- infra/telemetry-proxy/README.md | 49 -------------------------- infra/telemetry-proxy/worker.js | 53 ----------------------------- infra/telemetry-proxy/wrangler.toml | 15 -------- 3 files changed, 117 deletions(-) delete mode 100644 infra/telemetry-proxy/README.md delete mode 100644 infra/telemetry-proxy/worker.js delete mode 100644 infra/telemetry-proxy/wrangler.toml diff --git a/infra/telemetry-proxy/README.md b/infra/telemetry-proxy/README.md deleted file mode 100644 index b3dd7e1a..00000000 --- a/infra/telemetry-proxy/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# Stash telemetry proxy - -A Cloudflare Worker that receives the `stash` CLI's anonymous analytics at -`telemetry.cipherstash.com` and forwards them to PostHog. See `worker.js` for why -the CLI proxies instead of calling PostHog directly (first-party requests, and -so the US→EU migration needs no CLI re-release). - -## Deploy - -Requires [`wrangler`](https://developers.cloudflare.com/workers/wrangler/) and -access to the Cloudflare account that holds the `cipherstash.com` zone. - -```bash -cd infra/telemetry-proxy -wrangler deploy -``` - -The `custom_domain` route provisions `telemetry.cipherstash.com` and its TLS -certificate automatically on first deploy. - -## Verify - -```bash -# 404 for an unknown path (the Worker is not an open proxy) -curl -sS -o /dev/null -w '%{http_code}\n' https://telemetry.cipherstash.com/ - -# A capture POST is accepted and forwarded to PostHog -curl -sS -X POST https://telemetry.cipherstash.com/batch/ \ - -H 'content-type: application/json' \ - -d '{"api_key":"","batch":[]}' -``` - -## Moving to PostHog Cloud EU (future) - -The forward target is the `POSTHOG_HOST` var, not anything baked into the CLI. -To migrate the entire installed CLI fleet: - -1. Edit `POSTHOG_HOST` in `wrangler.toml` to `eu.i.posthog.com`. -2. `wrangler deploy`. - -No CLI change, no release. Every already-installed `stash` follows the proxy. - -## Wiring the CLI - -The CLI targets `https://telemetry.cipherstash.com` and carries the public, -write-only PostHog project key (embedded at release in -`packages/cli/src/telemetry/index.ts`; overridable with `STASH_POSTHOG_KEY` for -testing). Until a real key is embedded, CLI telemetry is dormant — no events are -sent regardless of this proxy. diff --git a/infra/telemetry-proxy/worker.js b/infra/telemetry-proxy/worker.js deleted file mode 100644 index d9e3add5..00000000 --- a/infra/telemetry-proxy/worker.js +++ /dev/null @@ -1,53 +0,0 @@ -/** - * CipherStash telemetry reverse proxy (Cloudflare Worker). - * - * The `stash` CLI sends anonymous analytics to `telemetry.cipherstash.com`, and - * this Worker forwards them to PostHog. Two reasons the CLI never talks to - * PostHog directly: - * - * 1. First-party + transparent — requests stay on our domain and sail through - * corporate firewall allowlists. - * 2. Indirection — a released CLI hard-codes whatever host it shipped with, - * forever. Because the target lives in the `POSTHOG_HOST` var, a future - * US→EU migration is a var change + redeploy here, with NO CLI re-release: - * every already-installed `stash` follows automatically. - * - * A bare DNS CNAME to PostHog can't do this — PostHog serves its own TLS cert - * and routes by Host header — so the domain must be terminated and rewritten, - * which is exactly what this Worker does. - */ - -// PostHog ingestion path prefixes. Restricting to these keeps the Worker from -// being a general-purpose open proxy while still covering posthog-node's -// batch/capture endpoints. -const ALLOWED_PREFIXES = ['/batch', '/capture', '/e', '/i/', '/flags', '/array'] - -export default { - async fetch(request, env) { - const url = new URL(request.url) - - const target = env.POSTHOG_HOST // "us.i.posthog.com" now; "eu.i.posthog.com" later - if (!target) { - // Misconfiguration — fail loudly rather than forward to an invalid host. - return new Response('Telemetry proxy misconfigured: POSTHOG_HOST unset', { - status: 500, - }) - } - - if (request.method !== 'POST' && request.method !== 'GET') { - return new Response('Method Not Allowed', { status: 405 }) - } - if (!ALLOWED_PREFIXES.some((prefix) => url.pathname.startsWith(prefix))) { - return new Response('Not Found', { status: 404 }) - } - - url.hostname = target - url.port = '' - url.protocol = 'https:' - - const forwarded = new Request(url, request) - forwarded.headers.set('Host', target) - - return fetch(forwarded) - }, -} diff --git a/infra/telemetry-proxy/wrangler.toml b/infra/telemetry-proxy/wrangler.toml deleted file mode 100644 index c313e47e..00000000 --- a/infra/telemetry-proxy/wrangler.toml +++ /dev/null @@ -1,15 +0,0 @@ -name = "stash-telemetry-proxy" -main = "worker.js" -compatibility_date = "2026-01-01" - -# Binds the Worker to our own hostname. Cloudflare provisions and renews the TLS -# certificate for the custom domain automatically. The zone (cipherstash.com) -# must be on this Cloudflare account. -routes = [ - { pattern = "telemetry.cipherstash.com", custom_domain = true } -] - -# The forward target. Change to "eu.i.posthog.com" and redeploy to move the whole -# CLI fleet to PostHog Cloud EU — no CLI release required. -[vars] -POSTHOG_HOST = "us.i.posthog.com" From 1b6b3e4618780cabebee64faeed339e80cbd292c Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 15 Jul 2026 16:53:37 +1000 Subject: [PATCH 07/10] feat(cli): embed the PostHog key at release via build-time define MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wire up the injection the placeholder always promised. tsup's esbuild `define` replaces the `__STASH_POSTHOG_KEY__` identifier with the key from the STASH_POSTHOG_KEY env var; release.yml feeds it from a public repo *variable* (vars.STASH_POSTHOG_KEY), so ONLY the official release build embeds a key. Dev, fork, and CI builds bake in an empty string and ship telemetry-dormant. Applied to both bundles; turbo.json lists the var on `build` so the cache key tracks it. Also fixes a latent bug the injection would have exposed: resolveStatus compared the resolved key against EMBEDDED_KEY, which once injected holds the real key — so an embedded build with no env override would have read as `unconfigured`. It now compares against a dedicated PLACEHOLDER_KEY sentinel (the un-rewritten string literal), so an injected build is correctly enabled. New tests cover the placeholder-passthrough (dormant) and injected-key (enabled) paths. The go-live switch is setting the repo variable; until then every release stays dormant. Verified both ways: dormant build reports "not active in this build", injected build reports "Telemetry is enabled" with no runtime env override. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- .github/workflows/release.yml | 7 ++++ .../src/telemetry/__tests__/telemetry.test.ts | 26 +++++++++++++ packages/cli/src/telemetry/index.ts | 37 +++++++++++++++---- packages/cli/tsup.config.ts | 16 ++++++++ turbo.json | 3 +- 5 files changed, 81 insertions(+), 8 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3f0e1987..947c0dc6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -69,3 +69,10 @@ jobs: # changesets/action writes a token .npmrc that shadows OIDC and # every publish fails with E404 (see npm/cli#8976). GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Embeds the CLI's PostHog project key at build time (see + # packages/cli/tsup.config.ts). A repo *variable*, not a secret: the + # key is public and write-only (like a web SDK key). Unset until GA, so + # every release before it is flipped ships telemetry-dormant. This is + # the single go-live switch — set it with: + # gh variable set STASH_POSTHOG_KEY --repo cipherstash/stack --body '' + STASH_POSTHOG_KEY: ${{ vars.STASH_POSTHOG_KEY }} diff --git a/packages/cli/src/telemetry/__tests__/telemetry.test.ts b/packages/cli/src/telemetry/__tests__/telemetry.test.ts index 5ff30f56..01c3d016 100644 --- a/packages/cli/src/telemetry/__tests__/telemetry.test.ts +++ b/packages/cli/src/telemetry/__tests__/telemetry.test.ts @@ -2,6 +2,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { CI_ENV_VARS } from '../../config/tty.js' import { ALLOWED_PROP_KEYS, + PLACEHOLDER_KEY, resolveStatus, sanitize, type TelemetryStatus, @@ -49,6 +50,31 @@ describe('resolveStatus gates', () => { }) }) + it('an un-injected build (placeholder passthrough) stays dormant', () => { + // Simulates a dev/fork/CI build: tsup baked in no key, so the resolved key + // is the placeholder sentinel. Must read as unconfigured whatever the gates. + vi.stubEnv('STASH_POSTHOG_KEY', PLACEHOLDER_KEY) + expect(resolveStatus(enabledState)).toEqual({ + enabled: false, + reason: 'unconfigured', + }) + }) + + it('a release-injected key is NOT mistaken for the placeholder → enabled', () => { + // The regression the sentinel guards against: an injected key with no env + // override must not read as unconfigured (the old `=== EMBEDDED_KEY` check + // would have). Any real key differs from the placeholder. + vi.stubEnv('STASH_POSTHOG_KEY', 'phc_realish_injected_key') + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + }) + + it('the placeholder sentinel matches the build-define identifier name', () => { + // tsup replaces the `__STASH_POSTHOG_KEY__` identifier at release; the + // sentinel must stay the identically-named string literal, or an un-injected + // build would not be detected as dormant. + expect(PLACEHOLDER_KEY).toBe('__STASH_POSTHOG_KEY__') + }) + it('is enabled with a key and no opt-out signals', () => { withKey() expect(resolveStatus(enabledState)).toEqual({ enabled: true }) diff --git a/packages/cli/src/telemetry/index.ts b/packages/cli/src/telemetry/index.ts index 353ecae5..94efbd8a 100644 --- a/packages/cli/src/telemetry/index.ts +++ b/packages/cli/src/telemetry/index.ts @@ -32,14 +32,33 @@ function posthogHost(): string { } /** - * Public, write-only PostHog project key — safe to embed, exactly like a web - * SDK key. This literal is replaced with the real key at release (build-time - * define); `STASH_POSTHOG_KEY` overrides it for testing / self-hosting. Until a - * real key is present telemetry stays fully dormant: {@link resolveStatus} - * returns `unconfigured`, so no banner shows and nothing is ever sent. + * The un-injected sentinel. A build that has NOT embedded a key resolves to + * exactly this string, and {@link resolveStatus} treats it as "no key" → + * dormant. It is a plain string literal (not the `__STASH_POSTHOG_KEY__` + * identifier), so the build-time `define` below never rewrites it. */ -const EMBEDDED_KEY = '__STASH_POSTHOG_KEY__' +export const PLACEHOLDER_KEY = '__STASH_POSTHOG_KEY__' +/** + * Build-time define. The release build (`.github/workflows/release.yml`) + * replaces the `__STASH_POSTHOG_KEY__` identifier with the real, public, + * write-only PostHog project key from the `STASH_POSTHOG_KEY` repo variable, via + * tsup's esbuild `define` (see `tsup.config.ts`). Safe to embed, exactly like a + * web SDK key. EVERY other build — local dev, a contributor's checkout, a fork, + * CI unit tests — leaves the identifier undefined, so {@link EMBEDDED_KEY} falls + * back to {@link PLACEHOLDER_KEY} and telemetry stays fully dormant. The `typeof` + * guard makes the reference safe even when the identifier was never defined + * (`typeof undefinedIdent` yields `"undefined"` rather than throwing). + */ +declare const __STASH_POSTHOG_KEY__: string | undefined + +const EMBEDDED_KEY = + typeof __STASH_POSTHOG_KEY__ === 'string' && __STASH_POSTHOG_KEY__.length > 0 + ? __STASH_POSTHOG_KEY__ + : PLACEHOLDER_KEY + +/** `STASH_POSTHOG_KEY` in the environment overrides the embedded key entirely + * (testing / self-hosting); otherwise the build-time value is used. */ function projectKey(): string { return process.env.STASH_POSTHOG_KEY ?? EMBEDDED_KEY } @@ -90,7 +109,11 @@ function envOptOut(name: string): boolean { * with the rest of the CLI via {@link isCiEnv}. */ export function resolveStatus(state: TelemetryState): TelemetryStatus { - if (projectKey() === EMBEDDED_KEY) { + // Compare against the sentinel, NOT EMBEDDED_KEY: once a real key is injected + // at release, EMBEDDED_KEY holds it, and `=== EMBEDDED_KEY` would then read a + // key-present build (no env override) as unconfigured. The placeholder passing + // through — no build-time key and no env override — is the "dormant" signal. + if (projectKey() === PLACEHOLDER_KEY) { return { enabled: false, reason: 'unconfigured' } } if (envOptOut('DO_NOT_TRACK')) { diff --git a/packages/cli/tsup.config.ts b/packages/cli/tsup.config.ts index 0a85deea..2f17f343 100644 --- a/packages/cli/tsup.config.ts +++ b/packages/cli/tsup.config.ts @@ -1,6 +1,18 @@ import { cpSync, existsSync } from 'node:fs' import { defineConfig } from 'tsup' +/** + * Build-time value for the embedded PostHog project key (see + * `src/telemetry/index.ts`). Only the release workflow sets `STASH_POSTHOG_KEY` + * (from a public repo variable), so every other build — dev, forks, CI — bakes + * in an empty string and the CLI ships telemetry-dormant. Applied to both bundles + * because either may inline the telemetry module. The value must be a JS + * expression string, hence `JSON.stringify`. + */ +const posthogKeyDefine = { + __STASH_POSTHOG_KEY__: JSON.stringify(process.env.STASH_POSTHOG_KEY ?? ''), +} + export default defineConfig([ { entry: ['src/index.ts'], @@ -17,6 +29,7 @@ export default defineConfig([ ...options.logOverride, 'empty-import-meta': 'silent', } + options.define = { ...options.define, ...posthogKeyDefine } }, onSuccess: async () => { // Copy bundled SQL files into dist so they ship with the package @@ -52,5 +65,8 @@ var require = __createRequire(import.meta.url);`, sourcemap: true, skipNodeModulesBundle: true, + esbuildOptions(options) { + options.define = { ...options.define, ...posthogKeyDefine } + }, }, ]) diff --git a/turbo.json b/turbo.json index 64780c60..4ef8a864 100644 --- a/turbo.json +++ b/turbo.json @@ -3,7 +3,8 @@ "tasks": { "build": { "dependsOn": ["^build"], - "inputs": ["$TURBO_DEFAULT$", ".env*"] + "inputs": ["$TURBO_DEFAULT$", ".env*"], + "env": ["STASH_POSTHOG_KEY"] }, "release": { "dependsOn": ["^build"], From 4e26a6b64d42e1ec48db20be88444868161a35e0 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 15 Jul 2026 17:43:01 +1000 Subject: [PATCH 08/10] fix(cli): address code-review findings on telemetry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ten findings from the high-effort review, all fixed: 1. subcommand value leak (privacy): trackCommand emitted argv[1] verbatim, so `stash wizard ""` sent the natural-language prompt (table/column names) to PostHog. New classify-command.ts coerces command/subcommand to the registry's known verbs before emit; anything else becomes ``. 2. non-TTY never activated: the first-run notice only persisted noticeShownAt on a TTY, so agents/piped callers stayed firstRun forever and never sent — the exact population the `caller` dimension measures. Now the disclosure is shown (to stderr) and persisted on every first run, TTY or not. 3. process.exit bypassed the tracking finally: commands exiting via process.exit emitted nothing and never flushed. run() now intercepts process.exit with an ExitSignal sentinel that unwinds through the finally (track + flush) before the real exit, honouring the exit code. Degrades gracefully if a command catch swallows it. 4. empty-string key enabled telemetry: STASH_POSTHOG_KEY='' slipped past the nullish fallback; projectKey() now trims and treats empty as unset. 5. first-run notice could show twice: maybeShowFirstRunNotice didn't reassign the state cache, so a same-run setTelemetryDisabled clobbered noticeShownAt. Fixed. 6. isCiEnv widening leaked into prompt gating: reverted the shared isCiEnv to its narrow (CI-only) behaviour used by region/database-url; telemetry uses a new isCiEnvBroad. Reverted the now-unnecessary test neutralizations. 7. eager cost on every invocation: posthog-node is now a dynamic import inside getClient (loaded only when an event is actually sent), and the state file read is lazy — both off the --version/--help fast paths. 8. shutdown could linger: pass requestTimeout to PostHog so a black-holed endpoint can't keep the socket alive past the flush window. 9. dormant tests depended on ambient STASH_POSTHOG_KEY: clearGateEnv now stubs it. 10. lifecycle path was untested: new telemetry-lifecycle.test.ts covers the emitter/flush contract (no-op when disabled/firstRun, swallows throws, safe double-shutdown, non-TTY notice persistence, bounded flush on a hung endpoint) and classify-command.test.ts covers the value allowlist. Verified end-to-end against a local capture: a process.exit command is now tracked with its exit code preserved, and a wizard prompt containing a table.column is scrubbed to `` (0 leakage). 453 unit tests pass. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- .../cli/src/__tests__/database-url.test.ts | 27 --- packages/cli/src/bin/main.ts | 48 ++++- .../commands/auth/__tests__/region.test.ts | 27 --- packages/cli/src/config/tty.ts | 28 ++- .../__tests__/classify-command.test.ts | 55 +++++ .../__tests__/telemetry-lifecycle.test.ts | 153 ++++++++++++++ .../src/telemetry/__tests__/telemetry.test.ts | 17 +- .../cli/src/telemetry/classify-command.ts | 57 ++++++ packages/cli/src/telemetry/index.ts | 189 ++++++++++++------ 9 files changed, 473 insertions(+), 128 deletions(-) create mode 100644 packages/cli/src/telemetry/__tests__/classify-command.test.ts create mode 100644 packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts create mode 100644 packages/cli/src/telemetry/classify-command.ts diff --git a/packages/cli/src/__tests__/database-url.test.ts b/packages/cli/src/__tests__/database-url.test.ts index 3f03e784..885ce42d 100644 --- a/packages/cli/src/__tests__/database-url.test.ts +++ b/packages/cli/src/__tests__/database-url.test.ts @@ -2,7 +2,6 @@ import fs from 'node:fs' import os from 'node:os' import path from 'node:path' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' -import { CI_ENV_VARS } from '../config/tty.js' import { messages } from '../messages.js' // Mock seams. Hoisted so the in-test reconfiguration touches the same fn @@ -40,30 +39,6 @@ let originalEnv: string | undefined let originalCi: string | undefined let originalIsTty: boolean | undefined let tmpDir: string -// isCiEnv() also consults provider vars (GITHUB_ACTIONS, GITLAB_CI, …); a real -// GITHUB_ACTIONS=true in this repo's own CI would otherwise flip "not CI" tests. -let savedProviderCi: Record = {} - -function neutralizeProviderCi(): void { - savedProviderCi = {} - for (const name of CI_ENV_VARS) { - if (name === 'CI') continue // handled by originalCi above/below - savedProviderCi[name] = process.env[name] - // biome-ignore lint/performance/noDelete: restore exact absence in afterEach. - delete process.env[name] - } -} - -function restoreProviderCi(): void { - for (const [name, value] of Object.entries(savedProviderCi)) { - if (value === undefined) { - // biome-ignore lint/performance/noDelete: restore exact absence. - delete process.env[name] - } else { - process.env[name] = value - } - } -} function noProject() { detect.detectSupabaseProject.mockReturnValue({ @@ -81,7 +56,6 @@ beforeEach(() => { delete process.env.DATABASE_URL // biome-ignore lint/performance/noDelete: ditto. delete process.env.CI - neutralizeProviderCi() tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'database-url-test-')) noProject() }) @@ -99,7 +73,6 @@ afterEach(() => { } else { process.env.CI = originalCi } - restoreProviderCi() Object.defineProperty(process.stdin, 'isTTY', { value: originalIsTty, configurable: true, diff --git a/packages/cli/src/bin/main.ts b/packages/cli/src/bin/main.ts index bb420e0a..bf17de7e 100644 --- a/packages/cli/src/bin/main.ts +++ b/packages/cli/src/bin/main.ts @@ -37,6 +37,7 @@ import { wizardCommand, } from '../commands/index.js' import { messages } from '../messages.js' +import { classifyCommand } from '../telemetry/classify-command.js' import { initTelemetry, maybeShowFirstRunNotice, @@ -412,6 +413,14 @@ async function runSchemaCommand( } } +/** + * Thrown by the `process.exit` override in {@link run} so a command that exits + * deep in the call stack still unwinds through the telemetry finally (track + + * flush) before the process actually terminates. Extends Error so it carries a + * stack and satisfies the "throw an Error" lint, though its payload is unused. + */ +class ExitSignal extends Error {} + // The CLI body. Loaded by the thin launcher in stash.ts via dynamic import so // that a missing native binary (evaluated when this module's command graph // loads) surfaces as friendly guidance rather than a raw stack trace. @@ -445,25 +454,50 @@ export async function run() { maybeShowFirstRunNotice(STASH) const startedAt = Date.now() - let succeeded = true + let success = true let errorType: string | undefined + let exitCode: number | undefined + let thrown: { err: unknown } | undefined + + // Commands terminate deep in dispatch via process.exit(), which would bypass + // the finally below — dropping the event and skipping the flush. Intercept it: + // record the outcome, unwind via ExitSignal to the finally (which tracks + + // flushes), then perform the real exit afterwards. If a command's own catch + // swallows the signal it degrades gracefully — the finally still tracks on the + // eventual return/throw, and the recorded exitCode is still honoured. + const realExit = process.exit.bind(process) + process.exit = ((code?: number): never => { + exitCode = code ?? 0 + success = exitCode === 0 + throw new ExitSignal() + }) as typeof process.exit try { await dispatch(command, subcommand, commandArgs, flags, values) } catch (err) { - succeeded = false - errorType = err instanceof Error ? err.constructor.name : 'Unknown' - throw err + if (!(err instanceof ExitSignal)) { + success = false + errorType = err instanceof Error ? err.constructor.name : 'Unknown' + thrown = { err } + } + // ExitSignal: success/exitCode were already set by the process.exit override. } finally { + process.exit = realExit + // Coerce command/subcommand to a known vocabulary before emit so a free-text + // positional (e.g. a `stash wizard ""` description) never leaves. + const safe = classifyCommand(command, subcommand) trackCommand({ - command, - subcommand, - success: succeeded, + command: safe.command, + subcommand: safe.subcommand, + success, durationMs: Date.now() - startedAt, errorType, }) await shutdownTelemetry() } + + if (thrown !== undefined) throw thrown.err + if (exitCode !== undefined) realExit(exitCode) } async function dispatch( diff --git a/packages/cli/src/commands/auth/__tests__/region.test.ts b/packages/cli/src/commands/auth/__tests__/region.test.ts index 819ec719..838bfac6 100644 --- a/packages/cli/src/commands/auth/__tests__/region.test.ts +++ b/packages/cli/src/commands/auth/__tests__/region.test.ts @@ -1,5 +1,4 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' -import { CI_ENV_VARS } from '../../../config/tty.js' import { messages } from '../../../messages.js' // region.ts imports `* as p from '@clack/prompts'` (no native `@cipherstash/auth` @@ -30,30 +29,6 @@ const { let originalRegionEnv: string | undefined let originalCi: string | undefined let originalIsTty: boolean | undefined -// isCiEnv() also consults provider vars (GITHUB_ACTIONS, GITLAB_CI, …); a real -// GITHUB_ACTIONS=true in this repo's own CI would otherwise flip "not CI" tests. -let savedProviderCi: Record = {} - -function neutralizeProviderCi(): void { - savedProviderCi = {} - for (const name of CI_ENV_VARS) { - if (name === 'CI') continue // handled by originalCi above/below - savedProviderCi[name] = process.env[name] - // biome-ignore lint/performance/noDelete: restore exact absence in afterEach. - delete process.env[name] - } -} - -function restoreProviderCi(): void { - for (const [name, value] of Object.entries(savedProviderCi)) { - if (value === undefined) { - // biome-ignore lint/performance/noDelete: restore exact absence. - delete process.env[name] - } else { - process.env[name] = value - } - } -} function setTty(value: boolean | undefined) { Object.defineProperty(process.stdin, 'isTTY', { @@ -77,7 +52,6 @@ beforeEach(() => { delete process.env[REGION_ENV_VAR] // biome-ignore lint/performance/noDelete: ditto. delete process.env.CI - neutralizeProviderCi() }) afterEach(() => { @@ -93,7 +67,6 @@ afterEach(() => { } else { process.env.CI = originalCi } - restoreProviderCi() setTty(originalIsTty) vi.restoreAllMocks() }) diff --git a/packages/cli/src/config/tty.ts b/packages/cli/src/config/tty.ts index 0ea35fbd..dbb45dbc 100644 --- a/packages/cli/src/config/tty.ts +++ b/packages/cli/src/config/tty.ts @@ -36,16 +36,30 @@ export const CI_ENV_VARS = [ ] as const /** - * True in a CI environment. `CI`/`CONTINUOUS_INTEGRATION` set to a truthy - * spelling (`true`/`1`) covers most providers — including GitHub Actions and - * GitLab — and {@link CI_PROVIDER_ENV_VARS} catches the ones that don't set - * `CI`. Shared by the region resolver (`commands/auth/region.ts`), the - * DATABASE_URL resolver (`config/database-url.ts`), and telemetry gating so they - * all decide "is this CI?" the same way. + * True when `CI` is set to a truthy spelling (`1`/`true`) — the near-universal + * marker, set by GitHub Actions, GitLab, CircleCI, and most others. This is the + * NARROW check that governs interactive prompting: the region resolver + * (`commands/auth/region.ts`), the DATABASE_URL resolver (`config/database-url.ts`), + * and {@link isInteractive}. It is intentionally conservative about suppressing a + * prompt — a developer whose shell happens to export a provider marker like + * `JENKINS_URL` should still get prompted. Telemetry uses the broader + * {@link isCiEnvBroad} instead, where erring toward "don't collect" is safer. */ export function isCiEnv(): boolean { const ciVar = process.env.CI?.trim() - if (ciVar !== undefined && /^(1|true)$/i.test(ciVar)) return true + return ciVar !== undefined && /^(1|true)$/i.test(ciVar) +} + +/** + * True in CI, detected broadly: {@link isCiEnv} plus `CONTINUOUS_INTEGRATION` and + * the {@link CI_PROVIDER_ENV_VARS} markers for systems that don't set `CI` + * (Jenkins, TeamCity, Azure Pipelines, …). Used ONLY for telemetry auto-disable, + * where a false positive merely skips collection — unlike prompt gating, where a + * false positive breaks an interactive command. Keep it separate from + * {@link isCiEnv} so broadening CI-for-telemetry never changes prompt behavior. + */ +export function isCiEnvBroad(): boolean { + if (isCiEnv()) return true if (process.env.CONTINUOUS_INTEGRATION?.trim()) return true return CI_PROVIDER_ENV_VARS.some((name) => Boolean(process.env[name]?.trim())) } diff --git a/packages/cli/src/telemetry/__tests__/classify-command.test.ts b/packages/cli/src/telemetry/__tests__/classify-command.test.ts new file mode 100644 index 00000000..34310ef7 --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/classify-command.test.ts @@ -0,0 +1,55 @@ +import { describe, expect, it } from 'vitest' +import { classifyCommand } from '../classify-command.js' + +describe('classifyCommand (telemetry value allowlist)', () => { + it('keeps a recognised command + subcommand path', () => { + expect(classifyCommand('eql', 'install')).toEqual({ + command: 'eql', + subcommand: 'install', + }) + expect(classifyCommand('auth', 'login')).toEqual({ + command: 'auth', + subcommand: 'login', + }) + }) + + it('keeps a recognised command with no subcommand', () => { + expect(classifyCommand('init', undefined)).toEqual({ + command: 'init', + subcommand: undefined, + }) + }) + + it('keeps the telemetry sub-verbs (not in the registry, but known-safe)', () => { + for (const verb of ['status', 'enable', 'disable']) { + expect(classifyCommand('telemetry', verb)).toEqual({ + command: 'telemetry', + subcommand: verb, + }) + } + }) + + it('coerces a free-text positional to — the wizard-prompt leak', () => { + // The core fix: `stash wizard "add encryption to patients.ssn"` must never + // send the prompt (which carries table/column names) as the subcommand value. + expect( + classifyCommand('wizard', 'add searchable encryption to patients.ssn'), + ).toEqual({ command: 'wizard', subcommand: '' }) + }) + + it('keeps the command but drops an unrecognised subcommand to ', () => { + expect(classifyCommand('eql', 'rm -rf /')).toEqual({ + command: 'eql', + subcommand: '', + }) + }) + + it('coerces an entirely unknown command to and drops its subcommand', () => { + expect(classifyCommand('definitely-not-a-command', 'secret-value')).toEqual( + { + command: '', + subcommand: undefined, + }, + ) + }) +}) diff --git a/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts b/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts new file mode 100644 index 00000000..2da7304e --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts @@ -0,0 +1,153 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { CI_ENV_VARS } from '../../config/tty.js' +import type { TelemetryState } from '../state.js' + +// Controllable doubles for posthog-node and the on-disk state, wired via +// vi.hoisted so the (hoisted) vi.mock factories can close over them. +const h = vi.hoisted(() => ({ + capture: vi.fn(), + shutdown: vi.fn(async () => {}), + PostHog: vi.fn(), + writeState: vi.fn(), + state: { value: undefined as TelemetryState | undefined }, +})) + +vi.mock('posthog-node', () => ({ PostHog: h.PostHog })) +vi.mock('../state.js', () => ({ + readState: () => h.state.value, + writeState: (s: TelemetryState) => { + h.writeState(s) + h.state.value = s + return s + }, +})) + +/** Re-import the module fresh so its lazily-initialised caches reset per test. */ +async function load() { + vi.resetModules() + return import('../index.js') +} + +const ENABLED: TelemetryState = { + anonymousId: 'anon-x', + telemetryDisabled: false, + noticeShownAt: '2026-01-01T00:00:00.000Z', +} + +describe('telemetry lifecycle (emitter + flush)', () => { + beforeEach(() => { + h.capture.mockReset() + h.shutdown.mockReset().mockResolvedValue(undefined) + h.PostHog.mockReset().mockImplementation(() => ({ + capture: h.capture, + shutdown: h.shutdown, + })) + h.writeState.mockReset() + h.state.value = { ...ENABLED } + // Enabled by default; neutralize every gate so status is { enabled: true }. + vi.stubEnv('STASH_POSTHOG_KEY', 'phc_test_key') + for (const name of [ + 'DO_NOT_TRACK', + 'STASH_TELEMETRY_DISABLED', + ...CI_ENV_VARS, + ]) { + vi.stubEnv(name, '') + } + }) + afterEach(() => { + vi.unstubAllEnvs() + vi.useRealTimers() + }) + + it('trackCommand builds no client and sends nothing when disabled', async () => { + vi.stubEnv('STASH_POSTHOG_KEY', '') // dormant + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await t.shutdownTelemetry() + expect(h.PostHog).not.toHaveBeenCalled() + expect(h.capture).not.toHaveBeenCalled() + }) + + it('trackCommand is a no-op on the first run (freebie)', async () => { + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // no noticeShownAt + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await t.shutdownTelemetry() + expect(h.PostHog).not.toHaveBeenCalled() + }) + + it('captures one anonymous event and flushes on shutdown', async () => { + const t = await load() + t.initTelemetry('9.9.9') + t.trackCommand({ + command: 'eql', + subcommand: 'install', + success: true, + durationMs: 5, + }) + await t.shutdownTelemetry() + expect(h.PostHog).toHaveBeenCalledTimes(1) + expect(h.capture).toHaveBeenCalledTimes(1) + const arg = h.capture.mock.calls[0][0] + expect(arg.distinctId).toBe('anon-x') + expect(arg.event).toBe('command_invoked') + expect(arg.properties.command).toBe('eql') + expect(arg.properties.cliVersion).toBe('9.9.9') + expect(arg.properties.$process_person_profile).toBe(false) + expect(h.shutdown).toHaveBeenCalledTimes(1) + }) + + it('swallows a throwing capture and still shuts down cleanly', async () => { + h.capture.mockImplementation(() => { + throw new Error('boom') + }) + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + }) + + it('shutdownTelemetry is a safe no-op when nothing was captured, even twice', async () => { + const t = await load() + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + expect(h.shutdown).not.toHaveBeenCalled() + }) + + it('shows + persists the first-run notice even when stderr is NOT a TTY, and keeps noticeShownAt across setTelemetryDisabled', async () => { + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // firstRun + const write = vi.spyOn(process.stderr, 'write').mockReturnValue(true) + const originalTty = process.stderr.isTTY + Object.defineProperty(process.stderr, 'isTTY', { + value: false, + configurable: true, + }) + try { + const t = await load() + t.maybeShowFirstRunNotice('npx stash') + expect(write).toHaveBeenCalledTimes(1) + expect(h.writeState).toHaveBeenCalledTimes(1) + expect(h.writeState.mock.calls[0][0].noticeShownAt).toBeDefined() + // Fix: the notice reassigns the state cache, so a later opt-out write does + // not clobber the just-persisted noticeShownAt (the "shows twice" bug). + t.setTelemetryDisabled(false) + const lastWrite = h.writeState.mock.calls.at(-1)?.[0] + expect(lastWrite?.noticeShownAt).toBeDefined() + } finally { + Object.defineProperty(process.stderr, 'isTTY', { + value: originalTty, + configurable: true, + }) + write.mockRestore() + } + }) + + it('shutdownTelemetry resolves within the flush timeout when shutdown() hangs', async () => { + // A black-holed endpoint: shutdown() never resolves. The bounded-flush + // guarantee must still let the process continue via the ~1500ms timeout. + // Real timers (fake timers don't drive the dynamic import); headroom below. + h.shutdown.mockImplementation(() => new Promise(() => {})) + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + }, 4000) +}) diff --git a/packages/cli/src/telemetry/__tests__/telemetry.test.ts b/packages/cli/src/telemetry/__tests__/telemetry.test.ts index 01c3d016..fd89550c 100644 --- a/packages/cli/src/telemetry/__tests__/telemetry.test.ts +++ b/packages/cli/src/telemetry/__tests__/telemetry.test.ts @@ -19,11 +19,14 @@ function withKey(): void { vi.stubEnv('STASH_POSTHOG_KEY', 'phc_test_key') } -/** Clear every env var that resolveStatus / isCiEnv consult. */ +/** Clear every env var that resolveStatus / isCiEnvBroad consult, INCLUDING + * STASH_POSTHOG_KEY — a real value in the ambient shell (this repo sets it as a + * GitHub Actions variable) would otherwise flip the dormant tests to enabled. */ function clearGateEnv(): void { for (const name of [ 'DO_NOT_TRACK', 'STASH_TELEMETRY_DISABLED', + 'STASH_POSTHOG_KEY', ...CI_ENV_VARS, ]) { vi.stubEnv(name, '') @@ -75,6 +78,18 @@ describe('resolveStatus gates', () => { expect(PLACEHOLDER_KEY).toBe('__STASH_POSTHOG_KEY__') }) + it('treats an empty/whitespace STASH_POSTHOG_KEY as unset → dormant', () => { + // The `?? EMBEDDED_KEY` nullish fallback would let '' slip through and flip a + // dormant build to enabled with an empty key; projectKey() trims it away. + for (const value of ['', ' ']) { + vi.stubEnv('STASH_POSTHOG_KEY', value) + expect(resolveStatus(enabledState)).toEqual({ + enabled: false, + reason: 'unconfigured', + }) + } + }) + it('is enabled with a key and no opt-out signals', () => { withKey() expect(resolveStatus(enabledState)).toEqual({ enabled: true }) diff --git a/packages/cli/src/telemetry/classify-command.ts b/packages/cli/src/telemetry/classify-command.ts new file mode 100644 index 00000000..1a2a63cc --- /dev/null +++ b/packages/cli/src/telemetry/classify-command.ts @@ -0,0 +1,57 @@ +import { registry } from '../cli/registry.js' + +/** + * Coerce the raw, argv-derived (command, subcommand) to a fixed vocabulary + * before telemetry emit. + * + * `subcommand` is `argv[1]` (see `parseArgs`): for most commands it's a fixed + * verb, but for a few it's a free-form positional — `stash wizard "add + * encryption to patients.ssn"` puts a natural-language prompt (routinely + * containing table/column names) in that slot. The event allowlist gates + * property KEYS, not values, so without this an arbitrary user string would + * leave the machine as the `subcommand` value, breaking the "no table/column + * names, no argument values" contract. Anything not on the known-verb list + * collapses to ``, so we learn "a wizard ran" without learning what for. + */ +const OTHER = '' + +/** + * Full command paths (`"eql install"`, `"auth login"`, `"init"`, …) from the + * registry — the single source of truth that also backs `stash manifest` and + * `--help` — plus the `telemetry` sub-verbs the registry doesn't enumerate. + * Hidden (deprecated) descriptors are excluded. + */ +const KNOWN_PATHS: ReadonlySet = new Set([ + ...registry.flatMap((group) => + group.commands.filter((c) => !c.hidden).map((c) => c.name), + ), + 'telemetry status', + 'telemetry enable', + 'telemetry disable', +]) + +/** The first token of every known path — the set of recognised top-level commands. */ +const KNOWN_COMMANDS: ReadonlySet = new Set( + [...KNOWN_PATHS].map((path) => path.split(' ')[0]), +) + +/** + * Return telemetry-safe (command, subcommand). An unrecognised command becomes + * `` (and drops its subcommand); a recognised command with an + * unrecognised subcommand keeps the command but reports `` for the + * subcommand, so a free-text positional never leaves as-is. + */ +export function classifyCommand( + command: string, + subcommand: string | undefined, +): { command: string; subcommand: string | undefined } { + if (!KNOWN_COMMANDS.has(command)) { + return { command: OTHER, subcommand: undefined } + } + if (subcommand === undefined) { + return { command, subcommand: undefined } + } + return KNOWN_PATHS.has(`${command} ${subcommand}`) + ? { command, subcommand } + : { command, subcommand: OTHER } +} diff --git a/packages/cli/src/telemetry/index.ts b/packages/cli/src/telemetry/index.ts index 94efbd8a..819d36eb 100644 --- a/packages/cli/src/telemetry/index.ts +++ b/packages/cli/src/telemetry/index.ts @@ -1,5 +1,5 @@ -import { PostHog } from 'posthog-node' -import { isCiEnv, resolveCaller } from '../config/tty.js' +import type { PostHog } from 'posthog-node' +import { isCiEnvBroad, resolveCaller } from '../config/tty.js' import { messages } from '../messages.js' import { readState, type TelemetryState, writeState } from './state.js' @@ -15,9 +15,14 @@ import { readState, type TelemetryState, writeState } from './state.js' * - Events are anonymous (no PostHog "person" profiles) and carry only a fixed * allowlist of coarse properties. Table/column/schema names, connection * strings, plaintext, ciphertext, and raw argument values can never leave — - * see {@link sanitize}. - * - Sending never blocks or slows the CLI: the client is built lazily, flushing - * is bounded by a timeout, and every failure is swallowed. + * {@link sanitize} enforces the key allowlist, and callers must coerce the + * VALUES of `command`/`subcommand` to a known vocabulary first (the raw argv + * token is a value, not a key, so sanitize alone would pass it through — see + * `classifyCommand` in `./classify-command.ts`). + * - Sending never blocks or slows the CLI: this module reads no disk and loads + * no posthog-node until a real event is actually sent (both are deferred off + * the `--version`/`--help` fast paths), flushing is bounded by a timeout, and + * every failure is swallowed. */ /** Default endpoint — our Cloudflare proxy, not PostHog directly, so a future @@ -58,9 +63,13 @@ const EMBEDDED_KEY = : PLACEHOLDER_KEY /** `STASH_POSTHOG_KEY` in the environment overrides the embedded key entirely - * (testing / self-hosting); otherwise the build-time value is used. */ + * (testing / self-hosting); otherwise the build-time value is used. An empty or + * whitespace-only override counts as unset — otherwise `STASH_POSTHOG_KEY=''` + * would slip past the nullish fallback and flip a dormant build to "enabled" + * with an empty key (asymmetric with the `.length > 0` guard on EMBEDDED_KEY). */ function projectKey(): string { - return process.env.STASH_POSTHOG_KEY ?? EMBEDDED_KEY + const override = process.env.STASH_POSTHOG_KEY?.trim() + return override ? override : EMBEDDED_KEY } /** Flush is bounded so a slow or unreachable endpoint can't hang `stash`. */ @@ -122,7 +131,7 @@ export function resolveStatus(state: TelemetryState): TelemetryStatus { if (envOptOut('STASH_TELEMETRY_DISABLED')) { return { enabled: false, reason: 'stash-disabled' } } - if (isCiEnv()) return { enabled: false, reason: 'ci' } + if (isCiEnvBroad()) return { enabled: false, reason: 'ci' } if (state.telemetryDisabled) return { enabled: false, reason: 'config' } return { enabled: true } } @@ -138,15 +147,42 @@ export function sanitize( return out } -// Module-level state, resolved once per process at import time. -let state = readState() -let status = resolveStatus(state) -/** True when this is the first-ever run (notice not yet shown). It stays true - * for the whole process even after we persist the notice, so the current run - * never sends — the freebie. */ -const firstRun = state.noticeShownAt === undefined +// Module state is resolved LAZILY on first telemetry use, not at import time, so +// fast paths that never call a telemetry function (`--version`, `--help`) touch +// no disk — the state file read (and its randomUUID on a fresh machine) is +// deferred out of the CLI's hottest paths. +let stateCache: TelemetryState | undefined +let statusCache: TelemetryStatus | undefined +let firstRunCache = false + +/** + * Read the state file once per process and derive status + first-run. Returns + * the resolved values so callers never touch the (possibly stale) module caches + * directly. `firstRun` stays true for the whole process even after the notice is + * persisted, so the current run never sends — the freebie. + */ +function init(): { + state: TelemetryState + status: TelemetryStatus + firstRun: boolean +} { + if (stateCache === undefined || statusCache === undefined) { + stateCache = readState() + statusCache = resolveStatus(stateCache) + firstRunCache = stateCache.noticeShownAt === undefined + } + return { state: stateCache, status: statusCache, firstRun: firstRunCache } +} + let cliVersion = '0.0.0' -let client: PostHog | null = null +// posthog-node is imported dynamically the first time an event is actually sent +// (see getClient), so it is NOT pulled into every `stash` invocation — only when +// telemetry is enabled AND a real command is being tracked. clientPromise is the +// "was anything ever sent this process?" signal shutdownTelemetry gates on. +let clientPromise: Promise | null = null +// The last capture()'s enqueue promise; shutdownTelemetry awaits it before +// flushing so an event fired on an exit path is delivered, not raced away. +let lastCapture: Promise | null = null /** Provide the CLI version for event properties. Call once at startup. */ export function initTelemetry(version: string): void { @@ -154,25 +190,32 @@ export function initTelemetry(version: string): void { } export function telemetryStatus(): TelemetryStatus { - return status + return init().status } -function getClient(): PostHog { - if (client === null) { - client = new PostHog(projectKey(), { - host: posthogHost(), - flushAt: 1, - flushInterval: 0, - // Fire-and-forget: a short-lived CLI must not retry a failed send. Retries - // (default 3, exponential backoff) would keep internal timers pending and - // hang the process for seconds when the endpoint is unreachable, defeating - // the bounded-flush guarantee. Drop the event instead. - fetchRetryCount: 0, - // Never resolve IP → geo; we don't want or store location. - disableGeoip: true, - }) +async function getClient(): Promise { + if (clientPromise === null) { + clientPromise = import('posthog-node').then( + ({ PostHog }) => + new PostHog(projectKey(), { + host: posthogHost(), + flushAt: 1, + flushInterval: 0, + // Fire-and-forget: a short-lived CLI must not retry a failed send. + // Retries (default 3, exponential backoff) would keep internal timers + // pending and hang the process. Drop the event instead. + fetchRetryCount: 0, + // Bound PostHog's OWN request (undici AbortSignal.timeout). Without + // this it defaults to 10s, so a black-holed endpoint keeps the socket + // — and the event loop — alive long past our flush window, defeating + // the bounded-flush guarantee. Match the flush budget. + requestTimeout: FLUSH_TIMEOUT_MS, + // Never resolve IP → geo; we don't want or store location. + disableGeoip: true, + }), + ) } - return client + return clientPromise } function baseProps(): Record { @@ -201,42 +244,56 @@ export interface CommandEvent { * first run. Never throws. */ export function trackCommand(event: CommandEvent): void { + const { state, status, firstRun } = init() if (!status.enabled || firstRun) return - try { - getClient().capture({ - distinctId: state.anonymousId, - event: 'command_invoked', - properties: { - // Sanitize the FULL property set (event + base) so the allowlist is the - // single enforced boundary — a future prop added to baseProps() can't - // bypass it. Only the explicit PostHog control key is added afterward. - ...sanitize({ ...event, ...baseProps() }), - // Keep events anonymous: no PostHog person profiles. - $process_person_profile: false, - }, - }) - } catch { - // Telemetry must never surface in the command path. + const properties = { + // Sanitize the FULL property set (event + base) so the allowlist is the + // single enforced boundary — a future prop added to baseProps() can't + // bypass it. Only the explicit PostHog control key is added afterward. + ...sanitize({ ...event, ...baseProps() }), + // Keep events anonymous: no PostHog person profiles. + $process_person_profile: false, } + // Fire-and-forget. getClient() dynamically imports posthog-node only now, so a + // disabled/dormant run never loads it. shutdownTelemetry awaits lastCapture + // before flushing so the event is delivered even on process.exit paths. + lastCapture = getClient() + .then((client) => { + client.capture({ + distinctId: state.anonymousId, + event: 'command_invoked', + properties, + }) + }) + .catch(() => { + // Telemetry must never surface in the command path. + }) } /** * Show the one-time first-run notice (to stderr, so it never pollutes piped or * `--json` stdout) and mark it shown. No-op when telemetry is disabled or the - * notice was already shown. The run that shows it sends nothing (see {@link firstRun}). + * notice was already shown. The run that shows it sends nothing (the freebie). * - * `noticeShownAt` is persisted ONLY when the notice was actually displayed — - * i.e. stderr is a TTY. A non-interactive first run therefore does not consume - * the freebie: telemetry stays dormant until a real run has shown the disclosure. - * `stashRef` is the runner-aware invocation (e.g. `npx stash`) so the opt-out - * hint is actionable before the CLI is on PATH. + * The disclosure is written on EVERY first run, TTY or not — a non-interactive + * caller (an agent harness, a piped invocation) still gets it in its stderr/log, + * and `noticeShownAt` is persisted so that machine advances past the freebie and + * starts sending on its next run. Gating persistence on a TTY (as before) left + * exactly the agent population the `caller` dimension exists to measure dormant + * forever. `stashRef` is the runner-aware invocation (e.g. `npx stash`) so the + * opt-out hint is actionable before the CLI is on PATH. */ export function maybeShowFirstRunNotice(stashRef: string): void { + const { state, status, firstRun } = init() if (!status.enabled || !firstRun) return - if (!process.stderr.isTTY) return process.stderr.write(`${messages.telemetry.notice(stashRef)}\n`) try { - writeState({ ...state, noticeShownAt: new Date().toISOString() }) + // Reassign the cache: a later setTelemetryDisabled() in the same process + // must not write from a stale `state` and clobber this noticeShownAt. + stateCache = writeState({ + ...state, + noticeShownAt: new Date().toISOString(), + }) } catch { // A write failure just means we may show the notice again next run. } @@ -244,18 +301,31 @@ export function maybeShowFirstRunNotice(stashRef: string): void { /** Persist the opt-out flag. Surfaces write errors (the command wants to know). */ export function setTelemetryDisabled(disabled: boolean): void { - state = writeState({ ...state, telemetryDisabled: disabled }) - status = resolveStatus(state) + const { state } = init() + stateCache = writeState({ ...state, telemetryDisabled: disabled }) + statusCache = resolveStatus(stateCache) } /** Flush any buffered events, bounded by {@link FLUSH_TIMEOUT_MS}. Never throws. */ export async function shutdownTelemetry(): Promise { - if (client === null) return + // clientPromise is null iff nothing was ever captured (disabled/dormant/first + // run), so there is nothing to flush and posthog-node was never loaded. + if (clientPromise === null) return // The timer MUST be cleared once shutdown() wins the race: an uncleared // pending setTimeout keeps the Node event loop alive, so the process would // hang for the full timeout after the flush already completed. let timer: ReturnType | undefined try { + // Wait for the capture enqueue (client construction + queueing) to finish, + // otherwise shutdown() could race ahead of the event we mean to deliver. + if (lastCapture !== null) { + try { + await lastCapture + } catch { + // A failed capture is already swallowed in trackCommand. + } + } + const client = await clientPromise await Promise.race([ client.shutdown(), new Promise((resolve) => { @@ -266,6 +336,7 @@ export async function shutdownTelemetry(): Promise { // Swallow — a failed flush must not fail the process. } finally { if (timer !== undefined) clearTimeout(timer) - client = null + clientPromise = null + lastCapture = null } } From d8e0c1d6db4c1b6febd4dee72be12581baa562c7 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 15 Jul 2026 18:24:16 +1000 Subject: [PATCH 09/10] fix(cli): replace the process.exit override with cooperative CliExit; harden telemetry per review round 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The round-1 fix intercepted process.exit globally with a thrown signal. The second review pass proved that unsafe: @clack/core calls process.exit(0) from a keypress handler (Ctrl+C during any spinner became an uncaughtException crash), and five broad catches (init's install-eql, plan, status, impl, the jiti config loader) swallowed the signal — init continued past an explicit cancel, impl skipped the cutover deploy-gate, and commands that recovered by design exited with a stale latched code. New design: no global interception. First-party sites verified to unwind cleanly to run() throw `CliExit(code)` (new cli/exit.ts) — main.ts's own unknown-command/subcommand defaults, requireValue, requireStack, the telemetry command, and the outermost CancelledError handlers of init/plan/impl (cancels are now tracked). Deep process.exit sites revert to pre-branch behaviour: immediate, uninterceptable, untracked — documented as a known measurement gap in the telemetry module doc and packages/cli/AGENTS.md. Also fixed from the review: - shutdownTelemetry: the capture enqueue (containing the dynamic posthog-node import) now sits INSIDE the Promise.race, so the 1500ms bound covers everything that can stall; the flush promise carries its own rejection handler so a timeout win can't leave an unhandledRejection. - First-run notice persists BEFORE printing: an unwritable HOME no longer prints the notice on every run forever — no persist, no print, no telemetry. - pty e2e harness sets STASH_TELEMETRY_DISABLED=1 so an ambient STASH_POSTHOG_KEY can't make tests send real events against the developer's real ~/.cipherstash. - errorType is now a closed vocabulary (classifyErrorType): user-defined error class names from jiti-loaded config code collapse to ``. - Wizard analytics aligned with the CLI's privacy contract: honors DO_NOT_TRACK / STASH_TELEMETRY_DISABLED / CI, random per-session id instead of sha256(username@hostname), geoip off, and error events carry fixed labels / class names instead of raw messages. - Docs de-staled: packages/cli/AGENTS.md telemetry section rewritten (it said the CLI had no telemetry), CI_ENV_VARS/resolveStatus/pty.ts comments now describe the isCiEnv (narrow, prompts) vs isCiEnvBroad (telemetry) split, and the changeset names the value-coercion boundary. - Tests: narrow-vs-broad isCiEnv contract pinned; classifyErrorType covered; notice write-failure covered; stderr.isTTY save/restore no longer creates a read-only own property; smoke e2e covers `telemetry status` and the bogus-subcommand CliExit path. Verified: 460 unit + 57 pty e2e tests pass; local capture shows unknown command and telemetry bogus-sub tracked with correct exit codes, wizard free-text still scrubbed to `` (0 leakage), and a read-only-HOME first run prints nothing. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- .changeset/cli-anonymous-telemetry.md | 6 +- .changeset/wizard-analytics-privacy.md | 11 +++ packages/cli/AGENTS.md | 19 ++++-- packages/cli/src/bin/main.ts | 63 +++++++---------- packages/cli/src/cli/exit.ts | 24 +++++++ packages/cli/src/commands/impl/index.ts | 5 +- packages/cli/src/commands/init/index.ts | 5 +- packages/cli/src/commands/plan/index.ts | 5 +- packages/cli/src/commands/telemetry/index.ts | 3 +- packages/cli/src/config/__tests__/tty.test.ts | 48 ++++++++++++- packages/cli/src/config/tty.ts | 7 +- .../__tests__/classify-command.test.ts | 23 ++++++- .../__tests__/telemetry-lifecycle.test.ts | 68 ++++++++++++++----- .../cli/src/telemetry/classify-command.ts | 38 +++++++++++ packages/cli/src/telemetry/index.ts | 62 ++++++++++++----- packages/cli/tests/e2e/smoke.e2e.test.ts | 17 +++++ packages/cli/tests/helpers/pty.ts | 18 +++-- packages/wizard/src/lib/analytics.ts | 55 +++++++++++---- packages/wizard/src/run.ts | 12 +++- 19 files changed, 380 insertions(+), 109 deletions(-) create mode 100644 .changeset/wizard-analytics-privacy.md create mode 100644 packages/cli/src/cli/exit.ts diff --git a/.changeset/cli-anonymous-telemetry.md b/.changeset/cli-anonymous-telemetry.md index 1cd88530..98b1e6c4 100644 --- a/.changeset/cli-anonymous-telemetry.md +++ b/.changeset/cli-anonymous-telemetry.md @@ -10,8 +10,10 @@ version, success/failure, duration, and a coarse caller class (e.g. `claude-code`, `cursor`, `interactive`) derived from environment markers so we can gauge agent- vs human-driven usage. Plaintext, schema, table/column names, connection strings, argument values, and any session/trace identifier are never -collected (enforced by a property allowlist at the emitter boundary). A one-time -notice is shown on first run, and nothing is sent on that run. +collected — enforced by a property-key allowlist at the emitter boundary plus +closed-vocabulary coercion of every argv- or error-derived value (unrecognised +commands, subcommands, and error class names all collapse to ``). A +one-time notice is shown on first run, and nothing is sent on that run. Telemetry is off by default in CI and can be disabled with `DO_NOT_TRACK=1` (the cross-tool standard), `STASH_TELEMETRY_DISABLED=1`, or diff --git a/.changeset/wizard-analytics-privacy.md b/.changeset/wizard-analytics-privacy.md new file mode 100644 index 00000000..c4799b7d --- /dev/null +++ b/.changeset/wizard-analytics-privacy.md @@ -0,0 +1,11 @@ +--- +'@cipherstash/wizard': patch +--- + +Align the wizard's analytics with the `stash` CLI's telemetry privacy contract. +The wizard now honors `DO_NOT_TRACK`, `STASH_TELEMETRY_DISABLED`, and CI +auto-detection; uses a random per-session identifier instead of one derived +from username@hostname; disables IP→geo resolution; and reports error events as +fixed labels / error class names instead of raw messages (which could embed +schema names or connection details). Analytics remain dormant unless a PostHog +key is configured at build time. diff --git a/packages/cli/AGENTS.md b/packages/cli/AGENTS.md index c3ba4d68..73795c3b 100644 --- a/packages/cli/AGENTS.md +++ b/packages/cli/AGENTS.md @@ -81,11 +81,20 @@ exercise the same code paths. test actually asserts on the string — premature extraction is worse than copy-paste here. For literals tests don't touch (e.g. command names like `init`, `eql install`), keep them inline. -- **Telemetry.** The CLI source no longer imports `posthog-node` (analytics - moved to `packages/wizard`). The dep is still listed in `package.json` - and should be removed in a follow-up. If you re-introduce telemetry to - the CLI, gate construction on an env var (the wizard's - `getClient()` pattern) so E2E tests can no-op it. +- **Telemetry.** The CLI has anonymous, opt-out usage analytics in + `src/telemetry/` (posthog-node, loaded lazily only when an event is + actually sent). It ships dormant — a real project key is embedded only by + the release build (`STASH_POSTHOG_KEY` repo variable → tsup define) — and + is force-disabled in every pty e2e child via `STASH_TELEMETRY_DISABLED=1` + in `tests/helpers/pty.ts`. Two contracts to preserve when touching it: + (1) event VALUES are closed vocabularies — `classifyCommand` / + `classifyErrorType` in `src/telemetry/classify-command.ts` must wrap + anything argv- or error-derived before emit; (2) never intercept + `process.exit` with a thrown signal — @clack/core exits from keypress + handlers and several commands have broad catches, so interception breaks + cancel flows (tried and reverted; see `src/cli/exit.ts`). Commands that + terminate via deep `process.exit()` are simply not tracked; cooperative + exits use `throw new CliExit(code)` from verified-unwindable sites only. ## Plan and rationale diff --git a/packages/cli/src/bin/main.ts b/packages/cli/src/bin/main.ts index bf17de7e..ee7e2ab7 100644 --- a/packages/cli/src/bin/main.ts +++ b/packages/cli/src/bin/main.ts @@ -19,6 +19,7 @@ import { readFileSync } from 'node:fs' import { dirname, join } from 'node:path' import { fileURLToPath } from 'node:url' import * as p from '@clack/prompts' +import { CliExit } from '../cli/exit.js' import { renderCommandHelp } from '../cli/help.js' // Commands that depend on @cipherstash/stack are lazy-loaded in the switch below. import { @@ -37,7 +38,10 @@ import { wizardCommand, } from '../commands/index.js' import { messages } from '../messages.js' -import { classifyCommand } from '../telemetry/classify-command.js' +import { + classifyCommand, + classifyErrorType, +} from '../telemetry/classify-command.js' import { initTelemetry, maybeShowFirstRunNotice, @@ -80,7 +84,7 @@ async function requireStack(importFn: () => Promise): Promise { Install it with: ${prodInstallCommand(PM, '@cipherstash/stack')} Or run: ${STASH} init`, ) - process.exit(1) as never + throw new CliExit(1) } throw err } @@ -237,7 +241,7 @@ async function runEqlCommand( p.log.error(`${messages.eql.unknownSubcommand}: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -301,7 +305,7 @@ async function runDbCommand( p.log.error(`${messages.db.unknownSubcommand}: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -376,7 +380,7 @@ async function runEncryptCommand( p.log.error(`Unknown encrypt subcommand: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -384,7 +388,7 @@ function requireValue(values: Record, key: string): string { const v = values[key] if (!v) { p.log.error(`Missing required --${key} value.`) - process.exit(1) + throw new CliExit(1) } return v } @@ -409,18 +413,10 @@ async function runSchemaCommand( p.log.error(`Unknown schema subcommand: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } -/** - * Thrown by the `process.exit` override in {@link run} so a command that exits - * deep in the call stack still unwinds through the telemetry finally (track + - * flush) before the process actually terminates. Extends Error so it carries a - * stack and satisfies the "throw an Error" lint, though its payload is unused. - */ -class ExitSignal extends Error {} - // The CLI body. Loaded by the thin launcher in stash.ts via dynamic import so // that a missing native binary (evaluated when this module's command graph // loads) surfaces as friendly guidance rather than a raw stack trace. @@ -457,32 +453,26 @@ export async function run() { let success = true let errorType: string | undefined let exitCode: number | undefined - let thrown: { err: unknown } | undefined - - // Commands terminate deep in dispatch via process.exit(), which would bypass - // the finally below — dropping the event and skipping the flush. Intercept it: - // record the outcome, unwind via ExitSignal to the finally (which tracks + - // flushes), then perform the real exit afterwards. If a command's own catch - // swallows the signal it degrades gracefully — the finally still tracks on the - // eventual return/throw, and the recorded exitCode is still honoured. - const realExit = process.exit.bind(process) - process.exit = ((code?: number): never => { - exitCode = code ?? 0 - success = exitCode === 0 - throw new ExitSignal() - }) as typeof process.exit + // Outcomes are tracked for commands that RETURN, THROW, or throw CliExit (the + // cooperative exit used by main.ts's own helpers and the outermost cancel + // handlers — see cli/exit.ts). Deep `process.exit()` calls terminate without + // an event by design: intercepting them globally proved unsafe (clack exits + // from keypress handlers; broad catches swallowed the signal). try { await dispatch(command, subcommand, commandArgs, flags, values) } catch (err) { - if (!(err instanceof ExitSignal)) { + if (err instanceof CliExit) { + exitCode = err.code + success = err.code === 0 + } else { success = false - errorType = err instanceof Error ? err.constructor.name : 'Unknown' - thrown = { err } + errorType = classifyErrorType(err) + // Rethrow to bootstrap's handler ("Fatal error" + exit 1). The finally + // below still runs — including the awaited flush — before propagation. + throw err } - // ExitSignal: success/exitCode were already set by the process.exit override. } finally { - process.exit = realExit // Coerce command/subcommand to a known vocabulary before emit so a free-text // positional (e.g. a `stash wizard ""` description) never leaves. const safe = classifyCommand(command, subcommand) @@ -496,8 +486,7 @@ export async function run() { await shutdownTelemetry() } - if (thrown !== undefined) throw thrown.err - if (exitCode !== undefined) realExit(exitCode) + if (exitCode !== undefined) process.exit(exitCode) } async function dispatch( @@ -571,6 +560,6 @@ async function dispatch( default: console.error(`${messages.cli.unknownCommand}: ${command}\n`) console.log(HELP) - process.exit(1) + throw new CliExit(1) } } diff --git a/packages/cli/src/cli/exit.ts b/packages/cli/src/cli/exit.ts new file mode 100644 index 00000000..36d1a49b --- /dev/null +++ b/packages/cli/src/cli/exit.ts @@ -0,0 +1,24 @@ +/** + * Cooperative exit for first-party command code: `throw new CliExit(code)` + * instead of `process.exit(code)`. + * + * Thrown from a command, it unwinds to `run()` (bin/main.ts), which records the + * outcome for telemetry, flushes, and then performs the real `process.exit` + * with the carried code. This only works from call stacks that reach `run()`'s + * catch without an intervening broad `catch` — which is why ONLY sites verified + * to unwind cleanly use it (main.ts's own dispatch helpers, the telemetry + * command, and the outermost CancelledError handlers of init/plan/impl). + * + * Deep exits stay `process.exit()` deliberately. An earlier version intercepted + * `process.exit` globally with a thrown signal; the review showed that breaks + * code we don't control — @clack/core calls `process.exit(0)` from a keypress + * handler (no enclosing try ⇒ uncaughtException), and several command-level + * broad catches swallowed the signal, continuing past hard stops. Those + * invocations are simply not tracked; see the telemetry module doc. + */ +export class CliExit extends Error { + constructor(readonly code: number) { + super(`exit ${code}`) + this.name = 'CliExit' + } +} diff --git a/packages/cli/src/commands/impl/index.ts b/packages/cli/src/commands/impl/index.ts index 93eb743c..86177838 100644 --- a/packages/cli/src/commands/impl/index.ts +++ b/packages/cli/src/commands/impl/index.ts @@ -1,6 +1,7 @@ import { existsSync, readFileSync } from 'node:fs' import { resolve } from 'node:path' import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { type AgentEnvironment, detectAgents } from '../init/detect-agents.js' import { effectiveStep, @@ -291,7 +292,9 @@ export async function implCommand( } catch (err) { if (err instanceof CancelledError) { p.cancel('Cancelled.') - process.exit(0) + // Cooperative exit: unwinds to run() so the cancel is tracked and the + // telemetry flush completes before the process exits 0 (see cli/exit.ts). + throw new CliExit(0) } throw err } diff --git a/packages/cli/src/commands/init/index.ts b/packages/cli/src/commands/init/index.ts index 1bb19aaa..630ffdc0 100644 --- a/packages/cli/src/commands/init/index.ts +++ b/packages/cli/src/commands/init/index.ts @@ -1,4 +1,5 @@ import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { HANDOFF_CHOICES } from '../impl/steps/how-to-proceed.js' import { planCommand } from '../plan/index.js' import { createBaseProvider } from './providers/base.js' @@ -140,7 +141,9 @@ export async function initCommand( } catch (err) { if (err instanceof CancelledError) { p.cancel('Setup cancelled.') - process.exit(0) + // Cooperative exit: unwinds to run() so the cancel is tracked and the + // telemetry flush completes before the process exits 0 (see cli/exit.ts). + throw new CliExit(0) } throw err } diff --git a/packages/cli/src/commands/plan/index.ts b/packages/cli/src/commands/plan/index.ts index b47ec37e..e780e3a6 100644 --- a/packages/cli/src/commands/plan/index.ts +++ b/packages/cli/src/commands/plan/index.ts @@ -2,6 +2,7 @@ import { existsSync } from 'node:fs' import { resolve } from 'node:path' import { readManifest } from '@cipherstash/migrate' import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { HANDOFF_CHOICES, howToProceedStep, @@ -224,7 +225,9 @@ export async function planCommand( } catch (err) { if (err instanceof CancelledError) { p.cancel('Cancelled.') - process.exit(0) + // Cooperative exit: unwinds to run() so the cancel is tracked and the + // telemetry flush completes before the process exits 0 (see cli/exit.ts). + throw new CliExit(0) } throw err } diff --git a/packages/cli/src/commands/telemetry/index.ts b/packages/cli/src/commands/telemetry/index.ts index a794d0e3..b74d2b80 100644 --- a/packages/cli/src/commands/telemetry/index.ts +++ b/packages/cli/src/commands/telemetry/index.ts @@ -1,4 +1,5 @@ import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { messages } from '../../messages.js' import { setTelemetryDisabled, @@ -59,6 +60,6 @@ export async function telemetryCommand(sub: string | undefined): Promise { break default: p.log.error(`${messages.telemetry.unknownSubcommand}: ${sub}`) - process.exit(1) + throw new CliExit(1) } } diff --git a/packages/cli/src/config/__tests__/tty.test.ts b/packages/cli/src/config/__tests__/tty.test.ts index 8efd07a5..6d7c301b 100644 --- a/packages/cli/src/config/__tests__/tty.test.ts +++ b/packages/cli/src/config/__tests__/tty.test.ts @@ -1,5 +1,51 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' -import { CALLER_ENV_VARS, resolveCaller } from '../tty.js' +import { + CALLER_ENV_VARS, + CI_ENV_VARS, + isCiEnv, + isCiEnvBroad, + resolveCaller, +} from '../tty.js' + +describe('isCiEnv vs isCiEnvBroad (the narrow/broad split)', () => { + beforeEach(() => { + for (const name of CI_ENV_VARS) vi.stubEnv(name, '') + }) + afterEach(() => { + vi.unstubAllEnvs() + }) + + it('isCiEnv is NARROW: a provider marker alone must not flip it', () => { + // The contract prompt gating depends on: a developer with JENKINS_URL or + // GITHUB_ACTIONS exported in their shell still gets interactive prompts. + // Re-broadening isCiEnv (e.g. delegating to isCiEnvBroad) breaks that and + // this test is what catches it. + for (const marker of ['GITHUB_ACTIONS', 'GITLAB_CI', 'JENKINS_URL']) { + vi.stubEnv(marker, 'true') + expect(isCiEnv()).toBe(false) + vi.stubEnv(marker, '') + } + }) + + it('isCiEnv reads only a truthy CI', () => { + expect(isCiEnv()).toBe(false) + vi.stubEnv('CI', 'true') + expect(isCiEnv()).toBe(true) + vi.stubEnv('CI', '1') + expect(isCiEnv()).toBe(true) + vi.stubEnv('CI', 'false') + expect(isCiEnv()).toBe(false) + }) + + it('isCiEnvBroad is BROAD: provider markers and CONTINUOUS_INTEGRATION count', () => { + expect(isCiEnvBroad()).toBe(false) + vi.stubEnv('JENKINS_URL', 'https://ci.example.com') + expect(isCiEnvBroad()).toBe(true) + vi.stubEnv('JENKINS_URL', '') + vi.stubEnv('CONTINUOUS_INTEGRATION', 'true') + expect(isCiEnvBroad()).toBe(true) + }) +}) describe('resolveCaller', () => { // This repo's own agent harness sets some caller markers ambiently, so clear diff --git a/packages/cli/src/config/tty.ts b/packages/cli/src/config/tty.ts index dbb45dbc..ceb3bf1b 100644 --- a/packages/cli/src/config/tty.ts +++ b/packages/cli/src/config/tty.ts @@ -25,9 +25,10 @@ export const CI_PROVIDER_ENV_VARS = [ ] as const /** - * Every env var {@link isCiEnv} consults. Exported so tests can neutralize all - * CI signals hermetically (clearing only `CI` leaves ambient provider vars — a - * real `GITHUB_ACTIONS=true` in this repo's own CI — able to flip the result). + * Every env var {@link isCiEnvBroad} consults ({@link isCiEnv} reads only `CI`). + * Exported so tests can neutralize all CI signals hermetically — clearing only + * `CI` leaves ambient provider vars (a real `GITHUB_ACTIONS=true` in this + * repo's own CI) able to flip the broad check. */ export const CI_ENV_VARS = [ 'CI', diff --git a/packages/cli/src/telemetry/__tests__/classify-command.test.ts b/packages/cli/src/telemetry/__tests__/classify-command.test.ts index 34310ef7..e2748860 100644 --- a/packages/cli/src/telemetry/__tests__/classify-command.test.ts +++ b/packages/cli/src/telemetry/__tests__/classify-command.test.ts @@ -1,5 +1,26 @@ import { describe, expect, it } from 'vitest' -import { classifyCommand } from '../classify-command.js' +import { classifyCommand, classifyErrorType } from '../classify-command.js' + +describe('classifyErrorType (telemetry value allowlist)', () => { + it('passes builtins and first-party error class names through', () => { + expect(classifyErrorType(new TypeError('x'))).toBe('TypeError') + expect(classifyErrorType(new Error('x'))).toBe('Error') + }) + + it('collapses a user-defined error class name to ', () => { + // The CLI runs user code in-process (stash.config.ts via jiti); a class + // named after a table/column must not leave inside errorType. + class PatientsSsnColumnMissingError extends Error {} + expect(classifyErrorType(new PatientsSsnColumnMissingError('x'))).toBe( + '', + ) + }) + + it('collapses non-Error throws to ', () => { + expect(classifyErrorType('a thrown string')).toBe('') + expect(classifyErrorType(undefined)).toBe('') + }) +}) describe('classifyCommand (telemetry value allowlist)', () => { it('keeps a recognised command + subcommand path', () => { diff --git a/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts b/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts index 2da7304e..ebb14be4 100644 --- a/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts +++ b/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts @@ -113,30 +113,62 @@ describe('telemetry lifecycle (emitter + flush)', () => { expect(h.shutdown).not.toHaveBeenCalled() }) - it('shows + persists the first-run notice even when stderr is NOT a TTY, and keeps noticeShownAt across setTelemetryDisabled', async () => { - h.state.value = { anonymousId: 'a', telemetryDisabled: false } // firstRun - const write = vi.spyOn(process.stderr, 'write').mockReturnValue(true) - const originalTty = process.stderr.isTTY + /** Save/restore stderr.isTTY faithfully: when the property wasn't an own + * property (vitest pipes stderr, so isTTY is undefined and absent), restore + * ABSENCE — a defineProperty with `value: undefined` would create a read-only + * own property that breaks later `isTTY = …` assignments in sibling tests. */ + function withStderrTty(isTTY: boolean, run: () => void | Promise) { + const original = Object.getOwnPropertyDescriptor(process.stderr, 'isTTY') Object.defineProperty(process.stderr, 'isTTY', { - value: false, + value: isTTY, configurable: true, }) + const restore = () => { + if (original) Object.defineProperty(process.stderr, 'isTTY', original) + else delete (process.stderr as { isTTY?: boolean }).isTTY + } + const result = run() + if (result instanceof Promise) return result.finally(restore) + restore() + return result + } + + it('shows + persists the first-run notice even when stderr is NOT a TTY, and keeps noticeShownAt across setTelemetryDisabled', async () => { + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // firstRun + const write = vi.spyOn(process.stderr, 'write').mockReturnValue(true) try { - const t = await load() - t.maybeShowFirstRunNotice('npx stash') - expect(write).toHaveBeenCalledTimes(1) - expect(h.writeState).toHaveBeenCalledTimes(1) - expect(h.writeState.mock.calls[0][0].noticeShownAt).toBeDefined() - // Fix: the notice reassigns the state cache, so a later opt-out write does - // not clobber the just-persisted noticeShownAt (the "shows twice" bug). - t.setTelemetryDisabled(false) - const lastWrite = h.writeState.mock.calls.at(-1)?.[0] - expect(lastWrite?.noticeShownAt).toBeDefined() + await withStderrTty(false, async () => { + const t = await load() + t.maybeShowFirstRunNotice('npx stash') + expect(write).toHaveBeenCalledTimes(1) + expect(h.writeState).toHaveBeenCalledTimes(1) + expect(h.writeState.mock.calls[0][0].noticeShownAt).toBeDefined() + // Fix: the notice reassigns the state cache, so a later opt-out write + // does not clobber the just-persisted noticeShownAt ("shows twice" bug). + t.setTelemetryDisabled(false) + const lastWrite = h.writeState.mock.calls.at(-1)?.[0] + expect(lastWrite?.noticeShownAt).toBeDefined() + }) } finally { - Object.defineProperty(process.stderr, 'isTTY', { - value: originalTty, - configurable: true, + write.mockRestore() + } + }) + + it('does NOT print the notice when the state write fails — no once-per-run nag loop', async () => { + // Persist-first: an unwritable HOME (sandboxed runner, read-only container) + // must not print the disclosure on every invocation forever. No persist ⇒ + // no print ⇒ no telemetry ever (firstRun never clears) — dormant, not noisy. + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // firstRun + const write = vi.spyOn(process.stderr, 'write').mockReturnValue(true) + try { + const t = await load() + h.writeState.mockImplementation(() => { + throw new Error('EROFS: read-only file system') }) + // The mocked ../state.js writeState wrapper re-throws through h.writeState. + expect(() => t.maybeShowFirstRunNotice('npx stash')).not.toThrow() + expect(write).not.toHaveBeenCalled() + } finally { write.mockRestore() } }) diff --git a/packages/cli/src/telemetry/classify-command.ts b/packages/cli/src/telemetry/classify-command.ts index 1a2a63cc..5d25d1db 100644 --- a/packages/cli/src/telemetry/classify-command.ts +++ b/packages/cli/src/telemetry/classify-command.ts @@ -35,6 +35,44 @@ const KNOWN_COMMANDS: ReadonlySet = new Set( [...KNOWN_PATHS].map((path) => path.split(' ')[0]), ) +/** + * Error class names allowed to leave as `errorType`. The same closed-vocabulary + * rule as commands: the CLI executes USER code in-process (stash.config.ts and + * the encrypt client, via jiti), so `err.constructor.name` is an open + * vocabulary — a user-defined `PatientsSsnColumnMissingError` would carry a + * column name off the machine. First-party classes + Node/JS builtins pass; + * everything else collapses to ``. + */ +const KNOWN_ERROR_TYPES: ReadonlySet = new Set([ + // JS builtins + 'Error', + 'TypeError', + 'RangeError', + 'SyntaxError', + 'ReferenceError', + 'EvalError', + 'URIError', + 'AggregateError', + // Node-flavoured + 'AbortError', + 'TimeoutError', + 'SystemError', + // First-party CLI errors + 'CliExit', + 'CancelledError', + 'BackfillConfigError', + // Config validation (zod) + 'ZodError', +]) + +/** Coerce an unknown thrown value to a telemetry-safe error class name. */ +export function classifyErrorType(err: unknown): string { + if (!(err instanceof Error)) return '' + return KNOWN_ERROR_TYPES.has(err.constructor.name) + ? err.constructor.name + : '' +} + /** * Return telemetry-safe (command, subcommand). An unrecognised command becomes * `` (and drops its subcommand); a recognised command with an diff --git a/packages/cli/src/telemetry/index.ts b/packages/cli/src/telemetry/index.ts index 819d36eb..eb87f816 100644 --- a/packages/cli/src/telemetry/index.ts +++ b/packages/cli/src/telemetry/index.ts @@ -23,6 +23,13 @@ import { readState, type TelemetryState, writeState } from './state.js' * no posthog-node until a real event is actually sent (both are deferred off * the `--version`/`--help` fast paths), flushing is bounded by a timeout, and * every failure is swallowed. + * - KNOWN MEASUREMENT GAP: commands that terminate via a deep `process.exit()` + * (mid-flow cancels in db install, auth login failures, config-loader exits, + * clack's own keypress exit) emit no event — the process ends before the + * flush could run. Global interception was tried and reverted (it broke + * cancel flows; see `cli/exit.ts`). First-party sites verified to unwind + * cleanly use `throw new CliExit(code)` instead, which IS tracked. Read + * success rates as covering returning/throwing/CliExit paths only. */ /** Default endpoint — our Cloudflare proxy, not PostHog directly, so a future @@ -114,8 +121,11 @@ function envOptOut(name: string): boolean { /** * Resolve whether telemetry is on, in precedence order. Env overrides win over * the persisted flag, so a user can force telemetry off in a context where the - * config file says otherwise (and never the reverse). CI detection is shared - * with the rest of the CLI via {@link isCiEnv}. + * config file says otherwise (and never the reverse). CI detection uses the + * BROAD check ({@link isCiEnvBroad} — provider markers included), deliberately + * separate from the narrow `isCiEnv` that gates interactive prompts: for + * telemetry a false CI positive merely skips collection, whereas for prompts it + * would break an interactive command. */ export function resolveStatus(state: TelemetryState): TelemetryStatus { // Compare against the sentinel, NOT EMBEDDED_KEY: once a real key is injected @@ -286,8 +296,13 @@ export function trackCommand(event: CommandEvent): void { export function maybeShowFirstRunNotice(stashRef: string): void { const { state, status, firstRun } = init() if (!status.enabled || !firstRun) return - process.stderr.write(`${messages.telemetry.notice(stashRef)}\n`) try { + // PERSIST FIRST, print only on success. On a machine where the state write + // fails (read-only HOME, sandboxed runner) the notice would otherwise print + // on every single run forever — worse than pre-notice behaviour. Persist + // failure ⇒ no print AND no telemetry ever (firstRun never clears), which is + // the honest fallback: no disclosure, no collection. + // // Reassign the cache: a later setTelemetryDisabled() in the same process // must not write from a stale `state` and clobber this noticeShownAt. stateCache = writeState({ @@ -295,8 +310,9 @@ export function maybeShowFirstRunNotice(stashRef: string): void { noticeShownAt: new Date().toISOString(), }) } catch { - // A write failure just means we may show the notice again next run. + return } + process.stderr.write(`${messages.telemetry.notice(stashRef)}\n`) } /** Persist the opt-out flag. Surfaces write errors (the command wants to know). */ @@ -311,29 +327,39 @@ export async function shutdownTelemetry(): Promise { // clientPromise is null iff nothing was ever captured (disabled/dormant/first // run), so there is nothing to flush and posthog-node was never loaded. if (clientPromise === null) return - // The timer MUST be cleared once shutdown() wins the race: an uncleared + // The timer MUST be cleared once the flush wins the race: an uncleared // pending setTimeout keeps the Node event loop alive, so the process would // hang for the full timeout after the flush already completed. let timer: ReturnType | undefined + // Capture the promises before the race so the finally can reset the module + // state regardless of which side wins. + const pendingCapture = lastCapture + const pendingClient = clientPromise try { - // Wait for the capture enqueue (client construction + queueing) to finish, - // otherwise shutdown() could race ahead of the event we mean to deliver. - if (lastCapture !== null) { - try { - await lastCapture - } catch { - // A failed capture is already swallowed in trackCommand. - } - } - const client = await clientPromise + // EVERYTHING is inside the race — including awaiting the capture enqueue + // (which contains the dynamic import of posthog-node) and the client + // resolution. If any of it stalls (a hung filesystem mid-import, a + // black-holed endpoint), the timeout still unblocks the process; nothing + // that can hang sits outside the bound. The swallow is attached to the + // flush promise ITSELF (not just the race): if the timeout wins while the + // flush is still pending, a later rejection would otherwise surface as an + // unhandledRejection crash. + const flush = (async () => { + // Wait for the capture enqueue (client construction + queueing) to + // finish, otherwise shutdown() could race ahead of the event we mean + // to deliver. Its rejections are already swallowed in trackCommand. + if (pendingCapture !== null) await pendingCapture + const client = await pendingClient + await client?.shutdown() + })().catch(() => { + // Swallow — a failed flush must not fail (or crash) the process. + }) await Promise.race([ - client.shutdown(), + flush, new Promise((resolve) => { timer = setTimeout(resolve, FLUSH_TIMEOUT_MS) }), ]) - } catch { - // Swallow — a failed flush must not fail the process. } finally { if (timer !== undefined) clearTimeout(timer) clientPromise = null diff --git a/packages/cli/tests/e2e/smoke.e2e.test.ts b/packages/cli/tests/e2e/smoke.e2e.test.ts index f059e3c3..f872b19e 100644 --- a/packages/cli/tests/e2e/smoke.e2e.test.ts +++ b/packages/cli/tests/e2e/smoke.e2e.test.ts @@ -99,6 +99,23 @@ describe('stash CLI — non-interactive smoke', () => { expect(r.output).toContain('bogus-sub') }) + it('telemetry status reports the current state and exits 0', async () => { + const r = render(['telemetry', 'status']) + const { exitCode } = await r.exit + expect(exitCode).toBe(0) + // Dev builds carry the placeholder key, so status reports the dormant build + // (the harness also sets STASH_TELEMETRY_DISABLED, but unconfigured wins). + expect(r.output).toContain('Telemetry is') + }) + + it('telemetry bogus-sub exits 1 (the CliExit cooperative-exit path)', async () => { + const r = render(['telemetry', 'bogus-sub']) + const { exitCode } = await r.exit + expect(exitCode).toBe(1) + expect(r.output).toContain(messages.telemetry.unknownSubcommand) + expect(r.output).toContain('bogus-sub') + }) + // `--migration` without `--supabase` fails flag validation before any I/O // or prompt, so these two cases can observe the install entry path // deterministically without a database. diff --git a/packages/cli/tests/helpers/pty.ts b/packages/cli/tests/helpers/pty.ts index b92cc1c9..8a559011 100644 --- a/packages/cli/tests/helpers/pty.ts +++ b/packages/cli/tests/helpers/pty.ts @@ -100,15 +100,23 @@ export function render(args: string[], opts: RenderOptions = {}): Rendered { NO_COLOR: '1', FORCE_COLOR: '0', } - // isCiEnv() consults provider markers (GITHUB_ACTIONS, GITLAB_CI, …) as well as - // CI. Those leak in from the ambient environment (this repo's own CI), so a - // test that de-CIs with `env: { CI: '' }` would otherwise STILL look like CI - // and skip interactive prompts. Strip every CI signal so the harness default - // below (and per-test overrides) fully control CI detection. + // Strip every CI marker so the harness default below (and per-test overrides) + // fully control CI detection. Prompt gating (isCiEnv) reads only `CI`, but + // telemetry's isCiEnvBroad consults the provider markers (GITHUB_ACTIONS, + // GITLAB_CI, …), which leak in from the ambient environment — this repo's own + // CI — and would otherwise flip telemetry gating in a test that de-CIs with + // `env: { CI: '' }`. for (const name of CI_ENV_VARS) delete env[name] // Match the convention the CLI itself uses (e.g. install.ts checks // `process.env.CI !== 'true'`) so test runs hit the same code paths. env.CI = 'true' + // Hard-disable telemetry regardless of the ambient shell. Without this, a + // developer with STASH_POSTHOG_KEY exported (the documented testing override) + // running the e2e suite would spawn telemetry-ENABLED CLIs against their real + // ~/.cipherstash — sending genuine events from tests and injecting the + // first-run notice into pty output. The CI='true' default is not enough: the + // de-CI'ing tests strip it. Tests exercising telemetry itself can override. + env.STASH_TELEMETRY_DISABLED = '1' Object.assign(env, opts.env ?? {}) // Use the absolute path to the current node binary — node-pty's diff --git a/packages/wizard/src/lib/analytics.ts b/packages/wizard/src/lib/analytics.ts index a8e78853..f58cdf35 100644 --- a/packages/wizard/src/lib/analytics.ts +++ b/packages/wizard/src/lib/analytics.ts @@ -3,43 +3,67 @@ * * Tracks wizard interactions, framework detection, completion rates, and errors. * Analytics are non-blocking — failures are silently ignored. + * + * Honors the same opt-outs as the `stash` CLI's telemetry: `DO_NOT_TRACK` + * (cross-tool standard), `STASH_TELEMETRY_DISABLED`, and CI auto-detection. + * The wizard is usually spawned by `stash wizard`, whose first-run notice + * promises those opt-outs on behalf of the whole CLI surface — this module + * must not make that promise false. */ +import { randomUUID } from 'node:crypto' import { PostHog } from 'posthog-node' import { POSTHOG_API_KEY, POSTHOG_HOST } from './constants.js' import type { Integration, WizardSession } from './types.js' let client: PostHog | undefined +/** + * An opt-out env var is "set" when present and not an explicit off value — + * matching the DO_NOT_TRACK convention (and the CLI's `envOptOut`) that setting + * the variable at all signals intent. + */ +function envOptOut(name: string): boolean { + const raw = process.env[name]?.trim().toLowerCase() + return raw != null && raw !== '' && raw !== '0' && raw !== 'false' +} + +function analyticsDisabled(): boolean { + return ( + envOptOut('DO_NOT_TRACK') || + envOptOut('STASH_TELEMETRY_DISABLED') || + Boolean(process.env.CI?.trim()) + ) +} + function getClient(): PostHog | undefined { if (!POSTHOG_API_KEY) return undefined + if (analyticsDisabled()) return undefined if (!client) { client = new PostHog(POSTHOG_API_KEY, { host: POSTHOG_HOST, flushAt: 1, flushInterval: 0, + // Never resolve IP → geo. + disableGeoip: true, }) } return client } -import { createHash } from 'node:crypto' -import { hostname, userInfo } from 'node:os' +/** + * Per-process random identifier. Deliberately NOT derived from anything + * identifying: the previous sha256(username@hostname) was reversible by + * brute-forcing common username/hostname pairs. Session-stable is enough for + * funnel analysis within one wizard run; cross-run identity is not worth an + * identifying hash. + */ +const SESSION_ID = randomUUID() -/** Generate a stable anonymous identifier for the session. */ function getDistinctId(): string { - try { - const user = userInfo().username - const host = hostname() - return createHash('sha256') - .update(`${user}@${host}`) - .digest('hex') - .slice(0, 16) - } catch { - return 'anonymous' - } + return SESSION_ID } // --- Event tracking --- @@ -103,6 +127,11 @@ export function trackWizardCompleted( }) } +/** + * `error` must be a FIXED LABEL or an error class name — never a raw + * message. Agent/DB error messages routinely embed file paths, table/column + * names, and connection-string fragments; those must not leave the machine. + */ export function trackWizardError(error: string, integration?: Integration) { getClient()?.capture({ distinctId: getDistinctId(), diff --git a/packages/wizard/src/run.ts b/packages/wizard/src/run.ts index 87259688..fab22186 100644 --- a/packages/wizard/src/run.ts +++ b/packages/wizard/src/run.ts @@ -262,7 +262,10 @@ export async function run(options: RunOptions) { }) } } else { - trackWizardError(result.error ?? 'unknown', selectedIntegration) + // Fixed label only — result.error is the agent's raw message and can + // embed schema names or connection details; it stays local (changelog + + // stderr below), never on the wire. + trackWizardError('agent_failed', selectedIntegration) changelog.note(`Agent failed: ${result.error ?? 'unknown error'}`) await changelog.flush() p.log.error(result.error ?? 'Agent failed without a specific error.') @@ -273,7 +276,12 @@ export async function run(options: RunOptions) { } catch (error) { const message = error instanceof Error ? error.message : 'Agent execution failed.' - trackWizardError(message, selectedIntegration) + // Class name only — the message can embed schema names or connection + // details; it stays local (changelog + stderr below), never on the wire. + trackWizardError( + error instanceof Error ? error.constructor.name : 'unknown', + selectedIntegration, + ) changelog.note(`Wizard threw: ${message}`) await changelog.flush() p.log.error(message) From fcc304767805e1de3df802a5b5d2cac76ad733d5 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 15 Jul 2026 18:48:32 +1000 Subject: [PATCH 10/10] docs(cli): disclose the random install identifier in the telemetry docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit flagged that the SKILL.md disclosure lists event fields but never mentions the distinct_id. Add it — with accurate wording: it is a locally generated random UUID stored in ~/.cipherstash/telemetry.json, used only to de-duplicate events in aggregate, and NOT a salted/derived machine id (the review's suggested phrasing described a mechanism we deliberately don't use). Mirrored in the changeset so the release notes carry the same disclosure. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w --- .changeset/cli-anonymous-telemetry.md | 5 ++++- skills/stash-cli/SKILL.md | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/.changeset/cli-anonymous-telemetry.md b/.changeset/cli-anonymous-telemetry.md index 98b1e6c4..4d3befc6 100644 --- a/.changeset/cli-anonymous-telemetry.md +++ b/.changeset/cli-anonymous-telemetry.md @@ -8,7 +8,10 @@ Add anonymous, opt-out usage analytics to the `stash` CLI, plus a Only coarse events are collected — command name, CLI version, OS/arch, Node version, success/failure, duration, and a coarse caller class (e.g. `claude-code`, `cursor`, `interactive`) derived from environment markers so we -can gauge agent- vs human-driven usage. Plaintext, schema, table/column names, +can gauge agent- vs human-driven usage. Events carry a random install +identifier (a locally generated UUID, not derived from any machine or user +attribute) used only to de-duplicate events in aggregate. Plaintext, schema, +table/column names, connection strings, argument values, and any session/trace identifier are never collected — enforced by a property-key allowlist at the emitter boundary plus closed-vocabulary coercion of every argv- or error-derived value (unrecognised diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index ffd82575..32e0df19 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -145,7 +145,10 @@ The resolved URL is returned in memory only. It is never written to disk or into The CLI collects **anonymous, opt-out** usage analytics — coarse events only (command name, CLI version, OS/arch, Node version, success/failure, duration, and a coarse caller class such as `claude-code`/`cursor`/`interactive` derived -from environment markers). It **never** collects plaintext, schema, table/column +from environment markers). Events carry a random install identifier — a UUID +generated locally and stored in `~/.cipherstash/telemetry.json`, not derived +from any machine, user, or hardware attribute — used only to de-duplicate +events in aggregate. It **never** collects plaintext, schema, table/column names, connection strings, argument values, or any session/trace identifier. A one-time notice is printed on first run, and nothing is sent on that first run.