diff --git a/.changeset/cli-anonymous-telemetry.md b/.changeset/cli-anonymous-telemetry.md new file mode 100644 index 00000000..4d3befc6 --- /dev/null +++ b/.changeset/cli-anonymous-telemetry.md @@ -0,0 +1,28 @@ +--- +'stash': minor +--- + +Add anonymous, opt-out usage analytics to the `stash` CLI, plus a +`stash telemetry [status|enable|disable]` command to manage it. + +Only coarse events are collected — command name, CLI version, OS/arch, Node +version, success/failure, duration, and a coarse caller class (e.g. +`claude-code`, `cursor`, `interactive`) derived from environment markers so we +can gauge agent- vs human-driven usage. Events carry a random install +identifier (a locally generated UUID, not derived from any machine or user +attribute) used only to de-duplicate events in aggregate. Plaintext, schema, +table/column names, +connection strings, argument values, and any session/trace identifier are never +collected — enforced by a property-key allowlist at the emitter boundary plus +closed-vocabulary coercion of every argv- or error-derived value (unrecognised +commands, subcommands, and error class names all collapse to ``). A +one-time notice is shown on first run, and nothing is sent on that run. + +Telemetry is off by default in CI and can be disabled with `DO_NOT_TRACK=1` +(the cross-tool standard), `STASH_TELEMETRY_DISABLED=1`, or +`stash telemetry disable` (persisted to `~/.cipherstash/telemetry.json`). + +Events are sent via a first-party proxy and never block or slow the CLI. The +feature ships dormant — no events are sent until a PostHog project key is +embedded at release. Updates the `stash-cli` skill to document the command and +opt-out controls. diff --git a/.changeset/wizard-analytics-privacy.md b/.changeset/wizard-analytics-privacy.md new file mode 100644 index 00000000..c4799b7d --- /dev/null +++ b/.changeset/wizard-analytics-privacy.md @@ -0,0 +1,11 @@ +--- +'@cipherstash/wizard': patch +--- + +Align the wizard's analytics with the `stash` CLI's telemetry privacy contract. +The wizard now honors `DO_NOT_TRACK`, `STASH_TELEMETRY_DISABLED`, and CI +auto-detection; uses a random per-session identifier instead of one derived +from username@hostname; disables IP→geo resolution; and reports error events as +fixed labels / error class names instead of raw messages (which could embed +schema names or connection details). Analytics remain dormant unless a PostHog +key is configured at build time. diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3f0e1987..947c0dc6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -69,3 +69,10 @@ jobs: # changesets/action writes a token .npmrc that shadows OIDC and # every publish fails with E404 (see npm/cli#8976). GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Embeds the CLI's PostHog project key at build time (see + # packages/cli/tsup.config.ts). A repo *variable*, not a secret: the + # key is public and write-only (like a web SDK key). Unset until GA, so + # every release before it is flipped ships telemetry-dormant. This is + # the single go-live switch — set it with: + # gh variable set STASH_POSTHOG_KEY --repo cipherstash/stack --body '' + STASH_POSTHOG_KEY: ${{ vars.STASH_POSTHOG_KEY }} diff --git a/packages/cli/AGENTS.md b/packages/cli/AGENTS.md index c3ba4d68..73795c3b 100644 --- a/packages/cli/AGENTS.md +++ b/packages/cli/AGENTS.md @@ -81,11 +81,20 @@ exercise the same code paths. test actually asserts on the string — premature extraction is worse than copy-paste here. For literals tests don't touch (e.g. command names like `init`, `eql install`), keep them inline. -- **Telemetry.** The CLI source no longer imports `posthog-node` (analytics - moved to `packages/wizard`). The dep is still listed in `package.json` - and should be removed in a follow-up. If you re-introduce telemetry to - the CLI, gate construction on an env var (the wizard's - `getClient()` pattern) so E2E tests can no-op it. +- **Telemetry.** The CLI has anonymous, opt-out usage analytics in + `src/telemetry/` (posthog-node, loaded lazily only when an event is + actually sent). It ships dormant — a real project key is embedded only by + the release build (`STASH_POSTHOG_KEY` repo variable → tsup define) — and + is force-disabled in every pty e2e child via `STASH_TELEMETRY_DISABLED=1` + in `tests/helpers/pty.ts`. Two contracts to preserve when touching it: + (1) event VALUES are closed vocabularies — `classifyCommand` / + `classifyErrorType` in `src/telemetry/classify-command.ts` must wrap + anything argv- or error-derived before emit; (2) never intercept + `process.exit` with a thrown signal — @clack/core exits from keypress + handlers and several commands have broad catches, so interception breaks + cancel flows (tried and reverted; see `src/cli/exit.ts`). Commands that + terminate via deep `process.exit()` are simply not tracked; cooperative + exits use `throw new CliExit(code)` from verified-unwindable sites only. ## Plan and rationale diff --git a/packages/cli/src/bin/main.ts b/packages/cli/src/bin/main.ts index c71794a8..ee7e2ab7 100644 --- a/packages/cli/src/bin/main.ts +++ b/packages/cli/src/bin/main.ts @@ -19,6 +19,7 @@ import { readFileSync } from 'node:fs' import { dirname, join } from 'node:path' import { fileURLToPath } from 'node:url' import * as p from '@clack/prompts' +import { CliExit } from '../cli/exit.js' import { renderCommandHelp } from '../cli/help.js' // Commands that depend on @cipherstash/stack are lazy-loaded in the switch below. import { @@ -31,11 +32,22 @@ import { manifestCommand, planCommand, statusCommand, + telemetryCommand, testConnectionCommand, upgradeCommand, wizardCommand, } from '../commands/index.js' import { messages } from '../messages.js' +import { + classifyCommand, + classifyErrorType, +} from '../telemetry/classify-command.js' +import { + initTelemetry, + maybeShowFirstRunNotice, + shutdownTelemetry, + trackCommand, +} from '../telemetry/index.js' function isModuleNotFound(err: unknown): boolean { return ( @@ -72,7 +84,7 @@ async function requireStack(importFn: () => Promise): Promise { Install it with: ${prodInstallCommand(PM, '@cipherstash/stack')} Or run: ${STASH} init`, ) - process.exit(1) as never + throw new CliExit(1) } throw err } @@ -92,6 +104,7 @@ Commands: wizard AI-guided encryption setup (reads your codebase) doctor Diagnose install problems (native binaries, runtime) manifest Print the structured, versioned command surface (--json for docs/agents) + telemetry Manage anonymous usage analytics (status, enable, disable) eql install Scaffold stash.config.ts (if missing) and install EQL extensions eql upgrade Upgrade EQL extensions to the latest version @@ -228,7 +241,7 @@ async function runEqlCommand( p.log.error(`${messages.eql.unknownSubcommand}: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -292,7 +305,7 @@ async function runDbCommand( p.log.error(`${messages.db.unknownSubcommand}: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -367,7 +380,7 @@ async function runEncryptCommand( p.log.error(`Unknown encrypt subcommand: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -375,7 +388,7 @@ function requireValue(values: Record, key: string): string { const v = values[key] if (!v) { p.log.error(`Missing required --${key} value.`) - process.exit(1) + throw new CliExit(1) } return v } @@ -400,7 +413,7 @@ async function runSchemaCommand( p.log.error(`Unknown schema subcommand: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -431,6 +444,58 @@ export async function run() { return } + // Anonymous, opt-out usage analytics. The notice shows once (to stderr) and + // the run that shows it sends nothing; both are no-ops when telemetry is off. + initTelemetry(pkg.version) + maybeShowFirstRunNotice(STASH) + + const startedAt = Date.now() + let success = true + let errorType: string | undefined + let exitCode: number | undefined + + // Outcomes are tracked for commands that RETURN, THROW, or throw CliExit (the + // cooperative exit used by main.ts's own helpers and the outermost cancel + // handlers — see cli/exit.ts). Deep `process.exit()` calls terminate without + // an event by design: intercepting them globally proved unsafe (clack exits + // from keypress handlers; broad catches swallowed the signal). + try { + await dispatch(command, subcommand, commandArgs, flags, values) + } catch (err) { + if (err instanceof CliExit) { + exitCode = err.code + success = err.code === 0 + } else { + success = false + errorType = classifyErrorType(err) + // Rethrow to bootstrap's handler ("Fatal error" + exit 1). The finally + // below still runs — including the awaited flush — before propagation. + throw err + } + } finally { + // Coerce command/subcommand to a known vocabulary before emit so a free-text + // positional (e.g. a `stash wizard ""` description) never leaves. + const safe = classifyCommand(command, subcommand) + trackCommand({ + command: safe.command, + subcommand: safe.subcommand, + success, + durationMs: Date.now() - startedAt, + errorType, + }) + await shutdownTelemetry() + } + + if (exitCode !== undefined) process.exit(exitCode) +} + +async function dispatch( + command: string, + subcommand: string | undefined, + commandArgs: string[], + flags: Record, + values: Record, +) { switch (command) { case 'init': await initCommand(flags, values) @@ -473,6 +538,9 @@ export async function run() { // the native binary is missing. manifestCommand({ json: flags.json, version: pkg.version }) break + case 'telemetry': + await telemetryCommand(subcommand) + break case 'wizard': { // Forward everything after `stash wizard` verbatim. The wizard package // owns its own flag parsing; we don't try to interpret its surface @@ -492,6 +560,6 @@ export async function run() { default: console.error(`${messages.cli.unknownCommand}: ${command}\n`) console.log(HELP) - process.exit(1) + throw new CliExit(1) } } diff --git a/packages/cli/src/cli/exit.ts b/packages/cli/src/cli/exit.ts new file mode 100644 index 00000000..36d1a49b --- /dev/null +++ b/packages/cli/src/cli/exit.ts @@ -0,0 +1,24 @@ +/** + * Cooperative exit for first-party command code: `throw new CliExit(code)` + * instead of `process.exit(code)`. + * + * Thrown from a command, it unwinds to `run()` (bin/main.ts), which records the + * outcome for telemetry, flushes, and then performs the real `process.exit` + * with the carried code. This only works from call stacks that reach `run()`'s + * catch without an intervening broad `catch` — which is why ONLY sites verified + * to unwind cleanly use it (main.ts's own dispatch helpers, the telemetry + * command, and the outermost CancelledError handlers of init/plan/impl). + * + * Deep exits stay `process.exit()` deliberately. An earlier version intercepted + * `process.exit` globally with a thrown signal; the review showed that breaks + * code we don't control — @clack/core calls `process.exit(0)` from a keypress + * handler (no enclosing try ⇒ uncaughtException), and several command-level + * broad catches swallowed the signal, continuing past hard stops. Those + * invocations are simply not tracked; see the telemetry module doc. + */ +export class CliExit extends Error { + constructor(readonly code: number) { + super(`exit ${code}`) + this.name = 'CliExit' + } +} diff --git a/packages/cli/src/cli/registry.ts b/packages/cli/src/cli/registry.ts index c388d5ed..66f527f5 100644 --- a/packages/cli/src/cli/registry.ts +++ b/packages/cli/src/cli/registry.ts @@ -231,6 +231,20 @@ export const registry: CommandGroup[] = [ }, ], }, + { + name: 'telemetry', + summary: 'Manage anonymous usage analytics', + long: [ + 'Manage the anonymous, opt-out usage analytics the CLI collects to', + 'improve the tool. `status` (the default) reports whether telemetry is', + 'on and which setting governs it; `enable` / `disable` write your saved', + 'preference. Telemetry is also disabled by the DO_NOT_TRACK or', + 'STASH_TELEMETRY_DISABLED environment variables and automatically in CI.', + 'No plaintext, schema, table/column names, or connection details are', + 'ever collected. See https://cipherstash.com/docs/reference/cli.', + ].join('\n'), + examples: ['telemetry status', 'telemetry disable', 'telemetry enable'], + }, ], }, { diff --git a/packages/cli/src/commands/impl/index.ts b/packages/cli/src/commands/impl/index.ts index 93eb743c..86177838 100644 --- a/packages/cli/src/commands/impl/index.ts +++ b/packages/cli/src/commands/impl/index.ts @@ -1,6 +1,7 @@ import { existsSync, readFileSync } from 'node:fs' import { resolve } from 'node:path' import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { type AgentEnvironment, detectAgents } from '../init/detect-agents.js' import { effectiveStep, @@ -291,7 +292,9 @@ export async function implCommand( } catch (err) { if (err instanceof CancelledError) { p.cancel('Cancelled.') - process.exit(0) + // Cooperative exit: unwinds to run() so the cancel is tracked and the + // telemetry flush completes before the process exits 0 (see cli/exit.ts). + throw new CliExit(0) } throw err } diff --git a/packages/cli/src/commands/index.ts b/packages/cli/src/commands/index.ts index 9069fe67..914dbfe3 100644 --- a/packages/cli/src/commands/index.ts +++ b/packages/cli/src/commands/index.ts @@ -9,4 +9,5 @@ export { initCommand } from './init/index.js' export { manifestCommand } from './manifest/index.js' export { planCommand } from './plan/index.js' export { statusCommand } from './status/index.js' +export { telemetryCommand } from './telemetry/index.js' export { wizardCommand } from './wizard/index.js' diff --git a/packages/cli/src/commands/init/index.ts b/packages/cli/src/commands/init/index.ts index 1bb19aaa..630ffdc0 100644 --- a/packages/cli/src/commands/init/index.ts +++ b/packages/cli/src/commands/init/index.ts @@ -1,4 +1,5 @@ import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { HANDOFF_CHOICES } from '../impl/steps/how-to-proceed.js' import { planCommand } from '../plan/index.js' import { createBaseProvider } from './providers/base.js' @@ -140,7 +141,9 @@ export async function initCommand( } catch (err) { if (err instanceof CancelledError) { p.cancel('Setup cancelled.') - process.exit(0) + // Cooperative exit: unwinds to run() so the cancel is tracked and the + // telemetry flush completes before the process exits 0 (see cli/exit.ts). + throw new CliExit(0) } throw err } diff --git a/packages/cli/src/commands/plan/index.ts b/packages/cli/src/commands/plan/index.ts index b47ec37e..e780e3a6 100644 --- a/packages/cli/src/commands/plan/index.ts +++ b/packages/cli/src/commands/plan/index.ts @@ -2,6 +2,7 @@ import { existsSync } from 'node:fs' import { resolve } from 'node:path' import { readManifest } from '@cipherstash/migrate' import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { HANDOFF_CHOICES, howToProceedStep, @@ -224,7 +225,9 @@ export async function planCommand( } catch (err) { if (err instanceof CancelledError) { p.cancel('Cancelled.') - process.exit(0) + // Cooperative exit: unwinds to run() so the cancel is tracked and the + // telemetry flush completes before the process exits 0 (see cli/exit.ts). + throw new CliExit(0) } throw err } diff --git a/packages/cli/src/commands/telemetry/index.ts b/packages/cli/src/commands/telemetry/index.ts new file mode 100644 index 00000000..b74d2b80 --- /dev/null +++ b/packages/cli/src/commands/telemetry/index.ts @@ -0,0 +1,65 @@ +import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' +import { messages } from '../../messages.js' +import { + setTelemetryDisabled, + type TelemetryDisabledReason, + telemetryStatus, +} from '../../telemetry/index.js' + +/** Human-readable explanation for why telemetry is currently off. */ +function reasonText(reason: TelemetryDisabledReason): string { + switch (reason) { + case 'do-not-track': + return 'disabled by the DO_NOT_TRACK environment variable' + case 'stash-disabled': + return 'disabled by the STASH_TELEMETRY_DISABLED environment variable' + case 'ci': + return 'disabled automatically in CI' + case 'config': + return 'disabled (run `stash telemetry enable` to turn it back on)' + case 'unconfigured': + return 'not active in this build' + } +} + +function printStatus(): void { + const status = telemetryStatus() + if (status.enabled) { + p.log.info('Telemetry is enabled. Anonymous usage analytics are collected.') + return + } + p.log.info(`Telemetry is ${reasonText(status.reason)}.`) + // An env override wins over the persisted flag, so flag `enable` won't help. + if (status.reason === 'do-not-track' || status.reason === 'stash-disabled') { + p.log.info( + 'An environment variable is overriding your saved preference; unset it to re-enable.', + ) + } +} + +/** + * `stash telemetry [status|enable|disable]` — manage anonymous CLI analytics. + * `enable`/`disable` write the persisted opt-out flag; `status` (the default) + * reports the current state and which gate governs it. + */ +export async function telemetryCommand(sub: string | undefined): Promise { + switch (sub) { + case undefined: + case 'status': + printStatus() + break + case 'enable': + setTelemetryDisabled(false) + p.log.success(messages.telemetry.enabled) + printStatus() + break + case 'disable': + setTelemetryDisabled(true) + p.log.success(messages.telemetry.disabled) + break + default: + p.log.error(`${messages.telemetry.unknownSubcommand}: ${sub}`) + throw new CliExit(1) + } +} diff --git a/packages/cli/src/config/__tests__/tty.test.ts b/packages/cli/src/config/__tests__/tty.test.ts new file mode 100644 index 00000000..6d7c301b --- /dev/null +++ b/packages/cli/src/config/__tests__/tty.test.ts @@ -0,0 +1,130 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + CALLER_ENV_VARS, + CI_ENV_VARS, + isCiEnv, + isCiEnvBroad, + resolveCaller, +} from '../tty.js' + +describe('isCiEnv vs isCiEnvBroad (the narrow/broad split)', () => { + beforeEach(() => { + for (const name of CI_ENV_VARS) vi.stubEnv(name, '') + }) + afterEach(() => { + vi.unstubAllEnvs() + }) + + it('isCiEnv is NARROW: a provider marker alone must not flip it', () => { + // The contract prompt gating depends on: a developer with JENKINS_URL or + // GITHUB_ACTIONS exported in their shell still gets interactive prompts. + // Re-broadening isCiEnv (e.g. delegating to isCiEnvBroad) breaks that and + // this test is what catches it. + for (const marker of ['GITHUB_ACTIONS', 'GITLAB_CI', 'JENKINS_URL']) { + vi.stubEnv(marker, 'true') + expect(isCiEnv()).toBe(false) + vi.stubEnv(marker, '') + } + }) + + it('isCiEnv reads only a truthy CI', () => { + expect(isCiEnv()).toBe(false) + vi.stubEnv('CI', 'true') + expect(isCiEnv()).toBe(true) + vi.stubEnv('CI', '1') + expect(isCiEnv()).toBe(true) + vi.stubEnv('CI', 'false') + expect(isCiEnv()).toBe(false) + }) + + it('isCiEnvBroad is BROAD: provider markers and CONTINUOUS_INTEGRATION count', () => { + expect(isCiEnvBroad()).toBe(false) + vi.stubEnv('JENKINS_URL', 'https://ci.example.com') + expect(isCiEnvBroad()).toBe(true) + vi.stubEnv('JENKINS_URL', '') + vi.stubEnv('CONTINUOUS_INTEGRATION', 'true') + expect(isCiEnvBroad()).toBe(true) + }) +}) + +describe('resolveCaller', () => { + // This repo's own agent harness sets some caller markers ambiently, so clear + // every consulted var before each case; otherwise the fallback tests would + // pick up e.g. CLAUDECODE from the process running the suite. + beforeEach(() => { + for (const name of CALLER_ENV_VARS) vi.stubEnv(name, '') + }) + afterEach(() => { + vi.unstubAllEnvs() + }) + + /** Force process.stdin.isTTY for the fallback cases; restored after each test. */ + function withStdinTty(isTTY: boolean, run: () => void): void { + const original = Object.getOwnPropertyDescriptor(process.stdin, 'isTTY') + Object.defineProperty(process.stdin, 'isTTY', { + value: isTTY, + configurable: true, + }) + try { + run() + } finally { + if (original) Object.defineProperty(process.stdin, 'isTTY', original) + else delete (process.stdin as { isTTY?: boolean }).isTTY + } + } + + it('detects Claude Code via CLAUDECODE', () => { + vi.stubEnv('CLAUDECODE', '1') + expect(resolveCaller()).toBe('claude-code') + }) + + it('detects Claude Code via the CLAUDE_CODE_ENTRYPOINT backup marker', () => { + vi.stubEnv('CLAUDE_CODE_ENTRYPOINT', 'cli') + expect(resolveCaller()).toBe('claude-code') + }) + + it('detects Cursor via CURSOR_TRACE_ID', () => { + vi.stubEnv('CURSOR_TRACE_ID', 'abc-123') + expect(resolveCaller()).toBe('cursor') + }) + + it('detects Codex via CODEX_SANDBOX', () => { + vi.stubEnv('CODEX_SANDBOX', 'seatbelt') + expect(resolveCaller()).toBe('codex') + }) + + it('confirmed agents outrank the editor marker (Cursor also sets TERM_PROGRAM=vscode)', () => { + vi.stubEnv('CURSOR_TRACE_ID', 'abc-123') + vi.stubEnv('TERM_PROGRAM', 'vscode') + expect(resolveCaller()).toBe('cursor') + }) + + it('the first agent marker wins when several are present', () => { + vi.stubEnv('CLAUDECODE', '1') + vi.stubEnv('CURSOR_TRACE_ID', 'abc-123') + expect(resolveCaller()).toBe('claude-code') + }) + + it('classifies a VS Code-family editor terminal as editor, not an agent', () => { + vi.stubEnv('TERM_PROGRAM', 'vscode') + expect(resolveCaller()).toBe('editor') + }) + + it('is case-insensitive on TERM_PROGRAM', () => { + vi.stubEnv('TERM_PROGRAM', 'vsCode') + expect(resolveCaller()).toBe('editor') + }) + + it('ignores whitespace-only marker values', () => { + vi.stubEnv('CLAUDECODE', ' ') + withStdinTty(true, () => expect(resolveCaller()).toBe('interactive')) + }) + + it('falls back to interactive when stdin is a TTY and no marker is set', () => { + withStdinTty(true, () => expect(resolveCaller()).toBe('interactive')) + }) + + it('falls back to non-interactive when stdin is not a TTY', () => { + withStdinTty(false, () => expect(resolveCaller()).toBe('non-interactive')) + }) +}) diff --git a/packages/cli/src/config/tty.ts b/packages/cli/src/config/tty.ts index 09fd476a..ceb3bf1b 100644 --- a/packages/cli/src/config/tty.ts +++ b/packages/cli/src/config/tty.ts @@ -4,16 +4,130 @@ */ /** - * True when `CI` is set to a common truthy spelling (`true`/`1`, - * case-insensitive) — not every CI provider sets `CI=true` exactly. Shared by - * the region resolver (`commands/auth/region.ts`) and the DATABASE_URL resolver - * (`config/database-url.ts`) so their non-interactive gating stays identical. + * Provider-specific markers for CI systems that don't reliably set `CI=true` + * (Jenkins, Azure Pipelines/`TF_BUILD`, TeamCity, …). GitHub Actions and GitLab + * both DO set `CI=true`, so their vars here are belt-and-suspenders. Detected by + * presence of any non-empty value. + */ +export const CI_PROVIDER_ENV_VARS = [ + 'GITHUB_ACTIONS', + 'GITLAB_CI', + 'CIRCLECI', + 'TRAVIS', + 'BUILDKITE', + 'JENKINS_URL', + 'TEAMCITY_VERSION', + 'TF_BUILD', // Azure Pipelines + 'APPVEYOR', + 'DRONE', + 'BITBUCKET_BUILD_NUMBER', + 'CODEBUILD_BUILD_ID', +] as const + +/** + * Every env var {@link isCiEnvBroad} consults ({@link isCiEnv} reads only `CI`). + * Exported so tests can neutralize all CI signals hermetically — clearing only + * `CI` leaves ambient provider vars (a real `GITHUB_ACTIONS=true` in this + * repo's own CI) able to flip the broad check. + */ +export const CI_ENV_VARS = [ + 'CI', + 'CONTINUOUS_INTEGRATION', + ...CI_PROVIDER_ENV_VARS, +] as const + +/** + * True when `CI` is set to a truthy spelling (`1`/`true`) — the near-universal + * marker, set by GitHub Actions, GitLab, CircleCI, and most others. This is the + * NARROW check that governs interactive prompting: the region resolver + * (`commands/auth/region.ts`), the DATABASE_URL resolver (`config/database-url.ts`), + * and {@link isInteractive}. It is intentionally conservative about suppressing a + * prompt — a developer whose shell happens to export a provider marker like + * `JENKINS_URL` should still get prompted. Telemetry uses the broader + * {@link isCiEnvBroad} instead, where erring toward "don't collect" is safer. */ export function isCiEnv(): boolean { const ciVar = process.env.CI?.trim() return ciVar !== undefined && /^(1|true)$/i.test(ciVar) } +/** + * True in CI, detected broadly: {@link isCiEnv} plus `CONTINUOUS_INTEGRATION` and + * the {@link CI_PROVIDER_ENV_VARS} markers for systems that don't set `CI` + * (Jenkins, TeamCity, Azure Pipelines, …). Used ONLY for telemetry auto-disable, + * where a false positive merely skips collection — unlike prompt gating, where a + * false positive breaks an interactive command. Keep it separate from + * {@link isCiEnv} so broadening CI-for-telemetry never changes prompt behavior. + */ +export function isCiEnvBroad(): boolean { + if (isCiEnv()) return true + if (process.env.CONTINUOUS_INTEGRATION?.trim()) return true + return CI_PROVIDER_ENV_VARS.some((name) => Boolean(process.env[name]?.trim())) +} + +/** + * Coarse classification of who invoked the CLI, for anonymous telemetry. A + * heuristic, not a guarantee: it recognises the major coding-agent harnesses by + * an env var each one sets, and otherwise falls back to whether stdin is a TTY. + * Only the fixed label ever leaves the machine (see the telemetry allowlist) — + * never the underlying env value, some of which carry session/trace IDs. Read + * the numbers as a lower bound on agent usage: a harness without a known marker + * lands in `interactive`/`non-interactive`. + */ +export type CallerKind = + | 'claude-code' + | 'cursor' + | 'codex' + | 'editor' + | 'interactive' + | 'non-interactive' + +/** + * Known coding-agent harnesses, each recognised by an env var it sets in the + * shell it drives. Ordered most-specific first; the first match wins. Extend + * this as new harnesses appear — every `kind` must stay a fixed, non-identifying + * string, because that label (not the env value) is what telemetry emits. + */ +const AGENT_MARKERS: ReadonlyArray<{ + readonly kind: CallerKind + readonly envVars: readonly string[] +}> = [ + // Claude Code sets CLAUDECODE=1; CLAUDE_CODE_ENTRYPOINT is a backup signal. + { kind: 'claude-code', envVars: ['CLAUDECODE', 'CLAUDE_CODE_ENTRYPOINT'] }, + // Cursor's agent/terminal exports a per-session trace id. + { kind: 'cursor', envVars: ['CURSOR_TRACE_ID'] }, + // OpenAI Codex CLI runs commands in a sandbox that exports CODEX_SANDBOX. + { kind: 'codex', envVars: ['CODEX_SANDBOX'] }, +] + +/** + * Every env var {@link resolveCaller} consults. Exported so tests can neutralize + * all caller signals hermetically — this repo's own agent harness sets some of + * these ambiently, which would otherwise flip the result. + */ +export const CALLER_ENV_VARS = [ + ...AGENT_MARKERS.flatMap((m) => m.envVars), + 'TERM_PROGRAM', +] as const + +/** + * Classify the caller. Confirmed agent harnesses win first; then a VS Code-family + * editor terminal (`TERM_PROGRAM=vscode`, which Cursor also sets, so it is checked + * after Cursor and kept distinct because it may be a human in an editor); then the + * plain TTY heuristic. + */ +export function resolveCaller(): CallerKind { + for (const marker of AGENT_MARKERS) { + if (marker.envVars.some((name) => Boolean(process.env[name]?.trim()))) { + return marker.kind + } + } + if (process.env.TERM_PROGRAM?.trim().toLowerCase() === 'vscode') { + return 'editor' + } + return process.stdin.isTTY ? 'interactive' : 'non-interactive' +} + /** * True when it's safe to show an interactive clack prompt: stdin is a TTY and * we're not in CI. Every prompt gate (the DATABASE_URL resolver, the config diff --git a/packages/cli/src/messages.ts b/packages/cli/src/messages.ts index 0e7e418a..9e671825 100644 --- a/packages/cli/src/messages.ts +++ b/packages/cli/src/messages.ts @@ -109,4 +109,22 @@ export const messages = { ) => `\`${pkg}\` is not installed in this project.\n\nInstall the CipherStash packages, then re-run:\n ${installCommands}\n\nOr run \`${stash} init\` to set everything up.`, }, + telemetry: { + /** + * The one-time first-run notice. Printed to stderr so it never pollutes + * piped or `--json` stdout. Nothing is sent on the run that shows it. + * `stashRef` is the runner-aware invocation (e.g. `npx stash`) so the + * opt-out command is actionable even before `stash` is on PATH. + */ + notice: (stashRef: string) => + [ + 'CipherStash collects anonymous CLI usage analytics to improve the tool.', + 'No plaintext, schema, table/column names, or connection details are ever collected.', + `We honor the DO_NOT_TRACK standard. Opt out any time: set DO_NOT_TRACK=1, or run \`${stashRef} telemetry disable\`.`, + 'Learn more: https://cipherstash.com/docs/reference/cli', + ].join('\n'), + enabled: 'Telemetry enabled.', + disabled: 'Telemetry disabled.', + unknownSubcommand: 'Unknown telemetry command', + }, } as const diff --git a/packages/cli/src/telemetry/__tests__/classify-command.test.ts b/packages/cli/src/telemetry/__tests__/classify-command.test.ts new file mode 100644 index 00000000..e2748860 --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/classify-command.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it } from 'vitest' +import { classifyCommand, classifyErrorType } from '../classify-command.js' + +describe('classifyErrorType (telemetry value allowlist)', () => { + it('passes builtins and first-party error class names through', () => { + expect(classifyErrorType(new TypeError('x'))).toBe('TypeError') + expect(classifyErrorType(new Error('x'))).toBe('Error') + }) + + it('collapses a user-defined error class name to ', () => { + // The CLI runs user code in-process (stash.config.ts via jiti); a class + // named after a table/column must not leave inside errorType. + class PatientsSsnColumnMissingError extends Error {} + expect(classifyErrorType(new PatientsSsnColumnMissingError('x'))).toBe( + '', + ) + }) + + it('collapses non-Error throws to ', () => { + expect(classifyErrorType('a thrown string')).toBe('') + expect(classifyErrorType(undefined)).toBe('') + }) +}) + +describe('classifyCommand (telemetry value allowlist)', () => { + it('keeps a recognised command + subcommand path', () => { + expect(classifyCommand('eql', 'install')).toEqual({ + command: 'eql', + subcommand: 'install', + }) + expect(classifyCommand('auth', 'login')).toEqual({ + command: 'auth', + subcommand: 'login', + }) + }) + + it('keeps a recognised command with no subcommand', () => { + expect(classifyCommand('init', undefined)).toEqual({ + command: 'init', + subcommand: undefined, + }) + }) + + it('keeps the telemetry sub-verbs (not in the registry, but known-safe)', () => { + for (const verb of ['status', 'enable', 'disable']) { + expect(classifyCommand('telemetry', verb)).toEqual({ + command: 'telemetry', + subcommand: verb, + }) + } + }) + + it('coerces a free-text positional to — the wizard-prompt leak', () => { + // The core fix: `stash wizard "add encryption to patients.ssn"` must never + // send the prompt (which carries table/column names) as the subcommand value. + expect( + classifyCommand('wizard', 'add searchable encryption to patients.ssn'), + ).toEqual({ command: 'wizard', subcommand: '' }) + }) + + it('keeps the command but drops an unrecognised subcommand to ', () => { + expect(classifyCommand('eql', 'rm -rf /')).toEqual({ + command: 'eql', + subcommand: '', + }) + }) + + it('coerces an entirely unknown command to and drops its subcommand', () => { + expect(classifyCommand('definitely-not-a-command', 'secret-value')).toEqual( + { + command: '', + subcommand: undefined, + }, + ) + }) +}) diff --git a/packages/cli/src/telemetry/__tests__/state.test.ts b/packages/cli/src/telemetry/__tests__/state.test.ts new file mode 100644 index 00000000..91b52be4 --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/state.test.ts @@ -0,0 +1,66 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { afterEach, beforeEach, describe, expect, it } from 'vitest' +import { readState, writeState } from '../state.js' + +let home: string +let originalHome: string | undefined + +beforeEach(() => { + originalHome = process.env.HOME + home = fs.mkdtempSync(path.join(os.tmpdir(), 'stash-telemetry-')) + process.env.HOME = home +}) + +afterEach(() => { + if (originalHome === undefined) { + delete process.env.HOME + } else { + process.env.HOME = originalHome + } + fs.rmSync(home, { recursive: true, force: true }) +}) + +const file = () => path.join(home, '.cipherstash', 'telemetry.json') + +describe('telemetry state', () => { + it('returns a fresh default (with an id) when no file exists', () => { + const state = readState() + expect(state.telemetryDisabled).toBe(false) + expect(state.noticeShownAt).toBeUndefined() + expect(state.anonymousId).toMatch(/[0-9a-f-]{36}/) + }) + + it('persists 0600 and round-trips', () => { + const written = writeState({ + anonymousId: 'anon-x', + telemetryDisabled: true, + noticeShownAt: '2026-07-14T00:00:00.000Z', + }) + expect(readState()).toEqual(written) + // Private to the user — never group/world readable. + expect(fs.statSync(file()).mode & 0o077).toBe(0) + }) + + it('recovers from a corrupt file with a fresh id, never throwing', () => { + fs.mkdirSync(path.dirname(file()), { recursive: true }) + fs.writeFileSync(file(), '{ this is not json') + const state = readState() + expect(state.anonymousId).toMatch(/[0-9a-f-]{36}/) + expect(state.telemetryDisabled).toBe(false) + }) + + it('coerces a partial/garbage object to valid defaults', () => { + fs.mkdirSync(path.dirname(file()), { recursive: true }) + fs.writeFileSync( + file(), + JSON.stringify({ telemetryDisabled: 'yes', noticeShownAt: 42 }), + ) + const state = readState() + // A non-boolean flag is not treated as opted out; a non-string ts is dropped. + expect(state.telemetryDisabled).toBe(false) + expect(state.noticeShownAt).toBeUndefined() + expect(state.anonymousId).toMatch(/[0-9a-f-]{36}/) + }) +}) diff --git a/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts b/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts new file mode 100644 index 00000000..ebb14be4 --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts @@ -0,0 +1,185 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { CI_ENV_VARS } from '../../config/tty.js' +import type { TelemetryState } from '../state.js' + +// Controllable doubles for posthog-node and the on-disk state, wired via +// vi.hoisted so the (hoisted) vi.mock factories can close over them. +const h = vi.hoisted(() => ({ + capture: vi.fn(), + shutdown: vi.fn(async () => {}), + PostHog: vi.fn(), + writeState: vi.fn(), + state: { value: undefined as TelemetryState | undefined }, +})) + +vi.mock('posthog-node', () => ({ PostHog: h.PostHog })) +vi.mock('../state.js', () => ({ + readState: () => h.state.value, + writeState: (s: TelemetryState) => { + h.writeState(s) + h.state.value = s + return s + }, +})) + +/** Re-import the module fresh so its lazily-initialised caches reset per test. */ +async function load() { + vi.resetModules() + return import('../index.js') +} + +const ENABLED: TelemetryState = { + anonymousId: 'anon-x', + telemetryDisabled: false, + noticeShownAt: '2026-01-01T00:00:00.000Z', +} + +describe('telemetry lifecycle (emitter + flush)', () => { + beforeEach(() => { + h.capture.mockReset() + h.shutdown.mockReset().mockResolvedValue(undefined) + h.PostHog.mockReset().mockImplementation(() => ({ + capture: h.capture, + shutdown: h.shutdown, + })) + h.writeState.mockReset() + h.state.value = { ...ENABLED } + // Enabled by default; neutralize every gate so status is { enabled: true }. + vi.stubEnv('STASH_POSTHOG_KEY', 'phc_test_key') + for (const name of [ + 'DO_NOT_TRACK', + 'STASH_TELEMETRY_DISABLED', + ...CI_ENV_VARS, + ]) { + vi.stubEnv(name, '') + } + }) + afterEach(() => { + vi.unstubAllEnvs() + vi.useRealTimers() + }) + + it('trackCommand builds no client and sends nothing when disabled', async () => { + vi.stubEnv('STASH_POSTHOG_KEY', '') // dormant + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await t.shutdownTelemetry() + expect(h.PostHog).not.toHaveBeenCalled() + expect(h.capture).not.toHaveBeenCalled() + }) + + it('trackCommand is a no-op on the first run (freebie)', async () => { + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // no noticeShownAt + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await t.shutdownTelemetry() + expect(h.PostHog).not.toHaveBeenCalled() + }) + + it('captures one anonymous event and flushes on shutdown', async () => { + const t = await load() + t.initTelemetry('9.9.9') + t.trackCommand({ + command: 'eql', + subcommand: 'install', + success: true, + durationMs: 5, + }) + await t.shutdownTelemetry() + expect(h.PostHog).toHaveBeenCalledTimes(1) + expect(h.capture).toHaveBeenCalledTimes(1) + const arg = h.capture.mock.calls[0][0] + expect(arg.distinctId).toBe('anon-x') + expect(arg.event).toBe('command_invoked') + expect(arg.properties.command).toBe('eql') + expect(arg.properties.cliVersion).toBe('9.9.9') + expect(arg.properties.$process_person_profile).toBe(false) + expect(h.shutdown).toHaveBeenCalledTimes(1) + }) + + it('swallows a throwing capture and still shuts down cleanly', async () => { + h.capture.mockImplementation(() => { + throw new Error('boom') + }) + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + }) + + it('shutdownTelemetry is a safe no-op when nothing was captured, even twice', async () => { + const t = await load() + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + expect(h.shutdown).not.toHaveBeenCalled() + }) + + /** Save/restore stderr.isTTY faithfully: when the property wasn't an own + * property (vitest pipes stderr, so isTTY is undefined and absent), restore + * ABSENCE — a defineProperty with `value: undefined` would create a read-only + * own property that breaks later `isTTY = …` assignments in sibling tests. */ + function withStderrTty(isTTY: boolean, run: () => void | Promise) { + const original = Object.getOwnPropertyDescriptor(process.stderr, 'isTTY') + Object.defineProperty(process.stderr, 'isTTY', { + value: isTTY, + configurable: true, + }) + const restore = () => { + if (original) Object.defineProperty(process.stderr, 'isTTY', original) + else delete (process.stderr as { isTTY?: boolean }).isTTY + } + const result = run() + if (result instanceof Promise) return result.finally(restore) + restore() + return result + } + + it('shows + persists the first-run notice even when stderr is NOT a TTY, and keeps noticeShownAt across setTelemetryDisabled', async () => { + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // firstRun + const write = vi.spyOn(process.stderr, 'write').mockReturnValue(true) + try { + await withStderrTty(false, async () => { + const t = await load() + t.maybeShowFirstRunNotice('npx stash') + expect(write).toHaveBeenCalledTimes(1) + expect(h.writeState).toHaveBeenCalledTimes(1) + expect(h.writeState.mock.calls[0][0].noticeShownAt).toBeDefined() + // Fix: the notice reassigns the state cache, so a later opt-out write + // does not clobber the just-persisted noticeShownAt ("shows twice" bug). + t.setTelemetryDisabled(false) + const lastWrite = h.writeState.mock.calls.at(-1)?.[0] + expect(lastWrite?.noticeShownAt).toBeDefined() + }) + } finally { + write.mockRestore() + } + }) + + it('does NOT print the notice when the state write fails — no once-per-run nag loop', async () => { + // Persist-first: an unwritable HOME (sandboxed runner, read-only container) + // must not print the disclosure on every invocation forever. No persist ⇒ + // no print ⇒ no telemetry ever (firstRun never clears) — dormant, not noisy. + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // firstRun + const write = vi.spyOn(process.stderr, 'write').mockReturnValue(true) + try { + const t = await load() + h.writeState.mockImplementation(() => { + throw new Error('EROFS: read-only file system') + }) + // The mocked ../state.js writeState wrapper re-throws through h.writeState. + expect(() => t.maybeShowFirstRunNotice('npx stash')).not.toThrow() + expect(write).not.toHaveBeenCalled() + } finally { + write.mockRestore() + } + }) + + it('shutdownTelemetry resolves within the flush timeout when shutdown() hangs', async () => { + // A black-holed endpoint: shutdown() never resolves. The bounded-flush + // guarantee must still let the process continue via the ~1500ms timeout. + // Real timers (fake timers don't drive the dynamic import); headroom below. + h.shutdown.mockImplementation(() => new Promise(() => {})) + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + }, 4000) +}) diff --git a/packages/cli/src/telemetry/__tests__/telemetry.test.ts b/packages/cli/src/telemetry/__tests__/telemetry.test.ts new file mode 100644 index 00000000..fd89550c --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/telemetry.test.ts @@ -0,0 +1,211 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { CI_ENV_VARS } from '../../config/tty.js' +import { + ALLOWED_PROP_KEYS, + PLACEHOLDER_KEY, + resolveStatus, + sanitize, + type TelemetryStatus, +} from '../index.js' +import type { TelemetryState } from '../state.js' + +const enabledState: TelemetryState = { + anonymousId: 'anon-1', + telemetryDisabled: false, +} + +/** The gates only run once a project key is present; otherwise it's dormant. */ +function withKey(): void { + vi.stubEnv('STASH_POSTHOG_KEY', 'phc_test_key') +} + +/** Clear every env var that resolveStatus / isCiEnvBroad consult, INCLUDING + * STASH_POSTHOG_KEY — a real value in the ambient shell (this repo sets it as a + * GitHub Actions variable) would otherwise flip the dormant tests to enabled. */ +function clearGateEnv(): void { + for (const name of [ + 'DO_NOT_TRACK', + 'STASH_TELEMETRY_DISABLED', + 'STASH_POSTHOG_KEY', + ...CI_ENV_VARS, + ]) { + vi.stubEnv(name, '') + } +} + +function reasonOf(status: TelemetryStatus): string { + return status.enabled ? 'enabled' : status.reason +} + +describe('resolveStatus gates', () => { + beforeEach(() => { + clearGateEnv() + }) + afterEach(() => { + vi.unstubAllEnvs() + }) + + it('is dormant (unconfigured) when no project key is set', () => { + // No STASH_POSTHOG_KEY and the embedded literal is still the placeholder. + expect(resolveStatus(enabledState)).toEqual({ + enabled: false, + reason: 'unconfigured', + }) + }) + + it('an un-injected build (placeholder passthrough) stays dormant', () => { + // Simulates a dev/fork/CI build: tsup baked in no key, so the resolved key + // is the placeholder sentinel. Must read as unconfigured whatever the gates. + vi.stubEnv('STASH_POSTHOG_KEY', PLACEHOLDER_KEY) + expect(resolveStatus(enabledState)).toEqual({ + enabled: false, + reason: 'unconfigured', + }) + }) + + it('a release-injected key is NOT mistaken for the placeholder → enabled', () => { + // The regression the sentinel guards against: an injected key with no env + // override must not read as unconfigured (the old `=== EMBEDDED_KEY` check + // would have). Any real key differs from the placeholder. + vi.stubEnv('STASH_POSTHOG_KEY', 'phc_realish_injected_key') + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + }) + + it('the placeholder sentinel matches the build-define identifier name', () => { + // tsup replaces the `__STASH_POSTHOG_KEY__` identifier at release; the + // sentinel must stay the identically-named string literal, or an un-injected + // build would not be detected as dormant. + expect(PLACEHOLDER_KEY).toBe('__STASH_POSTHOG_KEY__') + }) + + it('treats an empty/whitespace STASH_POSTHOG_KEY as unset → dormant', () => { + // The `?? EMBEDDED_KEY` nullish fallback would let '' slip through and flip a + // dormant build to enabled with an empty key; projectKey() trims it away. + for (const value of ['', ' ']) { + vi.stubEnv('STASH_POSTHOG_KEY', value) + expect(resolveStatus(enabledState)).toEqual({ + enabled: false, + reason: 'unconfigured', + }) + } + }) + + it('is enabled with a key and no opt-out signals', () => { + withKey() + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + }) + + it('DO_NOT_TRACK disables and outranks the persisted flag', () => { + withKey() + vi.stubEnv('DO_NOT_TRACK', '1') + // Even with telemetry "enabled" in config, the env override wins. + expect(reasonOf(resolveStatus(enabledState))).toBe('do-not-track') + }) + + it('STASH_TELEMETRY_DISABLED disables', () => { + withKey() + vi.stubEnv('STASH_TELEMETRY_DISABLED', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('stash-disabled') + }) + + it('DO_NOT_TRACK outranks STASH_TELEMETRY_DISABLED', () => { + withKey() + vi.stubEnv('DO_NOT_TRACK', '1') + vi.stubEnv('STASH_TELEMETRY_DISABLED', '1') + expect(reasonOf(resolveStatus(enabledState))).toBe('do-not-track') + }) + + it('auto-disables in CI', () => { + withKey() + vi.stubEnv('CI', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('ci') + }) + + it('honors a non-standard CI marker (GITHUB_ACTIONS)', () => { + withKey() + vi.stubEnv('GITHUB_ACTIONS', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('ci') + }) + + it('the persisted flag disables when no env override is present', () => { + withKey() + expect( + reasonOf(resolveStatus({ ...enabledState, telemetryDisabled: true })), + ).toBe('config') + }) + + it('treats DO_NOT_TRACK=0 / false / empty as not opted out', () => { + withKey() + for (const value of ['0', 'false', '']) { + vi.stubEnv('DO_NOT_TRACK', value) + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + } + }) + + it('honors DO_NOT_TRACK set to any other non-empty value', () => { + withKey() + // The convention is that setting the variable at all signals opt-out. + for (const value of ['1', 'true', 'on', 'yes', 'please']) { + vi.stubEnv('DO_NOT_TRACK', value) + expect(reasonOf(resolveStatus(enabledState))).toBe('do-not-track') + } + }) + + it('auto-disables for a provider marker without CI (GitLab)', () => { + withKey() + vi.stubEnv('GITLAB_CI', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('ci') + }) +}) + +describe('sanitize (property allowlist)', () => { + it('drops any key not on the allowlist', () => { + const out = sanitize({ + command: 'eql', + subcommand: 'install', + // Everything below is exactly what must never leave the machine. + table: 'users', + column: 'email', + databaseUrl: 'postgres://secret@host/db', + plaintext: 'alice@example.com', + argv: ['--database-url', 'postgres://…'], + }) + expect(out).toEqual({ command: 'eql', subcommand: 'install' }) + }) + + it('keeps the full coarse event shape', () => { + const event = { + command: 'encrypt', + subcommand: 'backfill', + success: true, + durationMs: 1234, + errorType: undefined, + cliVersion: '0.17.1', + os: 'darwin', + arch: 'arm64', + nodeVersion: '22.0.0', + caller: 'claude-code', + } + expect(sanitize(event)).toEqual(event) + for (const key of Object.keys(event)) { + expect(ALLOWED_PROP_KEYS.has(key)).toBe(true) + } + }) + + it('the allowlist contains no identifying or payload fields', () => { + for (const forbidden of [ + 'table', + 'column', + 'schema', + 'databaseUrl', + 'connectionString', + 'plaintext', + 'ciphertext', + 'argv', + 'args', + 'value', + ]) { + expect(ALLOWED_PROP_KEYS.has(forbidden)).toBe(false) + } + }) +}) diff --git a/packages/cli/src/telemetry/classify-command.ts b/packages/cli/src/telemetry/classify-command.ts new file mode 100644 index 00000000..5d25d1db --- /dev/null +++ b/packages/cli/src/telemetry/classify-command.ts @@ -0,0 +1,95 @@ +import { registry } from '../cli/registry.js' + +/** + * Coerce the raw, argv-derived (command, subcommand) to a fixed vocabulary + * before telemetry emit. + * + * `subcommand` is `argv[1]` (see `parseArgs`): for most commands it's a fixed + * verb, but for a few it's a free-form positional — `stash wizard "add + * encryption to patients.ssn"` puts a natural-language prompt (routinely + * containing table/column names) in that slot. The event allowlist gates + * property KEYS, not values, so without this an arbitrary user string would + * leave the machine as the `subcommand` value, breaking the "no table/column + * names, no argument values" contract. Anything not on the known-verb list + * collapses to ``, so we learn "a wizard ran" without learning what for. + */ +const OTHER = '' + +/** + * Full command paths (`"eql install"`, `"auth login"`, `"init"`, …) from the + * registry — the single source of truth that also backs `stash manifest` and + * `--help` — plus the `telemetry` sub-verbs the registry doesn't enumerate. + * Hidden (deprecated) descriptors are excluded. + */ +const KNOWN_PATHS: ReadonlySet = new Set([ + ...registry.flatMap((group) => + group.commands.filter((c) => !c.hidden).map((c) => c.name), + ), + 'telemetry status', + 'telemetry enable', + 'telemetry disable', +]) + +/** The first token of every known path — the set of recognised top-level commands. */ +const KNOWN_COMMANDS: ReadonlySet = new Set( + [...KNOWN_PATHS].map((path) => path.split(' ')[0]), +) + +/** + * Error class names allowed to leave as `errorType`. The same closed-vocabulary + * rule as commands: the CLI executes USER code in-process (stash.config.ts and + * the encrypt client, via jiti), so `err.constructor.name` is an open + * vocabulary — a user-defined `PatientsSsnColumnMissingError` would carry a + * column name off the machine. First-party classes + Node/JS builtins pass; + * everything else collapses to ``. + */ +const KNOWN_ERROR_TYPES: ReadonlySet = new Set([ + // JS builtins + 'Error', + 'TypeError', + 'RangeError', + 'SyntaxError', + 'ReferenceError', + 'EvalError', + 'URIError', + 'AggregateError', + // Node-flavoured + 'AbortError', + 'TimeoutError', + 'SystemError', + // First-party CLI errors + 'CliExit', + 'CancelledError', + 'BackfillConfigError', + // Config validation (zod) + 'ZodError', +]) + +/** Coerce an unknown thrown value to a telemetry-safe error class name. */ +export function classifyErrorType(err: unknown): string { + if (!(err instanceof Error)) return '' + return KNOWN_ERROR_TYPES.has(err.constructor.name) + ? err.constructor.name + : '' +} + +/** + * Return telemetry-safe (command, subcommand). An unrecognised command becomes + * `` (and drops its subcommand); a recognised command with an + * unrecognised subcommand keeps the command but reports `` for the + * subcommand, so a free-text positional never leaves as-is. + */ +export function classifyCommand( + command: string, + subcommand: string | undefined, +): { command: string; subcommand: string | undefined } { + if (!KNOWN_COMMANDS.has(command)) { + return { command: OTHER, subcommand: undefined } + } + if (subcommand === undefined) { + return { command, subcommand: undefined } + } + return KNOWN_PATHS.has(`${command} ${subcommand}`) + ? { command, subcommand } + : { command, subcommand: OTHER } +} diff --git a/packages/cli/src/telemetry/index.ts b/packages/cli/src/telemetry/index.ts new file mode 100644 index 00000000..eb87f816 --- /dev/null +++ b/packages/cli/src/telemetry/index.ts @@ -0,0 +1,368 @@ +import type { PostHog } from 'posthog-node' +import { isCiEnvBroad, resolveCaller } from '../config/tty.js' +import { messages } from '../messages.js' +import { readState, type TelemetryState, writeState } from './state.js' + +/** + * Anonymous, opt-out CLI usage analytics. + * + * Design (see the CLI analytics plan): + * - Opt-out by default, but **nothing is sent until the first-run notice has + * been shown once** — the run that shows it is always a freebie. + * - Four opt-out gates, any of which disables: `DO_NOT_TRACK`, + * `STASH_TELEMETRY_DISABLED`, CI auto-detection, and the persisted + * `stash telemetry disable` flag. + * - Events are anonymous (no PostHog "person" profiles) and carry only a fixed + * allowlist of coarse properties. Table/column/schema names, connection + * strings, plaintext, ciphertext, and raw argument values can never leave — + * {@link sanitize} enforces the key allowlist, and callers must coerce the + * VALUES of `command`/`subcommand` to a known vocabulary first (the raw argv + * token is a value, not a key, so sanitize alone would pass it through — see + * `classifyCommand` in `./classify-command.ts`). + * - Sending never blocks or slows the CLI: this module reads no disk and loads + * no posthog-node until a real event is actually sent (both are deferred off + * the `--version`/`--help` fast paths), flushing is bounded by a timeout, and + * every failure is swallowed. + * - KNOWN MEASUREMENT GAP: commands that terminate via a deep `process.exit()` + * (mid-flow cancels in db install, auth login failures, config-loader exits, + * clack's own keypress exit) emit no event — the process ends before the + * flush could run. Global interception was tried and reverted (it broke + * cancel flows; see `cli/exit.ts`). First-party sites verified to unwind + * cleanly use `throw new CliExit(code)` instead, which IS tracked. Read + * success rates as covering returning/throwing/CliExit paths only. + */ + +/** Default endpoint — our Cloudflare proxy, not PostHog directly, so a future + * US→EU migration is a proxy-target change with no CLI re-release. + * `STASH_POSTHOG_HOST` overrides it (testing against a real PostHog ingestion + * endpoint before the proxy is deployed, or self-hosting), symmetric with + * `STASH_POSTHOG_KEY`. */ +const DEFAULT_POSTHOG_HOST = 'https://telemetry.cipherstash.com' + +function posthogHost(): string { + return process.env.STASH_POSTHOG_HOST ?? DEFAULT_POSTHOG_HOST +} + +/** + * The un-injected sentinel. A build that has NOT embedded a key resolves to + * exactly this string, and {@link resolveStatus} treats it as "no key" → + * dormant. It is a plain string literal (not the `__STASH_POSTHOG_KEY__` + * identifier), so the build-time `define` below never rewrites it. + */ +export const PLACEHOLDER_KEY = '__STASH_POSTHOG_KEY__' + +/** + * Build-time define. The release build (`.github/workflows/release.yml`) + * replaces the `__STASH_POSTHOG_KEY__` identifier with the real, public, + * write-only PostHog project key from the `STASH_POSTHOG_KEY` repo variable, via + * tsup's esbuild `define` (see `tsup.config.ts`). Safe to embed, exactly like a + * web SDK key. EVERY other build — local dev, a contributor's checkout, a fork, + * CI unit tests — leaves the identifier undefined, so {@link EMBEDDED_KEY} falls + * back to {@link PLACEHOLDER_KEY} and telemetry stays fully dormant. The `typeof` + * guard makes the reference safe even when the identifier was never defined + * (`typeof undefinedIdent` yields `"undefined"` rather than throwing). + */ +declare const __STASH_POSTHOG_KEY__: string | undefined + +const EMBEDDED_KEY = + typeof __STASH_POSTHOG_KEY__ === 'string' && __STASH_POSTHOG_KEY__.length > 0 + ? __STASH_POSTHOG_KEY__ + : PLACEHOLDER_KEY + +/** `STASH_POSTHOG_KEY` in the environment overrides the embedded key entirely + * (testing / self-hosting); otherwise the build-time value is used. An empty or + * whitespace-only override counts as unset — otherwise `STASH_POSTHOG_KEY=''` + * would slip past the nullish fallback and flip a dormant build to "enabled" + * with an empty key (asymmetric with the `.length > 0` guard on EMBEDDED_KEY). */ +function projectKey(): string { + const override = process.env.STASH_POSTHOG_KEY?.trim() + return override ? override : EMBEDDED_KEY +} + +/** Flush is bounded so a slow or unreachable endpoint can't hang `stash`. */ +const FLUSH_TIMEOUT_MS = 1500 + +export type TelemetryDisabledReason = + | 'do-not-track' + | 'stash-disabled' + | 'ci' + | 'config' + | 'unconfigured' + +export type TelemetryStatus = + | { enabled: true } + | { enabled: false; reason: TelemetryDisabledReason } + +/** The only property keys allowed to leave the machine. Everything else is dropped. */ +export const ALLOWED_PROP_KEYS: ReadonlySet = new Set([ + 'command', + 'subcommand', + 'success', + 'durationMs', + 'errorType', + 'cliVersion', + 'os', + 'arch', + 'nodeVersion', + 'caller', +]) + +/** + * An opt-out env var is "set" when present and not an explicit off value. This + * is deliberately broad: `DO_NOT_TRACK=1`, `=true`, `=on`, or any other non-empty + * value opts out, matching the DO_NOT_TRACK convention that setting the variable + * at all signals intent. Only `''`, `'0'`, and `'false'` mean "not opted out". + */ +function envOptOut(name: string): boolean { + const raw = process.env[name]?.trim().toLowerCase() + return raw != null && raw !== '' && raw !== '0' && raw !== 'false' +} + +/** + * Resolve whether telemetry is on, in precedence order. Env overrides win over + * the persisted flag, so a user can force telemetry off in a context where the + * config file says otherwise (and never the reverse). CI detection uses the + * BROAD check ({@link isCiEnvBroad} — provider markers included), deliberately + * separate from the narrow `isCiEnv` that gates interactive prompts: for + * telemetry a false CI positive merely skips collection, whereas for prompts it + * would break an interactive command. + */ +export function resolveStatus(state: TelemetryState): TelemetryStatus { + // Compare against the sentinel, NOT EMBEDDED_KEY: once a real key is injected + // at release, EMBEDDED_KEY holds it, and `=== EMBEDDED_KEY` would then read a + // key-present build (no env override) as unconfigured. The placeholder passing + // through — no build-time key and no env override — is the "dormant" signal. + if (projectKey() === PLACEHOLDER_KEY) { + return { enabled: false, reason: 'unconfigured' } + } + if (envOptOut('DO_NOT_TRACK')) { + return { enabled: false, reason: 'do-not-track' } + } + if (envOptOut('STASH_TELEMETRY_DISABLED')) { + return { enabled: false, reason: 'stash-disabled' } + } + if (isCiEnvBroad()) return { enabled: false, reason: 'ci' } + if (state.telemetryDisabled) return { enabled: false, reason: 'config' } + return { enabled: true } +} + +/** Defence-in-depth: keep only allowlisted keys, whatever a caller passes. */ +export function sanitize( + properties: Record, +): Record { + const out: Record = {} + for (const [key, value] of Object.entries(properties)) { + if (ALLOWED_PROP_KEYS.has(key)) out[key] = value + } + return out +} + +// Module state is resolved LAZILY on first telemetry use, not at import time, so +// fast paths that never call a telemetry function (`--version`, `--help`) touch +// no disk — the state file read (and its randomUUID on a fresh machine) is +// deferred out of the CLI's hottest paths. +let stateCache: TelemetryState | undefined +let statusCache: TelemetryStatus | undefined +let firstRunCache = false + +/** + * Read the state file once per process and derive status + first-run. Returns + * the resolved values so callers never touch the (possibly stale) module caches + * directly. `firstRun` stays true for the whole process even after the notice is + * persisted, so the current run never sends — the freebie. + */ +function init(): { + state: TelemetryState + status: TelemetryStatus + firstRun: boolean +} { + if (stateCache === undefined || statusCache === undefined) { + stateCache = readState() + statusCache = resolveStatus(stateCache) + firstRunCache = stateCache.noticeShownAt === undefined + } + return { state: stateCache, status: statusCache, firstRun: firstRunCache } +} + +let cliVersion = '0.0.0' +// posthog-node is imported dynamically the first time an event is actually sent +// (see getClient), so it is NOT pulled into every `stash` invocation — only when +// telemetry is enabled AND a real command is being tracked. clientPromise is the +// "was anything ever sent this process?" signal shutdownTelemetry gates on. +let clientPromise: Promise | null = null +// The last capture()'s enqueue promise; shutdownTelemetry awaits it before +// flushing so an event fired on an exit path is delivered, not raced away. +let lastCapture: Promise | null = null + +/** Provide the CLI version for event properties. Call once at startup. */ +export function initTelemetry(version: string): void { + cliVersion = version +} + +export function telemetryStatus(): TelemetryStatus { + return init().status +} + +async function getClient(): Promise { + if (clientPromise === null) { + clientPromise = import('posthog-node').then( + ({ PostHog }) => + new PostHog(projectKey(), { + host: posthogHost(), + flushAt: 1, + flushInterval: 0, + // Fire-and-forget: a short-lived CLI must not retry a failed send. + // Retries (default 3, exponential backoff) would keep internal timers + // pending and hang the process. Drop the event instead. + fetchRetryCount: 0, + // Bound PostHog's OWN request (undici AbortSignal.timeout). Without + // this it defaults to 10s, so a black-holed endpoint keeps the socket + // — and the event loop — alive long past our flush window, defeating + // the bounded-flush guarantee. Match the flush budget. + requestTimeout: FLUSH_TIMEOUT_MS, + // Never resolve IP → geo; we don't want or store location. + disableGeoip: true, + }), + ) + } + return clientPromise +} + +function baseProps(): Record { + return { + cliVersion, + os: process.platform, + arch: process.arch, + nodeVersion: process.versions.node, + // Coarse, fixed-enum classification of the caller (agent harness vs + // interactive shell). Never the raw env value — see resolveCaller. + caller: resolveCaller(), + } +} + +export interface CommandEvent { + command: string + subcommand?: string + success: boolean + durationMs: number + /** The error constructor name on failure — a class, never a message. */ + errorType?: string +} + +/** + * Record that a command ran. A no-op when telemetry is disabled or on the + * first run. Never throws. + */ +export function trackCommand(event: CommandEvent): void { + const { state, status, firstRun } = init() + if (!status.enabled || firstRun) return + const properties = { + // Sanitize the FULL property set (event + base) so the allowlist is the + // single enforced boundary — a future prop added to baseProps() can't + // bypass it. Only the explicit PostHog control key is added afterward. + ...sanitize({ ...event, ...baseProps() }), + // Keep events anonymous: no PostHog person profiles. + $process_person_profile: false, + } + // Fire-and-forget. getClient() dynamically imports posthog-node only now, so a + // disabled/dormant run never loads it. shutdownTelemetry awaits lastCapture + // before flushing so the event is delivered even on process.exit paths. + lastCapture = getClient() + .then((client) => { + client.capture({ + distinctId: state.anonymousId, + event: 'command_invoked', + properties, + }) + }) + .catch(() => { + // Telemetry must never surface in the command path. + }) +} + +/** + * Show the one-time first-run notice (to stderr, so it never pollutes piped or + * `--json` stdout) and mark it shown. No-op when telemetry is disabled or the + * notice was already shown. The run that shows it sends nothing (the freebie). + * + * The disclosure is written on EVERY first run, TTY or not — a non-interactive + * caller (an agent harness, a piped invocation) still gets it in its stderr/log, + * and `noticeShownAt` is persisted so that machine advances past the freebie and + * starts sending on its next run. Gating persistence on a TTY (as before) left + * exactly the agent population the `caller` dimension exists to measure dormant + * forever. `stashRef` is the runner-aware invocation (e.g. `npx stash`) so the + * opt-out hint is actionable before the CLI is on PATH. + */ +export function maybeShowFirstRunNotice(stashRef: string): void { + const { state, status, firstRun } = init() + if (!status.enabled || !firstRun) return + try { + // PERSIST FIRST, print only on success. On a machine where the state write + // fails (read-only HOME, sandboxed runner) the notice would otherwise print + // on every single run forever — worse than pre-notice behaviour. Persist + // failure ⇒ no print AND no telemetry ever (firstRun never clears), which is + // the honest fallback: no disclosure, no collection. + // + // Reassign the cache: a later setTelemetryDisabled() in the same process + // must not write from a stale `state` and clobber this noticeShownAt. + stateCache = writeState({ + ...state, + noticeShownAt: new Date().toISOString(), + }) + } catch { + return + } + process.stderr.write(`${messages.telemetry.notice(stashRef)}\n`) +} + +/** Persist the opt-out flag. Surfaces write errors (the command wants to know). */ +export function setTelemetryDisabled(disabled: boolean): void { + const { state } = init() + stateCache = writeState({ ...state, telemetryDisabled: disabled }) + statusCache = resolveStatus(stateCache) +} + +/** Flush any buffered events, bounded by {@link FLUSH_TIMEOUT_MS}. Never throws. */ +export async function shutdownTelemetry(): Promise { + // clientPromise is null iff nothing was ever captured (disabled/dormant/first + // run), so there is nothing to flush and posthog-node was never loaded. + if (clientPromise === null) return + // The timer MUST be cleared once the flush wins the race: an uncleared + // pending setTimeout keeps the Node event loop alive, so the process would + // hang for the full timeout after the flush already completed. + let timer: ReturnType | undefined + // Capture the promises before the race so the finally can reset the module + // state regardless of which side wins. + const pendingCapture = lastCapture + const pendingClient = clientPromise + try { + // EVERYTHING is inside the race — including awaiting the capture enqueue + // (which contains the dynamic import of posthog-node) and the client + // resolution. If any of it stalls (a hung filesystem mid-import, a + // black-holed endpoint), the timeout still unblocks the process; nothing + // that can hang sits outside the bound. The swallow is attached to the + // flush promise ITSELF (not just the race): if the timeout wins while the + // flush is still pending, a later rejection would otherwise surface as an + // unhandledRejection crash. + const flush = (async () => { + // Wait for the capture enqueue (client construction + queueing) to + // finish, otherwise shutdown() could race ahead of the event we mean + // to deliver. Its rejections are already swallowed in trackCommand. + if (pendingCapture !== null) await pendingCapture + const client = await pendingClient + await client?.shutdown() + })().catch(() => { + // Swallow — a failed flush must not fail (or crash) the process. + }) + await Promise.race([ + flush, + new Promise((resolve) => { + timer = setTimeout(resolve, FLUSH_TIMEOUT_MS) + }), + ]) + } finally { + if (timer !== undefined) clearTimeout(timer) + clientPromise = null + lastCapture = null + } +} diff --git a/packages/cli/src/telemetry/state.ts b/packages/cli/src/telemetry/state.ts new file mode 100644 index 00000000..fab18b1b --- /dev/null +++ b/packages/cli/src/telemetry/state.ts @@ -0,0 +1,74 @@ +import { randomUUID } from 'node:crypto' +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' + +/** + * Machine-level telemetry state, persisted alongside the auth profile in + * `~/.cipherstash/` but in its own file. We only ever read/write `telemetry.json` + * here — never the auth secrets (`auth.json`, `secretkey.json`, `workspaces/`). + * + * The file holds three things: a random anonymous id (for de-duplicating events + * in aggregate, never derivable to a person), the persisted opt-out flag written + * by `stash telemetry disable`, and the timestamp of the first-run notice so the + * banner shows exactly once per install. + */ +export interface TelemetryState { + /** Random UUID; the PostHog `distinctId`. Not tied to any identity. */ + anonymousId: string + /** Set by `stash telemetry disable`. The lowest-precedence opt-out gate. */ + telemetryDisabled: boolean + /** ISO timestamp the first-run notice was shown; unset means "never shown". */ + noticeShownAt?: string +} + +/** Resolved at call time so it always tracks the current home directory. */ +function stateDir(): string { + return path.join(os.homedir(), '.cipherstash') +} +function stateFile(): string { + return path.join(stateDir(), 'telemetry.json') +} + +/** Coerce arbitrary parsed JSON into a valid state, filling gaps with defaults. */ +function normalize(value: unknown): TelemetryState { + const o = ( + typeof value === 'object' && value !== null ? value : {} + ) as Record + return { + anonymousId: + typeof o.anonymousId === 'string' && o.anonymousId.length > 0 + ? o.anonymousId + : randomUUID(), + telemetryDisabled: o.telemetryDisabled === true, + noticeShownAt: + typeof o.noticeShownAt === 'string' ? o.noticeShownAt : undefined, + } +} + +/** + * Read the state file. Never throws: a missing or corrupt file yields a fresh + * default (with a new anonymous id), so a bad file can never break a command. + * The fresh id is only ephemeral until something calls {@link writeState}. + */ +export function readState(): TelemetryState { + try { + return normalize(JSON.parse(fs.readFileSync(stateFile(), 'utf-8'))) + } catch { + return { anonymousId: randomUUID(), telemetryDisabled: false } + } +} + +/** + * Persist state (0600, private). Returns the normalized value actually written. + * Unlike {@link readState} this may throw — callers that must not fail a command + * (the emitter) wrap it; the `stash telemetry` command lets a write error surface. + */ +export function writeState(state: TelemetryState): TelemetryState { + const normalized = normalize(state) + fs.mkdirSync(stateDir(), { recursive: true }) + fs.writeFileSync(stateFile(), `${JSON.stringify(normalized, null, 2)}\n`, { + mode: 0o600, + }) + return normalized +} diff --git a/packages/cli/tests/e2e/smoke.e2e.test.ts b/packages/cli/tests/e2e/smoke.e2e.test.ts index f059e3c3..f872b19e 100644 --- a/packages/cli/tests/e2e/smoke.e2e.test.ts +++ b/packages/cli/tests/e2e/smoke.e2e.test.ts @@ -99,6 +99,23 @@ describe('stash CLI — non-interactive smoke', () => { expect(r.output).toContain('bogus-sub') }) + it('telemetry status reports the current state and exits 0', async () => { + const r = render(['telemetry', 'status']) + const { exitCode } = await r.exit + expect(exitCode).toBe(0) + // Dev builds carry the placeholder key, so status reports the dormant build + // (the harness also sets STASH_TELEMETRY_DISABLED, but unconfigured wins). + expect(r.output).toContain('Telemetry is') + }) + + it('telemetry bogus-sub exits 1 (the CliExit cooperative-exit path)', async () => { + const r = render(['telemetry', 'bogus-sub']) + const { exitCode } = await r.exit + expect(exitCode).toBe(1) + expect(r.output).toContain(messages.telemetry.unknownSubcommand) + expect(r.output).toContain('bogus-sub') + }) + // `--migration` without `--supabase` fails flag validation before any I/O // or prompt, so these two cases can observe the install entry path // deterministically without a database. diff --git a/packages/cli/tests/helpers/pty.ts b/packages/cli/tests/helpers/pty.ts index 3d7b787e..8a559011 100644 --- a/packages/cli/tests/helpers/pty.ts +++ b/packages/cli/tests/helpers/pty.ts @@ -4,6 +4,7 @@ import { dirname, join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' import { type IPty, spawn } from 'node-pty' import stripAnsi from 'strip-ansi' +import { CI_ENV_VARS } from '../../src/config/tty.js' const __dirname = dirname(fileURLToPath(import.meta.url)) @@ -98,11 +99,25 @@ export function render(args: string[], opts: RenderOptions = {}): Rendered { // debugging a failure. NO_COLOR: '1', FORCE_COLOR: '0', - // Match the convention the CLI itself uses (e.g. install.ts checks - // `process.env.CI !== 'true'`) so test runs hit the same code paths. - CI: 'true', - ...(opts.env ?? {}), } + // Strip every CI marker so the harness default below (and per-test overrides) + // fully control CI detection. Prompt gating (isCiEnv) reads only `CI`, but + // telemetry's isCiEnvBroad consults the provider markers (GITHUB_ACTIONS, + // GITLAB_CI, …), which leak in from the ambient environment — this repo's own + // CI — and would otherwise flip telemetry gating in a test that de-CIs with + // `env: { CI: '' }`. + for (const name of CI_ENV_VARS) delete env[name] + // Match the convention the CLI itself uses (e.g. install.ts checks + // `process.env.CI !== 'true'`) so test runs hit the same code paths. + env.CI = 'true' + // Hard-disable telemetry regardless of the ambient shell. Without this, a + // developer with STASH_POSTHOG_KEY exported (the documented testing override) + // running the e2e suite would spawn telemetry-ENABLED CLIs against their real + // ~/.cipherstash — sending genuine events from tests and injecting the + // first-run notice into pty output. The CI='true' default is not enough: the + // de-CI'ing tests strip it. Tests exercising telemetry itself can override. + env.STASH_TELEMETRY_DISABLED = '1' + Object.assign(env, opts.env ?? {}) // Use the absolute path to the current node binary — node-pty's // `posix_spawnp` doesn't inherit PATH lookup reliably across all macOS / diff --git a/packages/cli/tsup.config.ts b/packages/cli/tsup.config.ts index 0a85deea..2f17f343 100644 --- a/packages/cli/tsup.config.ts +++ b/packages/cli/tsup.config.ts @@ -1,6 +1,18 @@ import { cpSync, existsSync } from 'node:fs' import { defineConfig } from 'tsup' +/** + * Build-time value for the embedded PostHog project key (see + * `src/telemetry/index.ts`). Only the release workflow sets `STASH_POSTHOG_KEY` + * (from a public repo variable), so every other build — dev, forks, CI — bakes + * in an empty string and the CLI ships telemetry-dormant. Applied to both bundles + * because either may inline the telemetry module. The value must be a JS + * expression string, hence `JSON.stringify`. + */ +const posthogKeyDefine = { + __STASH_POSTHOG_KEY__: JSON.stringify(process.env.STASH_POSTHOG_KEY ?? ''), +} + export default defineConfig([ { entry: ['src/index.ts'], @@ -17,6 +29,7 @@ export default defineConfig([ ...options.logOverride, 'empty-import-meta': 'silent', } + options.define = { ...options.define, ...posthogKeyDefine } }, onSuccess: async () => { // Copy bundled SQL files into dist so they ship with the package @@ -52,5 +65,8 @@ var require = __createRequire(import.meta.url);`, sourcemap: true, skipNodeModulesBundle: true, + esbuildOptions(options) { + options.define = { ...options.define, ...posthogKeyDefine } + }, }, ]) diff --git a/packages/wizard/src/lib/analytics.ts b/packages/wizard/src/lib/analytics.ts index a8e78853..f58cdf35 100644 --- a/packages/wizard/src/lib/analytics.ts +++ b/packages/wizard/src/lib/analytics.ts @@ -3,43 +3,67 @@ * * Tracks wizard interactions, framework detection, completion rates, and errors. * Analytics are non-blocking — failures are silently ignored. + * + * Honors the same opt-outs as the `stash` CLI's telemetry: `DO_NOT_TRACK` + * (cross-tool standard), `STASH_TELEMETRY_DISABLED`, and CI auto-detection. + * The wizard is usually spawned by `stash wizard`, whose first-run notice + * promises those opt-outs on behalf of the whole CLI surface — this module + * must not make that promise false. */ +import { randomUUID } from 'node:crypto' import { PostHog } from 'posthog-node' import { POSTHOG_API_KEY, POSTHOG_HOST } from './constants.js' import type { Integration, WizardSession } from './types.js' let client: PostHog | undefined +/** + * An opt-out env var is "set" when present and not an explicit off value — + * matching the DO_NOT_TRACK convention (and the CLI's `envOptOut`) that setting + * the variable at all signals intent. + */ +function envOptOut(name: string): boolean { + const raw = process.env[name]?.trim().toLowerCase() + return raw != null && raw !== '' && raw !== '0' && raw !== 'false' +} + +function analyticsDisabled(): boolean { + return ( + envOptOut('DO_NOT_TRACK') || + envOptOut('STASH_TELEMETRY_DISABLED') || + Boolean(process.env.CI?.trim()) + ) +} + function getClient(): PostHog | undefined { if (!POSTHOG_API_KEY) return undefined + if (analyticsDisabled()) return undefined if (!client) { client = new PostHog(POSTHOG_API_KEY, { host: POSTHOG_HOST, flushAt: 1, flushInterval: 0, + // Never resolve IP → geo. + disableGeoip: true, }) } return client } -import { createHash } from 'node:crypto' -import { hostname, userInfo } from 'node:os' +/** + * Per-process random identifier. Deliberately NOT derived from anything + * identifying: the previous sha256(username@hostname) was reversible by + * brute-forcing common username/hostname pairs. Session-stable is enough for + * funnel analysis within one wizard run; cross-run identity is not worth an + * identifying hash. + */ +const SESSION_ID = randomUUID() -/** Generate a stable anonymous identifier for the session. */ function getDistinctId(): string { - try { - const user = userInfo().username - const host = hostname() - return createHash('sha256') - .update(`${user}@${host}`) - .digest('hex') - .slice(0, 16) - } catch { - return 'anonymous' - } + return SESSION_ID } // --- Event tracking --- @@ -103,6 +127,11 @@ export function trackWizardCompleted( }) } +/** + * `error` must be a FIXED LABEL or an error class name — never a raw + * message. Agent/DB error messages routinely embed file paths, table/column + * names, and connection-string fragments; those must not leave the machine. + */ export function trackWizardError(error: string, integration?: Integration) { getClient()?.capture({ distinctId: getDistinctId(), diff --git a/packages/wizard/src/run.ts b/packages/wizard/src/run.ts index 87259688..fab22186 100644 --- a/packages/wizard/src/run.ts +++ b/packages/wizard/src/run.ts @@ -262,7 +262,10 @@ export async function run(options: RunOptions) { }) } } else { - trackWizardError(result.error ?? 'unknown', selectedIntegration) + // Fixed label only — result.error is the agent's raw message and can + // embed schema names or connection details; it stays local (changelog + + // stderr below), never on the wire. + trackWizardError('agent_failed', selectedIntegration) changelog.note(`Agent failed: ${result.error ?? 'unknown error'}`) await changelog.flush() p.log.error(result.error ?? 'Agent failed without a specific error.') @@ -273,7 +276,12 @@ export async function run(options: RunOptions) { } catch (error) { const message = error instanceof Error ? error.message : 'Agent execution failed.' - trackWizardError(message, selectedIntegration) + // Class name only — the message can embed schema names or connection + // details; it stays local (changelog + stderr below), never on the wire. + trackWizardError( + error instanceof Error ? error.constructor.name : 'unknown', + selectedIntegration, + ) changelog.note(`Wizard threw: ${message}`) await changelog.flush() p.log.error(message) diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index fca5c955..32e0df19 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -16,7 +16,7 @@ Think Prisma Migrate or Drizzle Kit: a dev-time tool that prepares the database, Use this skill when: - The user wants to set up CipherStash or install EQL in a PostgreSQL database. -- Any `stash` command is being run: `init`, `plan`, `impl`, `status`, `auth`, `eql`, `db`, `encrypt`, `schema`, `manifest`, `doctor`, `wizard`, `env`. +- Any `stash` command is being run: `init`, `plan`, `impl`, `status`, `auth`, `eql`, `db`, `encrypt`, `schema`, `manifest`, `doctor`, `telemetry`, `wizard`, `env`. - A `stash.config.ts` file exists or needs to be created. - A `.cipherstash/` directory exists (`context.json`, `plan.md`, `migrations.json`, `setup-prompt.md`). - The user mentions "stash CLI", "EQL install", "encryption schema", or an encryption rollout/cutover. @@ -140,6 +140,33 @@ First hit wins: The resolved URL is returned in memory only. It is never written to disk or into `process.env`. +## Telemetry + +The CLI collects **anonymous, opt-out** usage analytics — coarse events only +(command name, CLI version, OS/arch, Node version, success/failure, duration, +and a coarse caller class such as `claude-code`/`cursor`/`interactive` derived +from environment markers). Events carry a random install identifier — a UUID +generated locally and stored in `~/.cipherstash/telemetry.json`, not derived +from any machine, user, or hardware attribute — used only to de-duplicate +events in aggregate. It **never** collects plaintext, schema, table/column +names, connection strings, argument values, or any session/trace identifier. A +one-time notice is printed on first run, and nothing is sent on that first run. + +Opt out in any of these ways (any one wins; env vars override the saved +preference): + +| Mechanism | Effect | +|---|---| +| `DO_NOT_TRACK=1` | Honors the cross-tool standard; disables telemetry | +| `STASH_TELEMETRY_DISABLED=1` | Disables telemetry | +| `CI=true` (or common CI markers) | Auto-disabled in CI | +| `npx stash telemetry disable` | Persists opt-out to `~/.cipherstash/telemetry.json` | + +`npx stash telemetry status` reports the current state and which setting governs +it; `npx stash telemetry enable` clears the saved opt-out (env overrides still +apply). State lives in `~/.cipherstash/telemetry.json` — a non-secret file +distinct from the auth credentials in that directory. + ## Configuration `stash.config.ts` in the project root: @@ -279,6 +306,7 @@ Flags below are the decision-relevant ones. Run `stash --help` for the | `status` | Rollout quest log (above) | | `manifest [--json]` | Print the structured, versioned command surface | | `doctor` | Diagnose install problems (native binaries, runtime). Runs before the CLI body loads, so it works when the native binary is broken. | +| `telemetry [status\|enable\|disable]` | Manage anonymous usage analytics (below) | | `wizard` | AI-guided encryption setup — thin wrapper over `@cipherstash/wizard` | ### Auth diff --git a/turbo.json b/turbo.json index 64780c60..4ef8a864 100644 --- a/turbo.json +++ b/turbo.json @@ -3,7 +3,8 @@ "tasks": { "build": { "dependsOn": ["^build"], - "inputs": ["$TURBO_DEFAULT$", ".env*"] + "inputs": ["$TURBO_DEFAULT$", ".env*"], + "env": ["STASH_POSTHOG_KEY"] }, "release": { "dependsOn": ["^build"],