diff --git a/README.md b/README.md index fe53acb..023d45d 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,8 @@ Restart pi after install so the extension loads. | ------ | --------- | | `bgrun` | Launch a command detached in the background. Optional `name` gives the job a short human-readable label. Returns `started: ` immediately. Wakes the session automatically on completion. | | `bgstatus` | Show job status. With an id: any job's state + exit code. Without: this session's running jobs (finished jobs hidden by default — pass `includeDone: true` or set `showCompletedJobs`). Jobs from other sessions are only listed when `adoptForeignJobs` is enabled. | -| `bgtail` | Print the last N lines of a job's log (default 40), **condensed for context**: ANSI escapes stripped, repeated lines collapsed, long lines and total size capped. Pass `raw: true` to skip condensing. | +| `bgtail` | Read the newest lines of a job's log (default 40), **condensed for context**: ANSI escapes stripped, repeated lines collapsed, long lines and total size capped. First read = full last-N tail; repeat reads return **only lines appended since your last read** (delta tailing) — polling a running job never re-pays for lines already seen. Pass `raw: true` for the unprocessed last-N window (still advances the bookmark). | +| `bggrep` | Regex search over a job's log: line-numbered matches, optional `context` lines, capped (~50 matches, ~2KB/line, ~8KB) and condensed. Runs inside the extension, so it reaches **any** jobs dir — including global logs that project-sandboxed tools (`ctx_execute_file`) cannot. With no `pattern`, a generic failure-signature default is used (override it — convenience, not guarantee). | | `bgclean` | Remove old job logs. **Default scope: this session's jobs only** — other sessions' logs are untouched. Pass `all: true` to sweep the whole shared jobs dir. Retention: `cleanupDays` config (7 days). Never removes a running job's log. | ## Slash commands @@ -85,11 +86,26 @@ exit code even after a restart. Two-tier read model — the log file stays complete on disk for deep analysis; only bounded digests ever enter the conversation: -- **Quick peek:** `bgtail ` — condensed last-40-lines (ANSI stripped, repeats -collapsed, ~8KB cap). The wake message itself already carries the exit code - and the log's last line, so many turns need no follow-up read at all. -- **Whole-log analysis:** `ctx_execute_file` on the job's log path to extract - only failure lines. Never `cat` or `Read` a full bgrun log. +- **Quick peek:** `bgtail ` — condensed newest lines (ANSI stripped, repeats + collapsed, ~2KB/line and ~8KB caps). The first read is the last-40-lines tail; each later + read returns only what was appended since, so repeated polling is nearly + free. The wake message itself already carries the exit code and the log's + last line, so many turns need no follow-up read at all. +- **Pattern search:** `bggrep [pattern] [context]` — line-numbered matches, + capped and condensed (~50 matches, ~2KB/line, ~8KB); works on global jobs dirs that `ctx_execute_file` + cannot reach. Pass your own pattern when you know the log's format. +- **Whole-log analysis:** `ctx_execute_file` on the job's log path (reachable + when logs are project-local) to extract only failure lines. Never `cat` or + `Read` a full bgrun log. + +**Why `bggrep` instead of `bash grep` on the log?** A bash grep's output is +uncapped — a retry-storm log can dump thousands of matching lines straight +into context, and safety depends on remembering `| head` on every call. +`bggrep` is bounded by design (~50 matches, ~2KB/line, ~8KB), takes the job id instead of +a reconstructed log path (no shell-quoting of the regex), runs on any jobs +dir — including global logs that project-sandboxed tools like +`ctx_execute_file` cannot reach — and reports match counts, line numbers, and +skip markers. Plain `grep` is fine only for a one-off search you know is tiny. ## Configuration @@ -118,11 +134,41 @@ config file (trusted projects only) ← environment variables**. } ``` +### Project-local logs + +A **relative** `jobsDir` (from any config layer, or `PI_BGRUN_DIR`) opts into +project-local logs: it resolves against the session's project root, so job logs +land inside the workspace — e.g. `"jobsDir": ".pi-bgrun/jobs"` in the project +config writes logs to `/.pi-bgrun/jobs`. + +Why you might want this: + +- Logs sit inside the project sandbox, so project-confined analysis tools + (e.g. context-mode's `ctx_execute_file` / `ctx_index`) can process whole logs + without pulling raw bytes into the context window. +- Each checkout/worktree gets its own logs — no cross-project clutter in the + shared dir. +- The dir is auto-added to the repo's `.git/info/exclude` (local-only — the + tracked `.gitignore` is never touched), so logs never pollute `git status`. + Works in linked worktrees too (`.git` file → pointed git dir). + +Rules and migration notes: + +- Absolute `jobsDir` values behave exactly as in older versions — nothing + moves, nothing breaks on upgrade. +- If the session cwd is not a recognizable project root (no `.git`/`.pi`), a + relative path falls back to the global dir rather than scattering logs + across arbitrary directories. +- Tools resolve a job's log from the session's job record first, so jobs + started before a config change stay readable after it. +- Existing logs in the old global dir are not migrated (they're ephemeral, + `cleanupDays`-retained); `bgclean all` sweeps them once you've switched. + Environment variables (same knobs, handy for one-off overrides): | Variable | Default | Description | | --- | --- | --- | -| `PI_BGRUN_DIR` | `~/.pi-bgrun/jobs` | Override where job logs are stored. | +| `PI_BGRUN_DIR` | `~/.pi-bgrun/jobs` | Override where job logs are stored. An absolute path is used as-is; a **relative** path resolves against the project root (see [project-local logs](#project-local-logs)), falling back to the default when there is no project root. | | `PI_BGRUN_FOREIGN_JOBS` | `false` | Adopt other sessions' running jobs into this session's widget and job list. Adopted jobs are polled so they leave the widget when they finish. | | `PI_BGRUN_SHOW_COMPLETED` | `false` | Include finished jobs in `bgstatus` listings by default. | | `PI_BGRUN_CLEANUP_DAYS` | `7` | Log retention for cleanup sweeps and the `bgclean` default. | diff --git a/extension/index.test.ts b/extension/index.test.ts index 484c47a..66c7a18 100644 --- a/extension/index.test.ts +++ b/extension/index.test.ts @@ -19,10 +19,12 @@ import { readFileSync, writeFileSync, existsSync, + mkdirSync, + appendFileSync, readdirSync, } from "node:fs"; import { join } from "node:path"; -import { tmpdir } from "node:os"; +import { homedir, tmpdir } from "node:os"; import { pathToFileURL } from "node:url"; interface CapturedWake { @@ -30,7 +32,13 @@ interface CapturedWake { options?: Record; } -function makeFakePi(opts: { idle?: boolean; priorEntries?: any[] } = {}): { +function makeFakePi( + opts: { + idle?: boolean; + priorEntries?: any[]; + ctxFields?: Record; + } = {}, +): { pi: any; wakes: CapturedWake[]; entries: any[]; @@ -60,6 +68,7 @@ function makeFakePi(opts: { idle?: boolean; priorEntries?: any[] } = {}): { hasUI: false, ui: { notify() {}, setWidget() {}, setStatus() {} }, sessionManager: { getEntries: () => entries }, + ...(opts.ctxFields as Record | undefined), }; const pi = { sendUserMessage(text: string, options?: Record) { @@ -1770,3 +1779,791 @@ test("formatSince: same-day shows time only; older days include the date", async assert.match(prevYearStr, /Dec 30/); assert.match(prevYearStr, /08:00:00/); }); + +// ── Project-local jobs dir ────────────────────────────────────────────────── + +test("resolveJobsDirPath: relative resolves against a project root; absolute and no-root fall back", async () => { + const mod = await import( + pathToFileURL(join(process.cwd(), "extension/index.ts")).href + ); + const proj = mkdtempSync(join(tmpdir(), "pi-bgrun-proj-")); + const scratch = mkdtempSync(join(tmpdir(), "pi-bgrun-scratch-")); + try { + mkdirSync(join(proj, ".git"), { recursive: true }); + + // absolute → used as-is, never flagged project-local (older configs keep + // working unchanged — the migration guarantee) + const absPath = join(proj, "abs-jobs"); + const abs = mod.resolveJobsDirPath(absPath, { cwd: proj }); + assert.equal(abs.dir, absPath); + assert.equal(abs.projectLocal, false); + + // relative + project root → resolved against the root, flagged project-local + const rel = mod.resolveJobsDirPath(".pi-bgrun/jobs", { cwd: proj }); + assert.equal(rel.dir, join(proj, ".pi-bgrun", "jobs")); + assert.equal(rel.projectLocal, true); + + // unset → global default + const none = mod.resolveJobsDirPath(undefined, { cwd: proj }); + assert.equal(none.dir, join(homedir(), ".pi-bgrun", "jobs")); + assert.equal(none.projectLocal, false); + + // relative + cwd that is not a project → global fallback, never cwd-relative + const fb = mod.resolveJobsDirPath(".pi-bgrun/jobs", { cwd: scratch }); + assert.equal(fb.dir, join(homedir(), ".pi-bgrun", "jobs")); + assert.equal(fb.projectLocal, false); + } finally { + rmSync(proj, { recursive: true, force: true }); + rmSync(scratch, { recursive: true, force: true }); + } +}); + +test("ensureGitExcluded: appends the jobs dir pattern to .git/info/exclude once per dir", async () => { + const mod = await import( + pathToFileURL(join(process.cwd(), "extension/index.ts")).href + ); + const repo = mkdtempSync(join(tmpdir(), "pi-bgrun-repo-")); + try { + mkdirSync(join(repo, ".git", "info"), { recursive: true }); + mod.ensureGitExcluded(join(repo, ".pi-bgrun", "jobs")); + mod.ensureGitExcluded(join(repo, ".pi-bgrun", "jobs")); + // a second, different jobs dir under the same repo adds its own pattern + mod.ensureGitExcluded(join(repo, ".pi-bgrun", "other")); + const exclude = readFileSync(join(repo, ".git", "info", "exclude"), "utf8"); + assert.match(exclude, /# pi-bgrun job logs/); + assert.equal( + exclude.split("\n").filter((l) => l.trim() === ".pi-bgrun/jobs/").length, + 1, + "pattern appears exactly once", + ); + assert.ok(exclude.split("\n").includes(".pi-bgrun/other/")); + } finally { + rmSync(repo, { recursive: true, force: true }); + } +}); + +test("ensureGitExcluded: linked worktree (.git file) writes to the pointed git dir", async () => { + const mod = await import( + pathToFileURL(join(process.cwd(), "extension/index.ts")).href + ); + const wt = mkdtempSync(join(tmpdir(), "pi-bgrun-wt-")); + const gd = mkdtempSync(join(tmpdir(), "pi-bgrun-gitdir-")); + try { + writeFileSync(join(wt, ".git"), `gitdir: ${gd}\n`); + mod.ensureGitExcluded(join(wt, ".pi-bgrun", "jobs")); + const exclude = readFileSync(join(gd, "info", "exclude"), "utf8"); + assert.match(exclude, /^\.pi-bgrun\/jobs\/$/m); + // nothing was created inside the worktree's own .git (it's a file) + assert.ok(!existsSync(join(wt, ".git", "info"))); + } finally { + rmSync(wt, { recursive: true, force: true }); + rmSync(gd, { recursive: true, force: true }); + } +}); + +test("ensureGitExcluded: gitdir pointer with spaces in the path", async () => { + const mod = await import( + pathToFileURL(join(process.cwd(), "extension/index.ts")).href + ); + const wt = mkdtempSync(join(tmpdir(), "pi-bgrun-wt-")); + const gd = join(tmpdir(), "pi-bgrun git dir with spaces"); + mkdirSync(gd, { recursive: true }); + try { + writeFileSync(join(wt, ".git"), `gitdir: ${gd}\n`); + assert.equal(mod.ensureGitExcluded(join(wt, ".pi-bgrun", "jobs")), true); + const exclude = readFileSync(join(gd, "info", "exclude"), "utf8"); + assert.match(exclude, /^\.pi-bgrun\/jobs\/$/m); + } finally { + rmSync(wt, { recursive: true, force: true }); + rmSync(gd, { recursive: true, force: true }); + } +}); + +test("ensureGitExcluded: retries after a transient failure — memoizes only on success", async () => { + const mod = await import( + pathToFileURL(join(process.cwd(), "extension/index.ts")).href + ); + const repo = mkdtempSync(join(tmpdir(), "pi-bgrun-repo-")); + try { + mkdirSync(join(repo, ".git", "info"), { recursive: true }); + // Block the exclude path with a directory → the append fails (EISDIR) + mkdirSync(join(repo, ".git", "info", "exclude")); + const jobsDir = join(repo, ".pi-bgrun", "jobs"); + assert.equal(mod.ensureGitExcluded(jobsDir), false); + + // Unblock: the next call must retry (failure was not memoized) and succeed + rmSync(join(repo, ".git", "info", "exclude"), { recursive: true }); + assert.equal(mod.ensureGitExcluded(jobsDir), true); + const exclude = readFileSync(join(repo, ".git", "info", "exclude"), "utf8"); + assert.match(exclude, /^\.pi-bgrun\/jobs\/$/m); + } finally { + rmSync(repo, { recursive: true, force: true }); + } +}); + +test("bgrun: relative jobsDir in project config → project-local log + auto git-exclude", async () => { + const proj = mkdtempSync(join(tmpdir(), "pi-bgrun-proj-")); + delete process.env.PI_BGRUN_DIR; + try { + mkdirSync(join(proj, ".git"), { recursive: true }); + mkdirSync(join(proj, ".pi"), { recursive: true }); + writeFileSync( + join(proj, ".pi", "pi-bgrun.json"), + JSON.stringify({ jobsDir: ".pi-bgrun/jobs" }), + ); + const { pi, wakes, tools, ctx } = makeFakePi({ + ctxFields: { cwd: proj, isProjectTrusted: () => true }, + }); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + + const res = await bgrun.execute( + "call-1", + { command: "echo project-local" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + assert.ok(id, "got a job id"); + + await waitForWakes(wakes, 1); + + const logPath = join(proj, ".pi-bgrun", "jobs", `${id}.log`); + assert.ok(existsSync(logPath), "log written inside the project"); + assert.match(readFileSync(logPath, "utf8"), /project-local/); + + const exclude = join(proj, ".git", "info", "exclude"); + assert.ok(existsSync(exclude), "exclude file created"); + assert.match(readFileSync(exclude, "utf8"), /^\.pi-bgrun\/jobs\/$/m); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(proj, { recursive: true, force: true }); + } +}); + +test("bgtail: prefers the session record's logPath when the jobsDir config changes", async () => { + const proj = mkdtempSync(join(tmpdir(), "pi-bgrun-proj-")); + delete process.env.PI_BGRUN_DIR; + try { + mkdirSync(join(proj, ".git"), { recursive: true }); + mkdirSync(join(proj, ".pi"), { recursive: true }); + writeFileSync( + join(proj, ".pi", "pi-bgrun.json"), + JSON.stringify({ jobsDir: ".pi-bgrun/jobs" }), + ); + const { pi, wakes, tools, ctx } = makeFakePi({ + ctxFields: { cwd: proj, isProjectTrusted: () => true }, + }); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + + const res = await bgrun.execute( + "call-1", + { command: "echo migrated-log" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + assert.ok(id, "got a job id"); + await waitForWakes(wakes, 1); + + // A ctx with no project config/trust now resolves the jobs dir to the + // GLOBAL default — only the session record's logPath can still find the + // log (the mid-upgrade config-change scenario). + const plainCtx = { ...ctx, cwd: undefined, isProjectTrusted: undefined }; + const tail = await bgtail.execute( + "call-2", + { id, lines: 10 }, + undefined, + undefined, + plainCtx, + ); + assert.equal(tail.details.notFound, false); + assert.match(tail.content[0].text as string, /migrated-log/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(proj, { recursive: true, force: true }); + } +}); + +// ── bggrep ────────────────────────────────────────────────────────────────── + +test("bggrep: line-numbered matches; explicit pattern wins; default pattern; no-match case", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + + const res = await bgrun.execute( + "c1", + { command: "printf 'alpha\\nerror: boom BANANA\\nomega\\n'" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + assert.ok(id, "got a job id"); + await waitForWakes(wakes, 1); + + // explicit pattern → only matching lines, with line numbers + const g = await bggrep.execute( + "c2", + { id, pattern: "BANANA" }, + undefined, + undefined, + ctx, + ); + assert.equal(g.details.matches, 1); + assert.equal(g.details.notFound, false); + assert.match(g.content[0].text as string, /L2: error: boom BANANA/); + assert.doesNotMatch(g.content[0].text as string, /alpha|omega/); + + // default pattern (no pattern passed) catches the failure signature + const g2 = await bggrep.execute("c3", { id }, undefined, undefined, ctx); + assert.equal(g2.details.matches, 1); + assert.match(g2.content[0].text as string, /1 match for \//); + assert.equal( + g2.details.pattern, + "--- FAIL:|^FAIL\\b|^panic:|fatal error:|AssertionError|Error:|error:|make: \\*\\*\\*.*Error|✗|✖", + ); + + // a log with no failure signatures → clean no-match (not an error) + const res2 = await bgrun.execute( + "c4", + { command: "echo all clear, nothing to see" }, + undefined, + undefined, + ctx, + ); + const id2 = ((res2.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 2); + const g3 = await bggrep.execute( + "c5", + { id: id2 }, + undefined, + undefined, + ctx, + ); + assert.equal(g3.details.matches, 0); + assert.equal(g3.isError, undefined); + assert.match(g3.content[0].text as string, /— none/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: context lines with gap markers between distant matches", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + + const res = await bgrun.execute( + "c1", + { + command: + "printf 'l1\\nMATCH one\\nl3\\nl4\\nl5\\nl6\\nl7\\nMATCH two\\nl9\\n'", + }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + assert.ok(id, "got a job id"); + await waitForWakes(wakes, 1); + + const g = await bggrep.execute( + "c2", + { id, pattern: "MATCH", context: 1 }, + undefined, + undefined, + ctx, + ); + assert.equal(g.details.matches, 2); + const text = g.content[0].text as string; + assert.match(text, /L2: MATCH one/); + assert.match(text, /L1: l1/); // context before + assert.match(text, /L8: MATCH two/); + assert.match(text, /L9: l9/); // context after + assert.match(text, /…\[3 lines skipped\]…/); // l4-l6 between the windows + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: invalid pattern errors clearly", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo hi" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + await assert.rejects( + bggrep.execute( + "c2", + { id, pattern: "([unclosed" }, + undefined, + undefined, + ctx, + ), + /bggrep: invalid pattern/, + ); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: caps at 50 matches with a not-shown note", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: 'for i in $(seq 1 60); do echo "boom $i"; done' }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const g = await bggrep.execute( + "c2", + { id, pattern: "boom" }, + undefined, + undefined, + ctx, + ); + assert.equal(g.details.matches, 60); + assert.equal(g.details.capped, true); + assert.match( + g.content[0].text as string, + /showing first 50; 10 more not shown/, + ); + assert.match(g.content[0].text as string, /L50: boom 50/); + assert.doesNotMatch(g.content[0].text as string, /L51: boom 51/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: prefers the session record's logPath when the jobsDir config changes", async () => { + const proj = mkdtempSync(join(tmpdir(), "pi-bgrun-proj-")); + delete process.env.PI_BGRUN_DIR; + try { + mkdirSync(join(proj, ".git"), { recursive: true }); + mkdirSync(join(proj, ".pi"), { recursive: true }); + writeFileSync( + join(proj, ".pi", "pi-bgrun.json"), + JSON.stringify({ jobsDir: ".pi-bgrun/jobs" }), + ); + const { pi, wakes, tools, ctx } = makeFakePi({ + ctxFields: { cwd: proj, isProjectTrusted: () => true }, + }); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo pattern-target-line" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + + const plainCtx = { ...ctx, cwd: undefined, isProjectTrusted: undefined }; + const g = await bggrep.execute( + "c2", + { id, pattern: "pattern-target" }, + undefined, + undefined, + plainCtx, + ); + assert.equal(g.details.notFound, false); + assert.equal(g.details.matches, 1); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(proj, { recursive: true, force: true }); + } +}); + +// ── bgtail delta tailing ──────────────────────────────────────────────────── + +test("bgtail: delta tailing — first read full tail, then only new lines, then none", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const res = await bgrun.execute( + "c1", + { command: "echo first line" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + assert.ok(id, "got a job id"); + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + // First read: full tail, no delta header + const t1 = await bgtail.execute("c2", { id }, undefined, undefined, ctx); + assert.match(t1.content[0].text as string, /first line/); + assert.equal(t1.details.newLines, undefined); + assert.doesNotMatch( + t1.content[0].text as string, + /new lines since last read/, + ); + + // Log grows: only the new lines come back, with a +N header + appendFileSync(logPath, "appended-A\nappended-B\n"); + const t2 = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + const text2 = t2.content[0].text as string; + assert.match(text2, /\+2 new lines since last read/); + assert.match(text2, /appended-A/); + assert.match(text2, /appended-B/); + assert.doesNotMatch(text2, /first line/); + assert.equal(t2.details.newLines, 2); + + // Nothing new: a tiny no-new-lines response (cheap polling) + const t3 = await bgtail.execute("c4", { id }, undefined, undefined, ctx); + assert.match(t3.content[0].text as string, /no new lines since last read/); + assert.equal(t3.details.linesShown, 0); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bgtail: raw:true keeps the verbatim window but still advances the bookmark", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const res = await bgrun.execute( + "c1", + { command: "echo baseline" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + appendFileSync(logPath, "post-raw line\n"); + const r = await bgtail.execute( + "c2", + { id, lines: 3, raw: true }, + undefined, + undefined, + ctx, + ); + assert.match(r.content[0].text as string, /post-raw line/); + assert.equal(r.details.condensed, false); + + // The raw read advanced the bookmark → the next condensed read is empty + const t = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + assert.match(t.content[0].text as string, /no new lines since last read/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bgtail: a shrunken log resets to a full tail with a note", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const res = await bgrun.execute( + "c1", + { command: "echo long original content line" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + // First read sets the bookmark; then the log is replaced by a shorter one + await bgtail.execute("c2", { id }, undefined, undefined, ctx); + writeFileSync(logPath, "tiny replacement\n"); + const t = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + const text = t.content[0].text as string; + assert.match(text, /log shrank since last read — showing full tail/); + assert.match(text, /tiny replacement/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bgtail: a replaced log with the same line count resets to a full tail", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const res = await bgrun.execute( + "c1", + { command: "printf 'aaaa\\nbbbb\\ncccc\\n'" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + // First read sets the bookmark (3 content lines, first line "aaaa") + await bgtail.execute("c2", { id }, undefined, undefined, ctx); + // Replacement: SAME line count, LARGER byte size (so the shrink checks + // cannot fire), different first line — only the first-line detector + // (append-only logs never mutate line 0) can catch this. + writeFileSync( + logPath, + "xxxxxxxxxxxxxxxxxx\nyyyyyyyyyyyyyyyyyy\nzzzzzzzzzzzzzzzzzz\n", + ); + const t = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + const text = t.content[0].text as string; + assert.match(text, /log was replaced since last read — showing full tail/); + assert.match(text, /xxxxxxxxxxxxxxxxxx/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep and bgtail normalize CRLF logs", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo something" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + writeFileSync(logPath, "alpha\r\nerror: boom\r\nomega\r\n"); + // A $-anchored pattern must match despite the CRLF source + const g = await bggrep.execute( + "c2", + { id, pattern: "boom$" }, + undefined, + undefined, + ctx, + ); + assert.match(g.content[0].text as string, /L2: error: boom/); + // And no stray \r leaks into either tool's output + assert.ok(!(g.content[0].text as string).includes("\r")); + const t = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + assert.ok(!(t.content[0].text as string).includes("\r")); + assert.match(t.content[0].text as string, /error: boom/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: empty log reports zero lines, and a missing log is notFound", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo x" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + + writeFileSync(join(dir, `${id}.log`), ""); + const g = await bggrep.execute( + "c2", + { id, pattern: "Error:" }, + undefined, + undefined, + ctx, + ); + assert.match( + g.content[0].text as string, + /0 matches for \/Error:\/ in 0 lines — none/, + ); + + const missing = await bggrep.execute( + "c3", + { id: "no-such-job-123", pattern: "x" }, + undefined, + undefined, + ctx, + ); + assert.equal(missing.isError, true); + assert.equal(missing.details.notFound, true); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: context windows combine with the 50-match cap", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo x" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + + // 240 lines, a hit every 4th line → 60 matches (cap 50); with context: 1 + // each window is [i-1, i+1] and consecutive windows leave a 1-line gap. + const lines: string[] = []; + for (let i = 1; i <= 240; i++) { + lines.push(i % 4 === 0 ? `hit ${i}` : `filler ${i}`); + } + writeFileSync(join(dir, `${id}.log`), lines.join("\n") + "\n"); + const r = await bggrep.execute( + "c2", + { id, pattern: "^hit", context: 1 }, + undefined, + undefined, + ctx, + ); + const text = r.content[0].text as string; + assert.equal(r.details.matches, 60); + assert.equal(r.details.capped, true); + assert.match(text, /showing first 50; 10 more not shown/); + assert.match(text, /L4: hit 4/); + assert.match(text, /…\[1 line skipped\]…/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bgtail and bggrep clamp nonsensical numeric params", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "printf 'one\\ntwo\\nthree\\nfour\\nfive\\n'" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + + // lines: 0 must not mean "everything" (slice(-0) pitfall) — clamps to 1 + const t = await bgtail.execute( + "c2", + { id, lines: 0 }, + undefined, + undefined, + ctx, + ); + assert.equal(t.details.linesShown, 1); + assert.match(t.content[0].text as string, /five/); + assert.ok(!(t.content[0].text as string).includes("four")); + + // negative context must not drop the match lines themselves — clamps to 0 + const g = await bggrep.execute( + "c3", + { id, pattern: "^three", context: -1 }, + undefined, + undefined, + ctx, + ); + assert.match(g.content[0].text as string, /L3: three/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/extension/index.ts b/extension/index.ts index dd3a559..d83817f 100644 --- a/extension/index.ts +++ b/extension/index.ts @@ -34,17 +34,19 @@ import { Type } from "typebox"; import { Box, Text } from "@earendil-works/pi-tui"; import { spawn } from "node:child_process"; import { - openSync, + appendFileSync, closeSync, - readFileSync, + existsSync, mkdirSync, + openSync, + readFileSync, readdirSync, renameSync, - unlinkSync, statSync, + unlinkSync, writeFileSync, } from "node:fs"; -import { join } from "node:path"; +import { dirname, isAbsolute, join, relative, sep } from "node:path"; import { homedir } from "node:os"; // Exit marker appended to every log so the file is self-describing: the exit @@ -54,6 +56,14 @@ const EXIT_MARKER = "__BGRUN_EXIT__="; const DEFAULT_CLEANUP_DAYS = 7; const STALE_POLL_MS = 30_000; // re-check interval for jobs with no live child handle +const GLOBAL_JOBS_DIR = join(homedir(), ".pi-bgrun", "jobs"); + +// Default regex for bggrep when the caller passes no pattern: common failure +// signatures across test runners and build tools. ONLY a convenience default — +// bggrep's contract is that the caller's own pattern always wins, because a +// generic default on arbitrary tools/languages misses more than it catches. +export const DEFAULT_GREP_PATTERN = + "--- FAIL:|^FAIL\\b|^panic:|fatal error:|AssertionError|Error:|error:|make: \\*\\*\\*.*Error|✗|✖"; // ── Configuration ─────────────────────────────────────────────────────────── // @@ -66,6 +76,10 @@ const STALE_POLL_MS = 30_000; // re-check interval for jobs with no live child h interface BgrunConfig { jobsDir: string; + // True when jobsDir came from a RELATIVE path resolved against the project + // root (project-local logs). Only then does bgrun auto-ignore the dir in + // .git/info/exclude — an absolute dir is the user's explicit choice. + jobsDirProjectLocal: boolean; // Adopt other sessions' running jobs (found in the shared jobs dir) into // this session's widget and job list. Default false — most sessions don't // want unrelated jobs from other projects cluttering the widget. @@ -112,6 +126,111 @@ function readConfigFile(path: string): BgrunConfigFile { return {}; } +// ── Project-local jobs dir ────────────────────────────────────────────────── +// +// A RELATIVE `jobsDir` (from any config layer, or PI_BGRUN_DIR) opts into +// project-local logs: it resolves against the session's project root, so logs +// land inside the workspace. That keeps them within the project sandbox — +// analysis tools confined to the project root (e.g. context-mode's +// ctx_execute_file/ctx_index) can then process whole logs without flooding +// context. Absolute paths behave exactly as in older versions +// (migration-safe), and with no recognizable project root a relative path +// falls back to the global dir instead of scattering logs across whatever +// directory pi happened to start in. + +function isProjectRootLike(dir: string): boolean { + // Cheap heuristic: a directory holding .git or pi's config dir is a project. + return ( + existsSync(join(dir, ".git")) || existsSync(join(dir, CONFIG_DIR_NAME)) + ); +} + +export function resolveJobsDirPath( + raw: string | undefined, + ctx?: { cwd?: string }, +): { dir: string; projectLocal: boolean } { + if (!raw) return { dir: GLOBAL_JOBS_DIR, projectLocal: false }; + if (isAbsolute(raw)) return { dir: raw, projectLocal: false }; + const root = ctx?.cwd ?? process.cwd(); + if (!root || !isProjectRootLike(root)) { + return { dir: GLOBAL_JOBS_DIR, projectLocal: false }; + } + return { dir: join(root, raw), projectLocal: true }; +} + +// Auto-ignore a project-local jobs dir in git so logs never pollute +// `git status`: appends the dir pattern to the enclosing repo's +// .git/info/exclude (local-only — the tracked .gitignore is never touched). +// Memoized only on SUCCESS — a transient failure (unwritable exclude file, +// .git appearing later) is retried on the next bgrun. Every step is +// best-effort and must never fail a bgrun. +const gitExcludedDirs = new Set(); + +// Returns true when the dir is settled (pattern written, already present, or +// legitimately nothing to do — no repo above, dir is the repo root itself). +// False only on failure, so the caller retries next time. +export function ensureGitExcluded(jobsDir: string): boolean { + if (gitExcludedDirs.has(jobsDir)) return true; + if (tryEnsureGitExcluded(jobsDir)) { + gitExcludedDirs.add(jobsDir); + return true; + } + return false; +} + +function tryEnsureGitExcluded(jobsDir: string): boolean { + try { + // Walk up from jobsDir to the enclosing work tree. + let cur = jobsDir; + for (;;) { + const dot = join(cur, ".git"); + if (existsSync(dot)) return appendExcludePattern(cur, dot, jobsDir); + const parent = dirname(cur); + if (parent === cur) return true; // filesystem root — no repo above; nothing to do + cur = parent; + } + } catch { + // best-effort — ignore hygiene must never break job creation + return false; + } +} + +function appendExcludePattern( + repoRoot: string, + dotGit: string, + jobsDir: string, +): boolean { + if (jobsDir === repoRoot) return true; // can't exclude the whole repo; nothing to do + // `.git` is a directory in a normal checkout, or a file pointing at the + // real git dir in linked worktrees (git worktree add) and submodules. + let gitDir = dotGit; + if (statSync(dotGit).isFile()) { + const m = readFileSync(dotGit, "utf8").match(/^gitdir:\s*(.+)$/m); + if (!m) return false; // unparseable .git file — retry later + gitDir = m[1].trim(); + } + const rel = relative(repoRoot, jobsDir); + // Defense-in-depth: the walk-up guarantees jobsDir sits under repoRoot, but + // a future caller or symlinked path could break that — ../-prefixed + // patterns are silently useless in gitignore semantics, so skip them. + if (rel.startsWith("..") || isAbsolute(rel)) return true; + const pattern = rel.split(sep).join("/") + "/"; + const excludePath = join(gitDir, "info", "exclude"); + let existing = ""; + try { + existing = readFileSync(excludePath, "utf8"); + } catch { + // no exclude file yet — we'll create it + } + if (existing.split("\n").some((l) => l.trim() === pattern)) return true; + mkdirSync(join(gitDir, "info"), { recursive: true }); + appendFileSync( + excludePath, + `\n# pi-bgrun job logs (auto-added)\n${pattern}\n`, + ); + return true; +} + // Resolved per call (cheap: at most two small file reads) so env/config // changes are picked up without module reloads — and tests can isolate. function resolveConfig(ctx?: { @@ -154,11 +273,11 @@ function resolveConfig(ctx?: { : undefined; const envDays = Number(process.env.PI_BGRUN_CLEANUP_DAYS); const daysEnv = Number.isFinite(envDays) && envDays > 0 ? envDays : undefined; + const { dir: jobsDir, projectLocal: jobsDirProjectLocal } = + resolveJobsDirPath(process.env.PI_BGRUN_DIR || dirFile, ctx); return { - jobsDir: - process.env.PI_BGRUN_DIR || - dirFile || - join(homedir(), ".pi-bgrun", "jobs"), + jobsDir, + jobsDirProjectLocal, adoptForeignJobs: parseBoolEnv(process.env.PI_BGRUN_FOREIGN_JOBS) ?? foreignFile ?? false, showCompletedJobs: @@ -711,7 +830,7 @@ export default function (pi: ExtensionAPI) { "Use bgrun (not bash) for any command expected to run >30s or emit >100 lines — tests, builds, linters.", "Give every bgrun job a short name (e.g. name: 'unit-tests') so it's recognizable in status output, the status widget, and wake messages.", "After bgrun returns a job id, continue other work; you will be woken automatically when it finishes.", - "Never cat or Read a full bgrun log — bgtail returns a condensed peek (ANSI stripped, repeats collapsed, ~8KB cap); use ctx_execute_file on the log path only when the condensed tail is insufficient.", + "Never cat or Read a full bgrun log — bgtail returns a condensed peek (ANSI stripped, repeats collapsed, ~8KB cap); use bggrep for pattern search or ctx_execute_file on the log path for whole-log analysis.", ], parameters: Type.Object({ command: Type.String({ @@ -733,7 +852,11 @@ export default function (pi: ExtensionAPI) { } const name = sanitizeName(rawName); - const jobsDir = resolveConfig(ctx).jobsDir; + const cfg = resolveConfig(ctx); + // Project-local logs are auto-ignored in .git/info/exclude (best-effort) + // so they never pollute `git status`. Absolute dirs are left untouched. + if (cfg.jobsDirProjectLocal) ensureGitExcluded(cfg.jobsDir); + const jobsDir = cfg.jobsDir; mkdirSync(jobsDir, { recursive: true }); const slug = makeSlug(name ?? command); @@ -948,7 +1071,22 @@ export default function (pi: ExtensionAPI) { return { text: out.join("\n"), truncated: notes }; } - // ── bgtail: read last N lines of a job's log, condensed for context ──────── + // ── bgtail: read the newest lines of a job's log, condensed for context ──── + // + // Delta tailing: each read bookmarks the total raw line count at read time + // (the high-water mark of what the caller has had the opportunity to see). + // The FIRST read for a job returns the full last-N tail; repeat reads return + // only lines appended since, so polling a running job never re-pays context + // for lines already seen. Deliberately-skipped prefix lines are never + // replayed as "new". raw: true keeps the verbatim last-N window (no delta + // header) but still advances the bookmark. A shrunken log (rotated/replaced) + // resets to a full tail. Bookmarks are in-memory only — a session restart + // starts fresh with a full tail. + + const tailBookmarks = new Map< + string, + { lines: number; bytes: number; first: string } + >(); // Shared by the bgtail tool (agent-facing) and the /bgtail slash command // (human-facing). @@ -960,25 +1098,101 @@ export default function (pi: ExtensionAPI) { details: Record; isError?: boolean; }> { - const { id, lines = 40, raw = false } = params; + const { id, lines: linesParam = 40, raw = false } = params; + // Clamp defensively — direct callers (e.g. the slash command) bypass the + // tool schema, and lines < 1 would corrupt slicing (slice(-0) = whole log). + const lines = Math.max(1, Math.floor(linesParam)); if (!id) throw new Error("bgtail: id is required"); - const logPath = join(resolveConfig(ctx).jobsDir, `${id}.log`); + // Prefer this session's record: its logPath stays correct even if the + // config (and thus the resolved jobs dir) changes mid-session — e.g. a + // user switching to project-local logs right after upgrading. + const logPath = + jobs.get(id)?.logPath ?? join(resolveConfig(ctx).jobsDir, `${id}.log`); try { const content = readFileSync(logPath, "utf8"); - const all = content - .split("\n") + // Content lines only: the exit marker and blanks are filtered BEFORE the + // window is sliced, so "last N lines" means the last N content lines + // (matching pre-delta behavior) and bookmarks count content lines. + // /\r?\n/ keeps CRLF logs from leaving a stray \r on every line. + const rawLines = content + .split(/\r?\n/) .filter((l) => !l.startsWith(EXIT_MARKER) && l.trim().length > 0); - const tail = all.slice(-lines); - const { text, truncated } = condenseLogLines(tail, { raw }); + const total = rawLines.length; + const first = rawLines[0]?.slice(0, 200) ?? ""; + const prev = tailBookmarks.get(id); + // Append-only logs never mutate earlier lines, so a changed first + // content line means the log was replaced or rotated — reset to a full + // tail. Catches same-size replacements the shrink checks cannot see. + // (A previously-empty log growing content is growth, not replacement.) + const replaced = + prev !== undefined && prev.lines > 0 && prev.first !== first; + const shrank = + prev !== undefined && + (prev.lines > total || prev.bytes > content.length); + let window: string[]; + let header: string | undefined; + let newLines: number | undefined; + if (raw || prev === undefined || shrank || replaced) { + // Full tail: first read, raw mode, or a shrunken/replaced log (reset). + window = rawLines.slice(-lines); + if (!raw && (shrank || replaced)) { + header = shrank + ? "log shrank since last read — showing full tail" + : "log was replaced since last read — showing full tail"; + } + } else { + const fresh = rawLines.slice(prev.lines); + newLines = fresh.length; + if (fresh.length === 0) { + tailBookmarks.set(id, { + lines: total, + bytes: content.length, + first, + }); + return { + content: [ + { + type: "text", + text: `(no new lines since last read — log at ${total} line${total === 1 ? "" : "s"})`, + }, + ], + details: { + id, + linesShown: 0, + logPath, + notFound: false, + condensed: true, + newLines: 0, + totalLines: total, + }, + }; + } + window = fresh.length > lines ? fresh.slice(-lines) : fresh; + header = + `+${fresh.length} new line${fresh.length === 1 ? "" : "s"} since last read — ` + + `log at ${total} lines${fresh.length > lines ? ` (showing last ${lines})` : ""}`; + } + tailBookmarks.set(id, { + lines: total, + bytes: content.length, + first, + }); + const shown = window; + const { text, truncated } = condenseLogLines(shown, { raw }); + // Delta reads early-return above, so an empty window here can only be + // a first read of an empty log (full-tail path). + const body = shown.length === 0 ? "(empty log)" : text; const notes = truncated.length > 0 ? `\n\n(${truncated.join("; ")})` : ""; + const head = header ? `${header}\n` : ""; return { - content: [{ type: "text", text: text + notes || "(empty log)" }], + content: [{ type: "text", text: head + body + notes }], details: { id, - linesShown: tail.length, + linesShown: shown.length, logPath, notFound: false, condensed: !raw, + ...(newLines === undefined ? {} : { newLines, totalLines: total }), ...(truncated.length > 0 ? { condenserNotes: truncated } : {}), }, }; @@ -997,14 +1211,17 @@ export default function (pi: ExtensionAPI) { name: "bgtail", label: "Tail Background Log", description: - "Print the last N lines of a background job's log (default 40), condensed for context: ANSI escapes stripped, repeated lines collapsed, long lines truncated, output capped (~8KB). Strips the exit-marker line. Pass raw: true for unprocessed output; use ctx_execute_file on the log path for whole-log failure analysis.", + "Read the newest lines of a background job's log, condensed for context: ANSI escapes stripped, repeated lines collapsed, long lines truncated, output capped (~8KB). Strips the exit-marker line. The first read returns the last N lines (default 40); REPEAT reads return only lines appended since your last read (delta tailing) — polling a running job never re-pays for the same lines. raw: true returns the unprocessed last-N window. A shrunken or replaced log resets to a full tail. For pattern search use bggrep; for whole-log analysis, ctx_execute_file on the log path.", promptSnippet: "Read the last N lines of a bgrun job's log", parameters: Type.Object({ id: Type.String({ description: "Job id (from bgrun's 'started: ' response)", }), lines: Type.Optional( - Type.Number({ description: "Number of lines to show (default 40)" }), + Type.Number({ + description: "Number of lines to show (default 40)", + minimum: 1, + }), ), raw: Type.Optional( Type.Boolean({ @@ -1018,6 +1235,152 @@ export default function (pi: ExtensionAPI) { }, }); + // ── bggrep: pattern search over a job's log, capped for context ─────────── + // + // The sandboxed whole-log path (ctx_execute_file) is confined to the + // project root, which a global jobs dir sits outside of — bggrep runs + // inside the extension with native fs access, so it works on any + // configured jobs dir. Matches are line-numbered (grep -n style), + // optionally with context lines, capped at MAX_GREP_MATCHES, and run + // through the same condenser as bgtail so a search can never flood context. + + const MAX_GREP_MATCHES = 50; + + async function bggrepCore( + params: { id: string; pattern?: string; context?: number }, + ctx?: ExtensionContext, + ): Promise<{ + content: { type: "text"; text: string }[]; + details: Record; + isError?: boolean; + }> { + const { id, pattern, context: contextParam = 0 } = params; + // Clamp defensively — negative context would exclude the match lines + // themselves from the context windows (lo > hi no-ops the inner loop). + const context = Math.max(0, Math.floor(contextParam)); + if (!id) throw new Error("bggrep: id is required"); + // Record-first, same as bgtail — correct across config changes. + const logPath = + jobs.get(id)?.logPath ?? join(resolveConfig(ctx).jobsDir, `${id}.log`); + const source = pattern ?? DEFAULT_GREP_PATTERN; + let re: RegExp; + try { + re = new RegExp(source); + } catch (err) { + throw new Error( + `bggrep: invalid pattern ${JSON.stringify(source)}: ${(err as Error).message}`, + ); + } + let rawLines: string[]; + try { + const content = readFileSync(logPath, "utf8"); + // /\r?\n/ normalizes CRLF (a trailing \r would break $-anchored patterns + // and leak into output); blank lines are KEPT so L numbers match the + // file. A trailing empty split element is dropped; "" yields zero lines. + const split = content === "" ? [] : content.split(/\r?\n/); + if (split.length > 0 && split[split.length - 1] === "") split.pop(); + rawLines = split.filter((l) => !l.startsWith(EXIT_MARKER)); + } catch { + return { + content: [ + { type: "text", text: `No log found for job ${id} at ${logPath}` }, + ], + details: { id, matches: 0, logPath, notFound: true }, + isError: true, + }; + } + const matchIdx: number[] = []; + for (let i = 0; i < rawLines.length; i++) { + if (re.test(rawLines[i])) matchIdx.push(i); + } + const header = + `${matchIdx.length} match${matchIdx.length === 1 ? "" : "es"} for /${source}/ ` + + `in ${rawLines.length} line${rawLines.length === 1 ? "" : "s"}`; + if (matchIdx.length === 0) { + return { + content: [{ type: "text", text: `${header} — none` }], + details: { + id, + matches: 0, + linesSearched: rawLines.length, + logPath, + notFound: false, + }, + }; + } + const capped = matchIdx.length > MAX_GREP_MATCHES; + const shownIdx = capped ? matchIdx.slice(0, MAX_GREP_MATCHES) : matchIdx; + // Context windows, merged where they overlap or touch (grep -C style). + const include = new Set(); + for (const i of shownIdx) { + const lo = Math.max(0, i - context); + const hi = Math.min(rawLines.length - 1, i + context); + for (let j = lo; j <= hi; j++) include.add(j); + } + const sorted = [...include].sort((a, b) => a - b); + const out: string[] = []; + let prev = -2; + for (const i of sorted) { + if (prev >= 0 && i > prev + 1) { + const gap = i - prev - 1; + out.push(`…[${gap} line${gap === 1 ? "" : "s"} skipped]…`); + } + out.push(`L${i + 1}: ${rawLines[i]}`); + prev = i; + } + const { text, truncated } = condenseLogLines(out); + const notes = truncated.length > 0 ? `\n\n(${truncated.join("; ")})` : ""; + const capNote = capped + ? ` — showing first ${MAX_GREP_MATCHES}; ${matchIdx.length - MAX_GREP_MATCHES} more not shown` + : ""; + return { + content: [{ type: "text", text: `${header}${capNote}\n${text}${notes}` }], + details: { + id, + matches: matchIdx.length, + linesSearched: rawLines.length, + logPath, + notFound: false, + pattern: source, + capped, + }, + }; + } + + pi.registerTool({ + name: "bggrep", + label: "Grep Background Log", + description: + "Search a background job's log with a regex; returns only matching lines with line numbers (optional context lines), capped (~50 matches, ~8KB) and condensed. Runs inside the extension, so it works on any jobs dir — including global logs that project-sandboxed tools (ctx_execute_file) cannot reach. Pass your own pattern whenever you know the log's format; with no pattern a generic failure-signature default is used (a convenience only — not a guarantee).", + promptSnippet: "Search a bgrun job's log for a pattern", + promptGuidelines: [ + "Never search a bgrun log with the bash tool — uncapped output can flood context, and it needs manual log-path reconstruction and regex shell-quoting; bggrep is bounded by design.", + "Prefer bggrep over bash grep or reading a bgrun log — matches are line-numbered, capped, and condensed.", + "Pass an explicit pattern when you know the tool's output format; the default only catches common failure signatures.", + ], + parameters: Type.Object({ + id: Type.String({ + description: "Job id (from bgrun's 'started: ' response)", + }), + pattern: Type.Optional( + Type.String({ + description: + "Regex to search for. Default: generic failure signatures — override when you know the format.", + }), + ), + context: Type.Optional( + Type.Number({ + description: + "Context lines around each match (default 0, grep -C style)", + minimum: 0, + }), + ), + }), + async execute(_toolCallId, params, _signal, _onUpdate, ctx) { + return bggrepCore(params, ctx); + }, + }); + // ── bgstatus: list jobs (in-memory while alive; dir scan after restart) ───── // Shared by the bgstatus tool (agent-facing) and the /bgstatus slash command diff --git a/skill/run-bg/SKILL.md b/skill/run-bg/SKILL.md index 203c807..18c81fa 100644 --- a/skill/run-bg/SKILL.md +++ b/skill/run-bg/SKILL.md @@ -30,7 +30,8 @@ no polling. |---|---| | Start | `bgrun(command: "make test-short", name: "unit-tests")` → `started: ` (name is an optional short label; use it so jobs are recognizable in `bgstatus`, the status widget, and wake messages) | | Status | `bgstatus()` for one job, or `bgstatus()` for this session's running jobs — finished jobs are hidden by default; pass `includeDone: true` to list them | -| Tail | `bgtail(, 40)` | +| Tail | `bgtail(, 40)` — first read: last-40 tail; later reads: only lines appended since (delta tailing) | +| Grep | `bggrep(, "pattern", context?)` — line-numbered matches, capped and condensed; default pattern = generic failure signatures (override when you know the format) | | Clean | `bgclean()` for this session's old logs; `bgclean all` to sweep every session's (default 7-day retention) | ## Workflow @@ -51,10 +52,11 @@ no polling. ### Reading results without flooding context -- **Quick peek (≤40 lines):** call `bgtail` with the job id and `lines: 40` — strips the `__BGRUN_EXIT__` marker. +- **Quick peek (≤40 lines):** call `bgtail` with the job id and `lines: 40` — strips the `__BGRUN_EXIT__` marker. The first read returns the last-40 tail; repeat reads return only lines appended since your last read (delta tailing) — polling a running job is nearly free. +- **Failure extraction:** `bggrep(, "pattern")` — line-numbered matches with optional context lines, capped and condensed. Works on global jobs dirs that `ctx_execute_file` cannot reach (it runs inside the extension). Pass your own pattern whenever you know the tool's output format; the default only catches common failure signatures. - **Whole-log failure analysis:** `ctx_execute_file` on the log path: - ``` + ```javascript ctx_execute_file( path: "~/.pi-bgrun/jobs/.log", language: "javascript", @@ -67,8 +69,22 @@ no polling. A 10 000-line `make test` log collapses to a ~30-line summary in context. -**Never `cat`, `Read`, or `bash cat` a full bgrun log.** Always `bgtail` or -`ctx_execute_file`. +**Why `bggrep` instead of `bash grep` on the log?** + +- `bash grep` output is uncapped — a retry-storm log can dump thousands of + matching lines (megabytes) straight into context, and staying safe depends + on remembering `| head` on every single call. `bggrep` is bounded by design + (~50 matches, ~2KB/line, ~8KB). +- It takes the job id — no log-path reconstruction, no shell-quoting of the + regex — and works on any jobs dir, including global logs that + project-sandboxed `ctx_execute_file` cannot reach. +- Output is self-describing: match count, line numbers, `…[N skipped]…` gap + markers, `— none` for no-match. + +Plain `grep` via bash is fine only for a one-off search you know is tiny. + +**Never `cat`, `Read`, `bash cat`, or `bash grep` a full bgrun log.** Always +`bgtail`, `bggrep`, or `ctx_execute_file`. ## After a pi restart or session switch @@ -85,7 +101,10 @@ no polling. - Call the tools; never hand-roll `nohup … &` inline. - One job = one id. Multiple concurrent jobs are fine — each has its own log. -- Logs live in `~/.pi-bgrun/jobs` (override with `PI_BGRUN_DIR`). +- Logs live in `~/.pi-bgrun/jobs` (override with `PI_BGRUN_DIR`). A **relative** + `jobsDir` in the project config (e.g. `.pi-bgrun/jobs`) puts logs inside the + project — auto-ignored via `.git/info/exclude` — which keeps them reachable + for project-sandboxed analysis tools like `ctx_execute_file`. - Cleanup: `bgclean` removes only THIS session's old logs; `bgclean all` sweeps every session's. Auto-sweeps at session start/shutdown are session-scoped plus a global orphan pass (default on — removes finished