diff --git a/README.md b/README.md index a8e6126..023d45d 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,8 @@ Restart pi after install so the extension loads. | ------ | --------- | | `bgrun` | Launch a command detached in the background. Optional `name` gives the job a short human-readable label. Returns `started: ` immediately. Wakes the session automatically on completion. | | `bgstatus` | Show job status. With an id: any job's state + exit code. Without: this session's running jobs (finished jobs hidden by default — pass `includeDone: true` or set `showCompletedJobs`). Jobs from other sessions are only listed when `adoptForeignJobs` is enabled. | -| `bgtail` | Print the last N lines of a job's log (default 40), **condensed for context**: ANSI escapes stripped, repeated lines collapsed, long lines and total size capped. Pass `raw: true` to skip condensing. | +| `bgtail` | Read the newest lines of a job's log (default 40), **condensed for context**: ANSI escapes stripped, repeated lines collapsed, long lines and total size capped. First read = full last-N tail; repeat reads return **only lines appended since your last read** (delta tailing) — polling a running job never re-pays for lines already seen. Pass `raw: true` for the unprocessed last-N window (still advances the bookmark). | +| `bggrep` | Regex search over a job's log: line-numbered matches, optional `context` lines, capped (~50 matches, ~2KB/line, ~8KB) and condensed. Runs inside the extension, so it reaches **any** jobs dir — including global logs that project-sandboxed tools (`ctx_execute_file`) cannot. With no `pattern`, a generic failure-signature default is used (override it — convenience, not guarantee). | | `bgclean` | Remove old job logs. **Default scope: this session's jobs only** — other sessions' logs are untouched. Pass `all: true` to sweep the whole shared jobs dir. Retention: `cleanupDays` config (7 days). Never removes a running job's log. | ## Slash commands @@ -85,11 +86,26 @@ exit code even after a restart. Two-tier read model — the log file stays complete on disk for deep analysis; only bounded digests ever enter the conversation: -- **Quick peek:** `bgtail ` — condensed last-40-lines (ANSI stripped, repeats -collapsed, ~8KB cap). The wake message itself already carries the exit code - and the log's last line, so many turns need no follow-up read at all. -- **Whole-log analysis:** `ctx_execute_file` on the job's log path to extract - only failure lines. Never `cat` or `Read` a full bgrun log. +- **Quick peek:** `bgtail ` — condensed newest lines (ANSI stripped, repeats + collapsed, ~2KB/line and ~8KB caps). The first read is the last-40-lines tail; each later + read returns only what was appended since, so repeated polling is nearly + free. The wake message itself already carries the exit code and the log's + last line, so many turns need no follow-up read at all. +- **Pattern search:** `bggrep [pattern] [context]` — line-numbered matches, + capped and condensed (~50 matches, ~2KB/line, ~8KB); works on global jobs dirs that `ctx_execute_file` + cannot reach. Pass your own pattern when you know the log's format. +- **Whole-log analysis:** `ctx_execute_file` on the job's log path (reachable + when logs are project-local) to extract only failure lines. Never `cat` or + `Read` a full bgrun log. + +**Why `bggrep` instead of `bash grep` on the log?** A bash grep's output is +uncapped — a retry-storm log can dump thousands of matching lines straight +into context, and safety depends on remembering `| head` on every call. +`bggrep` is bounded by design (~50 matches, ~2KB/line, ~8KB), takes the job id instead of +a reconstructed log path (no shell-quoting of the regex), runs on any jobs +dir — including global logs that project-sandboxed tools like +`ctx_execute_file` cannot reach — and reports match counts, line numbers, and +skip markers. Plain `grep` is fine only for a one-off search you know is tiny. ## Configuration diff --git a/extension/index.test.ts b/extension/index.test.ts index e3f8351..66c7a18 100644 --- a/extension/index.test.ts +++ b/extension/index.test.ts @@ -20,6 +20,7 @@ import { writeFileSync, existsSync, mkdirSync, + appendFileSync, readdirSync, } from "node:fs"; import { join } from "node:path"; @@ -1989,3 +1990,580 @@ test("bgtail: prefers the session record's logPath when the jobsDir config chang rmSync(proj, { recursive: true, force: true }); } }); + +// ── bggrep ────────────────────────────────────────────────────────────────── + +test("bggrep: line-numbered matches; explicit pattern wins; default pattern; no-match case", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + + const res = await bgrun.execute( + "c1", + { command: "printf 'alpha\\nerror: boom BANANA\\nomega\\n'" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + assert.ok(id, "got a job id"); + await waitForWakes(wakes, 1); + + // explicit pattern → only matching lines, with line numbers + const g = await bggrep.execute( + "c2", + { id, pattern: "BANANA" }, + undefined, + undefined, + ctx, + ); + assert.equal(g.details.matches, 1); + assert.equal(g.details.notFound, false); + assert.match(g.content[0].text as string, /L2: error: boom BANANA/); + assert.doesNotMatch(g.content[0].text as string, /alpha|omega/); + + // default pattern (no pattern passed) catches the failure signature + const g2 = await bggrep.execute("c3", { id }, undefined, undefined, ctx); + assert.equal(g2.details.matches, 1); + assert.match(g2.content[0].text as string, /1 match for \//); + assert.equal( + g2.details.pattern, + "--- FAIL:|^FAIL\\b|^panic:|fatal error:|AssertionError|Error:|error:|make: \\*\\*\\*.*Error|✗|✖", + ); + + // a log with no failure signatures → clean no-match (not an error) + const res2 = await bgrun.execute( + "c4", + { command: "echo all clear, nothing to see" }, + undefined, + undefined, + ctx, + ); + const id2 = ((res2.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 2); + const g3 = await bggrep.execute( + "c5", + { id: id2 }, + undefined, + undefined, + ctx, + ); + assert.equal(g3.details.matches, 0); + assert.equal(g3.isError, undefined); + assert.match(g3.content[0].text as string, /— none/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: context lines with gap markers between distant matches", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + + const res = await bgrun.execute( + "c1", + { + command: + "printf 'l1\\nMATCH one\\nl3\\nl4\\nl5\\nl6\\nl7\\nMATCH two\\nl9\\n'", + }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + assert.ok(id, "got a job id"); + await waitForWakes(wakes, 1); + + const g = await bggrep.execute( + "c2", + { id, pattern: "MATCH", context: 1 }, + undefined, + undefined, + ctx, + ); + assert.equal(g.details.matches, 2); + const text = g.content[0].text as string; + assert.match(text, /L2: MATCH one/); + assert.match(text, /L1: l1/); // context before + assert.match(text, /L8: MATCH two/); + assert.match(text, /L9: l9/); // context after + assert.match(text, /…\[3 lines skipped\]…/); // l4-l6 between the windows + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: invalid pattern errors clearly", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo hi" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + await assert.rejects( + bggrep.execute( + "c2", + { id, pattern: "([unclosed" }, + undefined, + undefined, + ctx, + ), + /bggrep: invalid pattern/, + ); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: caps at 50 matches with a not-shown note", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: 'for i in $(seq 1 60); do echo "boom $i"; done' }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const g = await bggrep.execute( + "c2", + { id, pattern: "boom" }, + undefined, + undefined, + ctx, + ); + assert.equal(g.details.matches, 60); + assert.equal(g.details.capped, true); + assert.match( + g.content[0].text as string, + /showing first 50; 10 more not shown/, + ); + assert.match(g.content[0].text as string, /L50: boom 50/); + assert.doesNotMatch(g.content[0].text as string, /L51: boom 51/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: prefers the session record's logPath when the jobsDir config changes", async () => { + const proj = mkdtempSync(join(tmpdir(), "pi-bgrun-proj-")); + delete process.env.PI_BGRUN_DIR; + try { + mkdirSync(join(proj, ".git"), { recursive: true }); + mkdirSync(join(proj, ".pi"), { recursive: true }); + writeFileSync( + join(proj, ".pi", "pi-bgrun.json"), + JSON.stringify({ jobsDir: ".pi-bgrun/jobs" }), + ); + const { pi, wakes, tools, ctx } = makeFakePi({ + ctxFields: { cwd: proj, isProjectTrusted: () => true }, + }); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo pattern-target-line" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + + const plainCtx = { ...ctx, cwd: undefined, isProjectTrusted: undefined }; + const g = await bggrep.execute( + "c2", + { id, pattern: "pattern-target" }, + undefined, + undefined, + plainCtx, + ); + assert.equal(g.details.notFound, false); + assert.equal(g.details.matches, 1); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(proj, { recursive: true, force: true }); + } +}); + +// ── bgtail delta tailing ──────────────────────────────────────────────────── + +test("bgtail: delta tailing — first read full tail, then only new lines, then none", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const res = await bgrun.execute( + "c1", + { command: "echo first line" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + assert.ok(id, "got a job id"); + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + // First read: full tail, no delta header + const t1 = await bgtail.execute("c2", { id }, undefined, undefined, ctx); + assert.match(t1.content[0].text as string, /first line/); + assert.equal(t1.details.newLines, undefined); + assert.doesNotMatch( + t1.content[0].text as string, + /new lines since last read/, + ); + + // Log grows: only the new lines come back, with a +N header + appendFileSync(logPath, "appended-A\nappended-B\n"); + const t2 = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + const text2 = t2.content[0].text as string; + assert.match(text2, /\+2 new lines since last read/); + assert.match(text2, /appended-A/); + assert.match(text2, /appended-B/); + assert.doesNotMatch(text2, /first line/); + assert.equal(t2.details.newLines, 2); + + // Nothing new: a tiny no-new-lines response (cheap polling) + const t3 = await bgtail.execute("c4", { id }, undefined, undefined, ctx); + assert.match(t3.content[0].text as string, /no new lines since last read/); + assert.equal(t3.details.linesShown, 0); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bgtail: raw:true keeps the verbatim window but still advances the bookmark", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const res = await bgrun.execute( + "c1", + { command: "echo baseline" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + appendFileSync(logPath, "post-raw line\n"); + const r = await bgtail.execute( + "c2", + { id, lines: 3, raw: true }, + undefined, + undefined, + ctx, + ); + assert.match(r.content[0].text as string, /post-raw line/); + assert.equal(r.details.condensed, false); + + // The raw read advanced the bookmark → the next condensed read is empty + const t = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + assert.match(t.content[0].text as string, /no new lines since last read/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bgtail: a shrunken log resets to a full tail with a note", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const res = await bgrun.execute( + "c1", + { command: "echo long original content line" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + // First read sets the bookmark; then the log is replaced by a shorter one + await bgtail.execute("c2", { id }, undefined, undefined, ctx); + writeFileSync(logPath, "tiny replacement\n"); + const t = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + const text = t.content[0].text as string; + assert.match(text, /log shrank since last read — showing full tail/); + assert.match(text, /tiny replacement/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bgtail: a replaced log with the same line count resets to a full tail", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const res = await bgrun.execute( + "c1", + { command: "printf 'aaaa\\nbbbb\\ncccc\\n'" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + // First read sets the bookmark (3 content lines, first line "aaaa") + await bgtail.execute("c2", { id }, undefined, undefined, ctx); + // Replacement: SAME line count, LARGER byte size (so the shrink checks + // cannot fire), different first line — only the first-line detector + // (append-only logs never mutate line 0) can catch this. + writeFileSync( + logPath, + "xxxxxxxxxxxxxxxxxx\nyyyyyyyyyyyyyyyyyy\nzzzzzzzzzzzzzzzzzz\n", + ); + const t = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + const text = t.content[0].text as string; + assert.match(text, /log was replaced since last read — showing full tail/); + assert.match(text, /xxxxxxxxxxxxxxxxxx/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep and bgtail normalize CRLF logs", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo something" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + const logPath = join(dir, `${id}.log`); + + writeFileSync(logPath, "alpha\r\nerror: boom\r\nomega\r\n"); + // A $-anchored pattern must match despite the CRLF source + const g = await bggrep.execute( + "c2", + { id, pattern: "boom$" }, + undefined, + undefined, + ctx, + ); + assert.match(g.content[0].text as string, /L2: error: boom/); + // And no stray \r leaks into either tool's output + assert.ok(!(g.content[0].text as string).includes("\r")); + const t = await bgtail.execute("c3", { id }, undefined, undefined, ctx); + assert.ok(!(t.content[0].text as string).includes("\r")); + assert.match(t.content[0].text as string, /error: boom/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: empty log reports zero lines, and a missing log is notFound", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo x" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + + writeFileSync(join(dir, `${id}.log`), ""); + const g = await bggrep.execute( + "c2", + { id, pattern: "Error:" }, + undefined, + undefined, + ctx, + ); + assert.match( + g.content[0].text as string, + /0 matches for \/Error:\/ in 0 lines — none/, + ); + + const missing = await bggrep.execute( + "c3", + { id: "no-such-job-123", pattern: "x" }, + undefined, + undefined, + ctx, + ); + assert.equal(missing.isError, true); + assert.equal(missing.details.notFound, true); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bggrep: context windows combine with the 50-match cap", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "echo x" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + + // 240 lines, a hit every 4th line → 60 matches (cap 50); with context: 1 + // each window is [i-1, i+1] and consecutive windows leave a 1-line gap. + const lines: string[] = []; + for (let i = 1; i <= 240; i++) { + lines.push(i % 4 === 0 ? `hit ${i}` : `filler ${i}`); + } + writeFileSync(join(dir, `${id}.log`), lines.join("\n") + "\n"); + const r = await bggrep.execute( + "c2", + { id, pattern: "^hit", context: 1 }, + undefined, + undefined, + ctx, + ); + const text = r.content[0].text as string; + assert.equal(r.details.matches, 60); + assert.equal(r.details.capped, true); + assert.match(text, /showing first 50; 10 more not shown/); + assert.match(text, /L4: hit 4/); + assert.match(text, /…\[1 line skipped\]…/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("bgtail and bggrep clamp nonsensical numeric params", async () => { + const dir = mkdtempSync(join(tmpdir(), "pi-bgrun-test-")); + process.env.PI_BGRUN_DIR = dir; + try { + const { pi, wakes, tools, ctx } = makeFakePi(); + await loadExtension(pi); + const bgrun = tools.get("bgrun")!; + const bgtail = tools.get("bgtail")!; + const bggrep = tools.get("bggrep")!; + const res = await bgrun.execute( + "c1", + { command: "printf 'one\\ntwo\\nthree\\nfour\\nfive\\n'" }, + undefined, + undefined, + ctx, + ); + const id = ((res.content[0].text as string).match(/^started: ([^\n]+)/) || + [])[1]; + await waitForWakes(wakes, 1); + + // lines: 0 must not mean "everything" (slice(-0) pitfall) — clamps to 1 + const t = await bgtail.execute( + "c2", + { id, lines: 0 }, + undefined, + undefined, + ctx, + ); + assert.equal(t.details.linesShown, 1); + assert.match(t.content[0].text as string, /five/); + assert.ok(!(t.content[0].text as string).includes("four")); + + // negative context must not drop the match lines themselves — clamps to 0 + const g = await bggrep.execute( + "c3", + { id, pattern: "^three", context: -1 }, + undefined, + undefined, + ctx, + ); + assert.match(g.content[0].text as string, /L3: three/); + } finally { + delete process.env.PI_BGRUN_DIR; + rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/extension/index.ts b/extension/index.ts index 153797e..d83817f 100644 --- a/extension/index.ts +++ b/extension/index.ts @@ -58,6 +58,13 @@ const DEFAULT_CLEANUP_DAYS = 7; const STALE_POLL_MS = 30_000; // re-check interval for jobs with no live child handle const GLOBAL_JOBS_DIR = join(homedir(), ".pi-bgrun", "jobs"); +// Default regex for bggrep when the caller passes no pattern: common failure +// signatures across test runners and build tools. ONLY a convenience default — +// bggrep's contract is that the caller's own pattern always wins, because a +// generic default on arbitrary tools/languages misses more than it catches. +export const DEFAULT_GREP_PATTERN = + "--- FAIL:|^FAIL\\b|^panic:|fatal error:|AssertionError|Error:|error:|make: \\*\\*\\*.*Error|✗|✖"; + // ── Configuration ─────────────────────────────────────────────────────────── // // Layered: defaults ← user config file ← project config file (trusted projects @@ -202,7 +209,12 @@ function appendExcludePattern( if (!m) return false; // unparseable .git file — retry later gitDir = m[1].trim(); } - const pattern = relative(repoRoot, jobsDir).split(sep).join("/") + "/"; + const rel = relative(repoRoot, jobsDir); + // Defense-in-depth: the walk-up guarantees jobsDir sits under repoRoot, but + // a future caller or symlinked path could break that — ../-prefixed + // patterns are silently useless in gitignore semantics, so skip them. + if (rel.startsWith("..") || isAbsolute(rel)) return true; + const pattern = rel.split(sep).join("/") + "/"; const excludePath = join(gitDir, "info", "exclude"); let existing = ""; try { @@ -818,7 +830,7 @@ export default function (pi: ExtensionAPI) { "Use bgrun (not bash) for any command expected to run >30s or emit >100 lines — tests, builds, linters.", "Give every bgrun job a short name (e.g. name: 'unit-tests') so it's recognizable in status output, the status widget, and wake messages.", "After bgrun returns a job id, continue other work; you will be woken automatically when it finishes.", - "Never cat or Read a full bgrun log — bgtail returns a condensed peek (ANSI stripped, repeats collapsed, ~8KB cap); use ctx_execute_file on the log path only when the condensed tail is insufficient.", + "Never cat or Read a full bgrun log — bgtail returns a condensed peek (ANSI stripped, repeats collapsed, ~8KB cap); use bggrep for pattern search or ctx_execute_file on the log path for whole-log analysis.", ], parameters: Type.Object({ command: Type.String({ @@ -1059,7 +1071,22 @@ export default function (pi: ExtensionAPI) { return { text: out.join("\n"), truncated: notes }; } - // ── bgtail: read last N lines of a job's log, condensed for context ──────── + // ── bgtail: read the newest lines of a job's log, condensed for context ──── + // + // Delta tailing: each read bookmarks the total raw line count at read time + // (the high-water mark of what the caller has had the opportunity to see). + // The FIRST read for a job returns the full last-N tail; repeat reads return + // only lines appended since, so polling a running job never re-pays context + // for lines already seen. Deliberately-skipped prefix lines are never + // replayed as "new". raw: true keeps the verbatim last-N window (no delta + // header) but still advances the bookmark. A shrunken log (rotated/replaced) + // resets to a full tail. Bookmarks are in-memory only — a session restart + // starts fresh with a full tail. + + const tailBookmarks = new Map< + string, + { lines: number; bytes: number; first: string } + >(); // Shared by the bgtail tool (agent-facing) and the /bgtail slash command // (human-facing). @@ -1071,7 +1098,10 @@ export default function (pi: ExtensionAPI) { details: Record; isError?: boolean; }> { - const { id, lines = 40, raw = false } = params; + const { id, lines: linesParam = 40, raw = false } = params; + // Clamp defensively — direct callers (e.g. the slash command) bypass the + // tool schema, and lines < 1 would corrupt slicing (slice(-0) = whole log). + const lines = Math.max(1, Math.floor(linesParam)); if (!id) throw new Error("bgtail: id is required"); // Prefer this session's record: its logPath stays correct even if the // config (and thus the resolved jobs dir) changes mid-session — e.g. a @@ -1080,20 +1110,89 @@ export default function (pi: ExtensionAPI) { jobs.get(id)?.logPath ?? join(resolveConfig(ctx).jobsDir, `${id}.log`); try { const content = readFileSync(logPath, "utf8"); - const all = content - .split("\n") + // Content lines only: the exit marker and blanks are filtered BEFORE the + // window is sliced, so "last N lines" means the last N content lines + // (matching pre-delta behavior) and bookmarks count content lines. + // /\r?\n/ keeps CRLF logs from leaving a stray \r on every line. + const rawLines = content + .split(/\r?\n/) .filter((l) => !l.startsWith(EXIT_MARKER) && l.trim().length > 0); - const tail = all.slice(-lines); - const { text, truncated } = condenseLogLines(tail, { raw }); + const total = rawLines.length; + const first = rawLines[0]?.slice(0, 200) ?? ""; + const prev = tailBookmarks.get(id); + // Append-only logs never mutate earlier lines, so a changed first + // content line means the log was replaced or rotated — reset to a full + // tail. Catches same-size replacements the shrink checks cannot see. + // (A previously-empty log growing content is growth, not replacement.) + const replaced = + prev !== undefined && prev.lines > 0 && prev.first !== first; + const shrank = + prev !== undefined && + (prev.lines > total || prev.bytes > content.length); + let window: string[]; + let header: string | undefined; + let newLines: number | undefined; + if (raw || prev === undefined || shrank || replaced) { + // Full tail: first read, raw mode, or a shrunken/replaced log (reset). + window = rawLines.slice(-lines); + if (!raw && (shrank || replaced)) { + header = shrank + ? "log shrank since last read — showing full tail" + : "log was replaced since last read — showing full tail"; + } + } else { + const fresh = rawLines.slice(prev.lines); + newLines = fresh.length; + if (fresh.length === 0) { + tailBookmarks.set(id, { + lines: total, + bytes: content.length, + first, + }); + return { + content: [ + { + type: "text", + text: `(no new lines since last read — log at ${total} line${total === 1 ? "" : "s"})`, + }, + ], + details: { + id, + linesShown: 0, + logPath, + notFound: false, + condensed: true, + newLines: 0, + totalLines: total, + }, + }; + } + window = fresh.length > lines ? fresh.slice(-lines) : fresh; + header = + `+${fresh.length} new line${fresh.length === 1 ? "" : "s"} since last read — ` + + `log at ${total} lines${fresh.length > lines ? ` (showing last ${lines})` : ""}`; + } + tailBookmarks.set(id, { + lines: total, + bytes: content.length, + first, + }); + const shown = window; + const { text, truncated } = condenseLogLines(shown, { raw }); + // Delta reads early-return above, so an empty window here can only be + // a first read of an empty log (full-tail path). + const body = shown.length === 0 ? "(empty log)" : text; const notes = truncated.length > 0 ? `\n\n(${truncated.join("; ")})` : ""; + const head = header ? `${header}\n` : ""; return { - content: [{ type: "text", text: text + notes || "(empty log)" }], + content: [{ type: "text", text: head + body + notes }], details: { id, - linesShown: tail.length, + linesShown: shown.length, logPath, notFound: false, condensed: !raw, + ...(newLines === undefined ? {} : { newLines, totalLines: total }), ...(truncated.length > 0 ? { condenserNotes: truncated } : {}), }, }; @@ -1112,14 +1211,17 @@ export default function (pi: ExtensionAPI) { name: "bgtail", label: "Tail Background Log", description: - "Print the last N lines of a background job's log (default 40), condensed for context: ANSI escapes stripped, repeated lines collapsed, long lines truncated, output capped (~8KB). Strips the exit-marker line. Pass raw: true for unprocessed output; use ctx_execute_file on the log path for whole-log failure analysis.", + "Read the newest lines of a background job's log, condensed for context: ANSI escapes stripped, repeated lines collapsed, long lines truncated, output capped (~8KB). Strips the exit-marker line. The first read returns the last N lines (default 40); REPEAT reads return only lines appended since your last read (delta tailing) — polling a running job never re-pays for the same lines. raw: true returns the unprocessed last-N window. A shrunken or replaced log resets to a full tail. For pattern search use bggrep; for whole-log analysis, ctx_execute_file on the log path.", promptSnippet: "Read the last N lines of a bgrun job's log", parameters: Type.Object({ id: Type.String({ description: "Job id (from bgrun's 'started: ' response)", }), lines: Type.Optional( - Type.Number({ description: "Number of lines to show (default 40)" }), + Type.Number({ + description: "Number of lines to show (default 40)", + minimum: 1, + }), ), raw: Type.Optional( Type.Boolean({ @@ -1133,6 +1235,152 @@ export default function (pi: ExtensionAPI) { }, }); + // ── bggrep: pattern search over a job's log, capped for context ─────────── + // + // The sandboxed whole-log path (ctx_execute_file) is confined to the + // project root, which a global jobs dir sits outside of — bggrep runs + // inside the extension with native fs access, so it works on any + // configured jobs dir. Matches are line-numbered (grep -n style), + // optionally with context lines, capped at MAX_GREP_MATCHES, and run + // through the same condenser as bgtail so a search can never flood context. + + const MAX_GREP_MATCHES = 50; + + async function bggrepCore( + params: { id: string; pattern?: string; context?: number }, + ctx?: ExtensionContext, + ): Promise<{ + content: { type: "text"; text: string }[]; + details: Record; + isError?: boolean; + }> { + const { id, pattern, context: contextParam = 0 } = params; + // Clamp defensively — negative context would exclude the match lines + // themselves from the context windows (lo > hi no-ops the inner loop). + const context = Math.max(0, Math.floor(contextParam)); + if (!id) throw new Error("bggrep: id is required"); + // Record-first, same as bgtail — correct across config changes. + const logPath = + jobs.get(id)?.logPath ?? join(resolveConfig(ctx).jobsDir, `${id}.log`); + const source = pattern ?? DEFAULT_GREP_PATTERN; + let re: RegExp; + try { + re = new RegExp(source); + } catch (err) { + throw new Error( + `bggrep: invalid pattern ${JSON.stringify(source)}: ${(err as Error).message}`, + ); + } + let rawLines: string[]; + try { + const content = readFileSync(logPath, "utf8"); + // /\r?\n/ normalizes CRLF (a trailing \r would break $-anchored patterns + // and leak into output); blank lines are KEPT so L numbers match the + // file. A trailing empty split element is dropped; "" yields zero lines. + const split = content === "" ? [] : content.split(/\r?\n/); + if (split.length > 0 && split[split.length - 1] === "") split.pop(); + rawLines = split.filter((l) => !l.startsWith(EXIT_MARKER)); + } catch { + return { + content: [ + { type: "text", text: `No log found for job ${id} at ${logPath}` }, + ], + details: { id, matches: 0, logPath, notFound: true }, + isError: true, + }; + } + const matchIdx: number[] = []; + for (let i = 0; i < rawLines.length; i++) { + if (re.test(rawLines[i])) matchIdx.push(i); + } + const header = + `${matchIdx.length} match${matchIdx.length === 1 ? "" : "es"} for /${source}/ ` + + `in ${rawLines.length} line${rawLines.length === 1 ? "" : "s"}`; + if (matchIdx.length === 0) { + return { + content: [{ type: "text", text: `${header} — none` }], + details: { + id, + matches: 0, + linesSearched: rawLines.length, + logPath, + notFound: false, + }, + }; + } + const capped = matchIdx.length > MAX_GREP_MATCHES; + const shownIdx = capped ? matchIdx.slice(0, MAX_GREP_MATCHES) : matchIdx; + // Context windows, merged where they overlap or touch (grep -C style). + const include = new Set(); + for (const i of shownIdx) { + const lo = Math.max(0, i - context); + const hi = Math.min(rawLines.length - 1, i + context); + for (let j = lo; j <= hi; j++) include.add(j); + } + const sorted = [...include].sort((a, b) => a - b); + const out: string[] = []; + let prev = -2; + for (const i of sorted) { + if (prev >= 0 && i > prev + 1) { + const gap = i - prev - 1; + out.push(`…[${gap} line${gap === 1 ? "" : "s"} skipped]…`); + } + out.push(`L${i + 1}: ${rawLines[i]}`); + prev = i; + } + const { text, truncated } = condenseLogLines(out); + const notes = truncated.length > 0 ? `\n\n(${truncated.join("; ")})` : ""; + const capNote = capped + ? ` — showing first ${MAX_GREP_MATCHES}; ${matchIdx.length - MAX_GREP_MATCHES} more not shown` + : ""; + return { + content: [{ type: "text", text: `${header}${capNote}\n${text}${notes}` }], + details: { + id, + matches: matchIdx.length, + linesSearched: rawLines.length, + logPath, + notFound: false, + pattern: source, + capped, + }, + }; + } + + pi.registerTool({ + name: "bggrep", + label: "Grep Background Log", + description: + "Search a background job's log with a regex; returns only matching lines with line numbers (optional context lines), capped (~50 matches, ~8KB) and condensed. Runs inside the extension, so it works on any jobs dir — including global logs that project-sandboxed tools (ctx_execute_file) cannot reach. Pass your own pattern whenever you know the log's format; with no pattern a generic failure-signature default is used (a convenience only — not a guarantee).", + promptSnippet: "Search a bgrun job's log for a pattern", + promptGuidelines: [ + "Never search a bgrun log with the bash tool — uncapped output can flood context, and it needs manual log-path reconstruction and regex shell-quoting; bggrep is bounded by design.", + "Prefer bggrep over bash grep or reading a bgrun log — matches are line-numbered, capped, and condensed.", + "Pass an explicit pattern when you know the tool's output format; the default only catches common failure signatures.", + ], + parameters: Type.Object({ + id: Type.String({ + description: "Job id (from bgrun's 'started: ' response)", + }), + pattern: Type.Optional( + Type.String({ + description: + "Regex to search for. Default: generic failure signatures — override when you know the format.", + }), + ), + context: Type.Optional( + Type.Number({ + description: + "Context lines around each match (default 0, grep -C style)", + minimum: 0, + }), + ), + }), + async execute(_toolCallId, params, _signal, _onUpdate, ctx) { + return bggrepCore(params, ctx); + }, + }); + // ── bgstatus: list jobs (in-memory while alive; dir scan after restart) ───── // Shared by the bgstatus tool (agent-facing) and the /bgstatus slash command diff --git a/skill/run-bg/SKILL.md b/skill/run-bg/SKILL.md index 8f6d3ce..18c81fa 100644 --- a/skill/run-bg/SKILL.md +++ b/skill/run-bg/SKILL.md @@ -30,7 +30,8 @@ no polling. |---|---| | Start | `bgrun(command: "make test-short", name: "unit-tests")` → `started: ` (name is an optional short label; use it so jobs are recognizable in `bgstatus`, the status widget, and wake messages) | | Status | `bgstatus()` for one job, or `bgstatus()` for this session's running jobs — finished jobs are hidden by default; pass `includeDone: true` to list them | -| Tail | `bgtail(, 40)` | +| Tail | `bgtail(, 40)` — first read: last-40 tail; later reads: only lines appended since (delta tailing) | +| Grep | `bggrep(, "pattern", context?)` — line-numbered matches, capped and condensed; default pattern = generic failure signatures (override when you know the format) | | Clean | `bgclean()` for this session's old logs; `bgclean all` to sweep every session's (default 7-day retention) | ## Workflow @@ -51,7 +52,8 @@ no polling. ### Reading results without flooding context -- **Quick peek (≤40 lines):** call `bgtail` with the job id and `lines: 40` — strips the `__BGRUN_EXIT__` marker. +- **Quick peek (≤40 lines):** call `bgtail` with the job id and `lines: 40` — strips the `__BGRUN_EXIT__` marker. The first read returns the last-40 tail; repeat reads return only lines appended since your last read (delta tailing) — polling a running job is nearly free. +- **Failure extraction:** `bggrep(, "pattern")` — line-numbered matches with optional context lines, capped and condensed. Works on global jobs dirs that `ctx_execute_file` cannot reach (it runs inside the extension). Pass your own pattern whenever you know the tool's output format; the default only catches common failure signatures. - **Whole-log failure analysis:** `ctx_execute_file` on the log path: ```javascript @@ -67,8 +69,22 @@ no polling. A 10 000-line `make test` log collapses to a ~30-line summary in context. -**Never `cat`, `Read`, or `bash cat` a full bgrun log.** Always `bgtail` or -`ctx_execute_file`. +**Why `bggrep` instead of `bash grep` on the log?** + +- `bash grep` output is uncapped — a retry-storm log can dump thousands of + matching lines (megabytes) straight into context, and staying safe depends + on remembering `| head` on every single call. `bggrep` is bounded by design + (~50 matches, ~2KB/line, ~8KB). +- It takes the job id — no log-path reconstruction, no shell-quoting of the + regex — and works on any jobs dir, including global logs that + project-sandboxed `ctx_execute_file` cannot reach. +- Output is self-describing: match count, line numbers, `…[N skipped]…` gap + markers, `— none` for no-match. + +Plain `grep` via bash is fine only for a one-off search you know is tiny. + +**Never `cat`, `Read`, `bash cat`, or `bash grep` a full bgrun log.** Always +`bgtail`, `bggrep`, or `ctx_execute_file`. ## After a pi restart or session switch