Skip to content

feat: bggrep tool + bgtail delta tailing - #2

Closed
lloydsk wants to merge 5 commits into
lloydsk/reduce-context-usagefrom
lloydsk/bggrep-delta
Closed

feat: bggrep tool + bgtail delta tailing#2
lloydsk wants to merge 5 commits into
lloydsk/reduce-context-usagefrom
lloydsk/bggrep-delta

Conversation

@lloydsk

@lloydsk lloydsk commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #1 (project-local jobs dir) — review/merge that first; this branch contains its commits plus the new work below.

What

Two context-efficiency features for reading job logs:

bggrep — capped in-extension pattern search

  • Regex search over a job's log: line-numbered matches (grep -n style), optional context lines with …[N lines skipped]… gap markers, capped at 50 matches + the ~8KB condenser budget.
  • Runs inside the extension with native fs access, so it works on any jobs dir — including global logs that project-sandboxed tools (ctx_execute_file) cannot reach. That gap repeatedly forced agents into uncapped raw greps; bggrep closes it.
  • With no pattern, a generic failure-signature default is used — deliberately a convenience only (overridden by passing your own pattern): a general-audience tool across languages/task types misses more than it catches, so heuristics are never load-bearing.
  • Record-first log resolution (correct across mid-session config changes), same as bgtail.

bgtail delta tailing

  • First read for a job: full last-N tail (unchanged semantics).
  • Repeat reads: only lines appended since your last read, with a +N new lines header — polling a running job never re-pays context for lines already seen; a no-change poll returns a single tiny line.
  • Bookmark = high-water mark (total content lines + byte size at read time); deliberately-skipped prefix lines are never replayed as "new"; a shrunken/replaced log (detected by line count or bytes) resets to a full tail with a note; raw: true keeps the verbatim last-N window but still advances the bookmark; bookmarks are in-memory only — a session restart starts fresh.
  • The /bgtail slash command inherits everything via the shared core.

Testing

tsc clean; 51/51 node:test — 8 new: bggrep (line numbers + explicit/default/no-match, context windows + gap markers, invalid-pattern error, 50-match cap, record preference) and delta tailing (full→new→none sequence, raw-advances-bookmark, byte-level shrink reset). The delta rewrite also fixed a regression caught by existing tests: the exit marker is now filtered before the window is sliced, so "last N lines" keeps meaning the last N content lines.

Docs

README tools table + two-tier read model; SKILL.md tools table + reading guidance.

@lloydsk

lloydsk commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the fresh-eyes review in f7808cf (56/56 tests, tsc clean):

Finding Resolution
Replacement log with same line count/byte size slips past shrink check Fixed — bookmark now stores the first content line; append-only logs never mutate line 0, so a changed first line = replacement → full-tail reset. Invariant-based, catches rotation regardless of size
bggrep empty log: "in 1 lines" via "".split("\n") === [""] Fixedin 0 lines
CRLF: trailing \r breaks $-anchored patterns, leaks into output Fixed in both bgtail and bggrep (/\r?\n/ split)
No schema minimums: lines: 0slice(-0) returns everything; negative context drops the match lines themselves Fixed — schema minimums + defensive clamps in the shared cores (slash-command path bypasses schemas)
Dead branch in bgtail's empty-body ternary Removed (delta early-return makes it unreachable)
2KB/line cap undocumented Fixed in README + SKILL.md wherever ~8KB appears
Test gaps Added — replacement reset, CRLF, empty+notFound, context+cap interaction, param clamps
Relative jobsDir traversal → useless ../ exclude pattern Hardened in this branch — appendExcludePattern skips ../-prefixed patterns (unreachable via the walk-up today; defense-in-depth)

Declined by design (from the review's request-changes on #1): ancestor walk-up for subdirectory cwds — an adjudicated design decision; the README's Rules section documents the actual behavior (cwd must itself be a project root, else global fallback). Path traversal in jobsDir is user-owned config, not an attack surface.

@lloydsk
lloydsk added this pull request to stack #3 September 10, 2026 16:35
…migration-safe) (#1)

* feat: project-local jobs dir via relative jobsDir (auto git-exclude, config-change-safe log lookup)

A relative jobsDir (any config layer or PI_BGRUN_DIR) now resolves against the
session's project root, putting logs inside the workspace — reachable for
project-sandboxed analysis tools (ctx_execute_file/ctx_index) and scoped per
checkout. Absolute paths behave exactly as before (migration-safe). With no
recognizable project root a relative path falls back to the global dir instead
of scattering logs. Project-local dirs are auto-added to .git/info/exclude
(local-only, linked-worktree aware) so logs never pollute git status. bgtail
now prefers the session record's logPath, so jobs stay readable across
mid-session config changes (e.g. switching to project-local after upgrade).

* chore: formatter pass on new tests; language-tag SKILL.md example fence (MD040)

* fix: exclude-write retries on transient failure; gitdir pointers with spaces

ensureGitExcluded now memoizes only on success: a failed attempt (unwritable
exclude file, unparseable .git pointer) is retried on the next bgrun instead of
being permanently skipped. The gitdir pointer regex accepts paths containing
spaces (was \S+, which truncated at the first space).

* chore: formatter reflow in spaces-path test
bggrep: capped in-extension regex search over job logs — line-numbered
matches, optional context with gap markers, 50-match + ~8KB condenser caps;
works on any jobs dir (native fs), closing the gap where project-sandboxed
tools cannot reach global logs; generic failure-pattern default is a
convenience only, always overridable.

bgtail delta tailing: first read = full last-N tail (unchanged); repeat
reads return only lines appended since the last read (high-water bookmark of
content lines + bytes); shrunken/replaced logs reset to a full tail; raw:true
keeps the verbatim window but advances the bookmark. Exit marker now filtered
before slicing so last-N means last N content lines.
… sandbox gap

Rationale captured in the three places that steer behavior: the run-bg skill
(rules + why-not section), the README read-model section, and the bggrep
tool's prompt guidelines. Never bash-grep a bgrun log; plain grep only for
one-off searches known to be tiny.
…ty-log

bgtail delta: store the first content line in the bookmark — append-only
logs never mutate line 0, so a changed first line means the log was replaced
or rotated; catches same-line-count/same-size replacements the shrink checks
cannot see. Also normalize CRLF (stray \r broke nothing but leaked into
output), clamp lines to >= 1 (slice(-0) pitfall), and drop an unreachable
body branch.

bggrep: empty log now reports 'in 0 lines' (was 'in 1 lines' via
''.split('\n') === ['']); CRLF normalized so $-anchored patterns match;
context clamped to >= 0 (negative context dropped the match lines
themselves); schema minimums added for lines/context.

appendExcludePattern: skip ../-prefixed patterns (unreachable via the
walk-up today, defense-in-depth for future callers/symlinks).

Docs: ~2KB/line cap now documented alongside ~8KB in README + SKILL.md.
Tests: +6 (replacement reset, CRLF, empty+notFound, context+cap, clamps) — 57 total.
@lloydsk
lloydsk force-pushed the lloydsk/bggrep-delta branch from f7808cf to 237e706 Compare September 10, 2026 16:40
@lloydsk
lloydsk deleted the branch lloydsk/reduce-context-usage September 10, 2026 16:40
@lloydsk lloydsk closed this Sep 10, 2026
@lloydsk
lloydsk deleted the lloydsk/bggrep-delta branch September 10, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant