Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ External knowledge sources are configured with active `*.inputs.toml` manifests
- **B2 project fallback identity + historical re-resolution**: when the global registry (`~/.config/backscroll/projects.toml`) does not match a session's cwd, `projects.Identify()` derives a sanitized fallback id from the cwd basename (lowercase, `[a-z0-9-_]`; registry always wins when it matches). `rebuild` runs a re-resolution pass: `ReresolveProjects` decodes Claude's session-dir encoding (dashes-for-slashes; if the decoded path exists on disk its basename wins, else the last dash segment — the encoding is lossy and this heuristic is documented as ambiguous for dir names containing dashes) and relabels rows stuck at `project='unknown'`, returning distinct files resolved. Fallback labels are not revisited once set — followed by registry-aware re-resolution below.
- **B2.1 registry-aware re-resolution**: `projects.Identify()` now carries a `FromRegistry` flag to distinguish registry matches from fallback labels. `rebuild` includes a second phase that loads `~/.config/backscroll/projects.toml` and re-resolves historical sessions labeled with fallback IDs. For each session path, it decodes the cwd, applies cross-host normalization, and calls `Identify` with the registry. If a registry entry matches (FromRegistry=true) and differs from the stored fallback, the rows are updated. Only registry matches count — fallback-only paths are not re-labeled (no churn). This allows a future registry entry to correct historical misattribution without touching already-correct sessions. Wired into rebuild via `ReresolveProjectsWithRegistry` (queries.go).
- **B3 retroactive mining over stored text**: for sessions that expire from disk before Template/Correction/Sequence mining runs, `BackfillDerived()` recovers templates, correction signals, and lossy tool_events from stored text in search_items. Stale template re-mining (F2a) is integrated: BackfillDerived first discovers stale paths via `StaleTemplatePaths(CurrentNormalizationVersion)` and re-mines templates under v2 heuristics, updating existing templates to normalization_version=2. Template mining reuses `internal/templates/Miner` over tool-text rows. Correction detection filters input to prose only (role='user' AND content_type IN ('text','code')) to avoid tool_result false positives; lexicon, denial, and rephrase detectors run on prose; interrupt detector runs on all user messages. Lossy tool_events reverse-parse toolfmt input serialization (<ToolName> <key=val> ...; heuristic: first token has no '=', at least one subsequent has '=') to extract tool_name and command_head; outputs unattributable without tool_use_id so skipped; extraction_version=0 marks lossy rows (inputs only, unrecoverable once disk source expires). `rebuild` command now runs after mutation-class startup sync and then: (1) re-derives FTS from DB, (2) BackfillDerived (stale-path discovery + re-mining + expired-file discovery + batch mining + progress reporting), (3) re-resolves projects, and (4) applies registry-aware re-resolution. It performs no second sync. The recover path retains the remediation lock through post-install sync.
- **Early input validation**: CLI commands validate positional arguments and semantic flag values through Cobra `Args`, which runs before the root `PersistentPreRunE`; the root also checks required flags and flag groups before invoking the startup policy for the command class. Invalid invocations therefore fail before opening, creating, synchronizing, or recovering the database. `internal/services` owns pure query validation as the shared CLI/MCP boundary. Direct service calls validate before database access. Search accepts limit 0 for the historical internal default of 100, or an explicit limit from 1 through 300. Context accepts max_tokens from 64 through 16384 and uses 2000 by default.
- **Early input validation**: CLI commands validate positional arguments and semantic flag values through Cobra `Args`, which runs before the root `PersistentPreRunE`; the root also checks required flags and flag groups before invoking the startup policy for the command class. Invalid invocations therefore fail before opening, creating, synchronizing, or recovering the database. `internal/services` owns pure query validation, including opt-in relaxation syntax, as the shared CLI/MCP boundary. Direct service calls validate before database access. Search accepts limit 0 for the historical internal default of 100, or an explicit limit from 1 through 300. Context accepts max_tokens from 64 through 16384 and uses 2000 by default.
- **Coverage gate**: CI enforces ≥85% aggregate statement coverage via `go test ./... -race -coverprofile`. Local check: `bash scripts/check-coverage.sh`. Tests that depend on local machine state (e.g. `~/.config/backscroll/projects.toml`) must use `t.Setenv("HOME", tempDir)` to stay reproducible on CI. Likewise, `InputsDir` branches requiring `BACKSCROLL_CONFIG_DIR` to be unset must set it to `""` via `t.Setenv`. To test the `Validate` orphan path, insert directly into `search_items` without a matching `indexed_files` row.
- **Zero-result guidance**: when `search`/`list` return no rows, actionable suggestions (`--all-projects`, `--content-type tool`, `backscroll status`) are printed to STDERR — never STDOUT, so `--json` stays a clean empty payload.
- **Search robot output contract**: robot mode on search emits `result_N_field=value` lines exactly once-wrapped (the robot path writes lines directly; passing pre-formatted lines through the picokit formatter double-wraps them as `result_N=result_N_field=...`). Search robot string values escape backslash as `\\`, carriage return as `\r`, and newline as `\n`. Minimal fields use bounded snippets; full fields retain indexed content. Positive `--max-tokens` budgets count the complete escaped payload with one Picokit estimate, including omission metadata; never sum independently rounded line/result estimates. Truncation removes whole trailing results, and emits no output when even the omission record cannot fit.
Expand Down
23 changes: 23 additions & 0 deletions cmd/backscroll/search_relaxation_output_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import (
"strings"
"testing"

"github.com/pablontiv/backscroll/internal/models"
"github.com/pablontiv/backscroll/internal/services"
"github.com/pablontiv/backscroll/internal/storage"
picokitoutput "github.com/pablontiv/picokit/output"
)
Expand Down Expand Up @@ -34,6 +36,27 @@ func TestSearchRelaxationBudgetKeepsProvenanceWithEachResult(t *testing.T) {
}
}

func TestSearchRelaxationValidationMatchesServiceBytes(t *testing.T) {
testEnv(t)
query := `"unclosed`
serviceErr := services.ValidateSearchRequest(services.SearchRequest{
Query: query,
Options: models.SearchOptions{Limit: 20},
Relax: true,
})
if serviceErr == nil {
t.Fatal("service validation accepted invalid relaxation syntax")
}

stdout, stderr, cliErr := runCmd("search", "--text", query, "--relax", "--robot")
if cliErr == nil || cliErr.Error() != serviceErr.Error() {
t.Fatalf("CLI error = %v, service error = %v", cliErr, serviceErr)
}
if stdout != "" || stderr != "" {
t.Fatalf("CLI output = (%q, %q), want empty output", stdout, stderr)
}
}

func TestSearchRelaxationInvalidSyntaxPrecedesStartup(t *testing.T) {
testEnv(t)
for _, query := range []string{`"unclosed`, "+", `""`, "+ word", "handshake adaptation ...", "violet && handshake", "+...", `"... &&"`} {
Expand Down
17 changes: 17 additions & 0 deletions internal/services/query_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,10 @@ func TestValidateSearchRequestExactErrorsAndOrder(t *testing.T) {
request: SearchRequest{Query: `"unclosed`, Options: models.SearchOptions{Limit: 20}, Relax: true},
want: "invalid --relax query: quoted phrases must be closed and whitespace-delimited",
},
{
name: "relaxation syntax stays opt-in",
request: SearchRequest{Query: `"unclosed`, Options: models.SearchOptions{Limit: 20}},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
Expand All @@ -158,6 +162,19 @@ func TestValidateSearchRequestExactErrorsAndOrder(t *testing.T) {
}
}

func TestQueryServiceRejectsInvalidRelaxationBeforeDatabaseAccess(t *testing.T) {
service := QueryService{}
response, err := service.Search(context.Background(), SearchRequest{
Query: `"unclosed`,
Options: models.SearchOptions{Limit: 20},
Relax: true,
})
assertExactValidationError(t, err, "invalid --relax query: quoted phrases must be closed and whitespace-delimited")
if len(response.Results) != 0 || response.Provenance.RelaxationStages != nil {
t.Fatalf("response = %+v, want empty response", response)
}
}

func TestValidateExplicitSearchScopeComposesBaseValidation(t *testing.T) {
tests := []struct {
name string
Expand Down
Loading