diff --git a/docs/analytics-source.md b/docs/analytics-source.md new file mode 100644 index 00000000..1f212280 --- /dev/null +++ b/docs/analytics-source.md @@ -0,0 +1,271 @@ +--- +title: Analytics source preparation +nav_order: 8 +permalink: /analytics-source/ +--- + +# Analytics source preparation + +`gitcrawl analytics owner/repo` supports collector-owned publication repair, +identity enrichment and ongoing GraphQL collection for a full-history archive. +It uses the configured native source database and the existing token helper. +It does not run embeddings or classification models. + +```sh +gitcrawl --config SOURCE_CONFIG analytics owner/repo --json +gitcrawl --config SOURCE_CONFIG analytics owner/repo --status --json +gitcrawl --config SOURCE_CONFIG analytics owner/repo --apply --json +gitcrawl --config SOURCE_CONFIG --github-token-command TOKEN_HELPER \ + analytics owner/repo --enrich --watch --json +``` + +Without apply/enrich/watch/once, the command audits retained publication evidence +read-only. `--apply` performs an idempotent, bounded repair after taking an +operator-managed consistent backup. It stores `submitted_at_gh` and +`publication_at_gh` for current comments and source revisions. Review submission +is distinct from draft creation; pending reviews have no published event. Raw +payloads, existing IDs, genuine creation times and original observation times +remain unchanged, and normalization creates no fictional provider edit. + +Source schema 14 added these fields and the actor/coverage evidence tables. +Historical normalization completion is recorded in +`analytics_collection_state.publication_repair_generation`; downstream consumers +must reconcile the affected datasets when this generation changes rather than +relying solely on append-only revision IDs. + +GraphQL history now retains actor node IDs. `--enrich` recovers missing identities +through original content nodes, not login guessing, and stores public actor +profiles. Unavailable nodes are explicit unknowns; permission or transport +failures are not converted to successful missing-data evidence. New source actors +enter the profile queue, and profiles become eligible for refresh after 24 hours. +Operational queue tables are separate from publishable evidence. + +`--watch` polls updated issues and PRs every two minutes, overlapping the prior +verified watermark by five minutes. It paginates without GitHub search's hit cap +and hydrates relevant updated threads. Issue and PR lanes have independent durable +checkpoints and process at most two discovery pages per cycle. Eight two-thread +requests progress independently per page. A rejected batch splits into individual +requests; an item may be passed only after its failure is durably queued. Unrelated +items and the other discovery lane continue. Core retries process at most eight +due items before discovery, with a two-minute request deadline and bounded +exponential backoff. After persisting core coverage, targeted review recovery +uses the time until the next nominal two-minute core poll; it does not add a +two-minute idle wait after recovery. Each wave rechecks actual GraphQL quota and +admits up to 256 items through 32 recovery workers, with up to eight PRs per +request. Waves are also limited to a 16 MiB estimated response budget, using a +conservative 64 KiB/item initial estimate that grows with measured bytes. Individual +recovery responses are capped at 32 MiB; rejected batches are isolated normally. +Ordinary capture retains its existing eight two-item workers. Admission +uses the authoritative GraphQL `rateLimit` response rather than REST resource +counters, which can differ. Recovery and its quota probe omit redundant REST +`/rate_limit` preflights: an explicit GraphQL probe binds actual quota to the +selected credential, and every content/page request checks that credential and +unexpired balance against the reserve. Rotation requires a new probe. Ordinary +core transport retains its REST preflight. If that fresh response carries an +expired GraphQL snapshot (including crossing reset during preflight), it performs +a bounded quota-only GraphQL refresh under the existing request lock. The refresh +uses the selected credential and API origin; rotation before content dispatch +fails closed. A stale, invalid, failed, or below-reserve refresh cannot authorize +content. The refreshed balance is checked against the pending page's estimate, +without changing its identity, cursor, or completeness validation. Each history +session also enforces its configured floor against observed GraphQL balances +before pagination. A +client retains the lowest observed GraphQL balance until the reset boundary +passes. An upward sample or a shifted future reset cannot increase admission; +REST snapshot refreshes do not overwrite this evidence. Logs distinguish the +raw provider balance/reset from the conservative effective admission values. A +32-point-per-item admission margin and per-request native quota checks preserve +3,000 points for recovery, leaving 1,500 points above ordinary capture's floor. +Missing or expired quota stops provider recovery; local discovery can continue. +The request window yields before core polling and cancellation retains retry +receipts and the last committed scan. Quota and numeric per-query cost logs make +the actual spending observable without credentials or content bodies. +No partial connection is accepted as complete evidence. Non-nested +continuation pages use 100 nodes to avoid repeated small round trips. Identity/profile enrichment uses eight disjoint 100-node requests with normal +quota guards and can run concurrently under +the same source owner. `--once` performs one resumable update cycle. + +Discovery skips unresolved core retry items. A review-only recovery queue does +not defer a newly discovered core edit: that edit is collected as core work. If +it fails, a core retry obligation prevents repeated discovery attempts until the +bounded scheduler selects it. This prevents enrichment backoff from hiding fresh +core data while its verified coverage watermark advances. +When both queues contain an item, its core retry owns the backoff; the review +retry pass cannot dispatch the same item. Recovery resolves only after an +accepted membership observation exists, including a proven empty set. + +Targeted recovery fetches only PR identity/update metadata and fully paginated +review threads with their complete inline comments and reply-to identities. It +does not request or project the PR body/title, issue comments, or review history. +An independent native child observation reserves only the review-thread family; +existing review state, revisions and exact membership publish in one transaction +after repository/node/number binding. Canonical threads, comments, revisions and +vectors remain untouched. Current content changes remain ordinary capture's job. +A review-only success cannot resolve a core traversal failure. + +Up to one quarter of a recovery wave is reserved for already-attempted due retries; +the remaining slots serve first-pass work, with unused capacity shared. This avoids +waiting behind the entire seeded census. Explicit retained `NOT_FOUND` evidence +has at least a 15-minute backoff; it never implies deletion or empty membership. +`review_state_wave` logs actual peak busy workers, worker/provider/database time, +items, response bytes and reported query points. Private success receipts identify +review-only work and its fetch/persistence timings. + +Hydration workers reuse the watch owner's open store instead of repeating +full-archive migration checks for each batch. Independent guarded clients overlap +network work; native transactions still coordinate source writes. Enrichment +continues checking its queues every two minutes after the initial drain. New +observations enqueue unresolved content identities or known actor profiles +directly, avoiding repeated whole-archive scans during watch operation. + +A completed historical discovery receipt supplies the initial discovery baseline. +The pre-upgrade checkpoint is retained; migration copies its in-flight cursor into +the corresponding independent lane. `through:owner/repo:issues` and +`through:owner/repo:pullRequests` record discovery progress; the aggregate watermark +is their minimum. These are not proof that queued failures have been repaired. +`analytics_coverage.complete` continues to describe verified core issue/PR/comment +traversal. Core failures clear it until reconciled; the previously verified +watermark remains available while another page is in progress. Historical +review-state enrichment does not invalidate core contributor/response coverage. + +## Review-state and failure contracts (schema 15) + +GraphQL collection requires explicit `isResolved` and `isOutdated` values and +retains review-thread IDs, source state, source comments and immediate reply IDs. +It writes the existing tables: + +- `pull_request_review_threads`: current observation, keyed by + `(thread_id, review_thread_id)`, including `is_resolved`, `is_outdated`, + `comments_json`, retained `raw_json`, and actual `fetched_at`. +- `pull_request_review_thread_revisions`: append-only changes, with integer `id` + and actual `recorded_at`; unchanged observations do not manufacture revisions. +- `pull_request_review_thread_syncs`: last complete observation per `thread_id`. + New nullable `review_thread_ids_json` contains its exact native ID membership. + NULL means not acquired under this contract; `[]` means a complete empty set. +- `analytics_review_state_coverage`: keyed by `repository`, with `cursor`, + `ceiling`, `scanned`, `queued`, `pending_items`, `scan_complete`, `complete`, + and `observed_at`. This separate completion contract requires a finished + targeted scan and no outstanding review-state recovery items. + +Absence from a complete membership set never invents a provider deletion or +removes retained history. Existing comment IDs/replies and +`thread_child_observation_memberships` keep their existing contract. Consumers +must distinguish historical rows from the latest provider membership. + +The watch inspects bounded chunks of 5,000 primary-key rows within each recovery +window up to a captured +ceiling, queuing only PRs with retained review threads or unknown connection data. +Complete retained zero-count GraphQL connections materialize `[]` using their +actual retained observation time. Conditional writes preserve a newer live +observation; incomplete or undated evidence instead enters provider recovery. +`review_state_recovery:owner/repo` records cursor, ceiling, scanned/queued counts +and scan completion. Queue completion is separately required for review-state +coverage, never for the existing core coverage flag. This +targeted recovery does not restart historical discovery or enable remote syncing; +missing historical resolution states cannot be reconstructed from old payloads. + +Operational tables are **not public conversation datasets**: + +- `analytics_fetch_attempts(id, repository, number, operation, started_at, + finished_at, status, error_class, error_text, evidence_json)` retains successful + and rejected GraphQL work. `number=0` identifies a discovery-lane request. + Rejection evidence is bounded structural metadata, IDs, counts, pagination and + body lengths/hashes; it contains no credentials, response headers or prose bodies. + Partial-response receipts also retain `graphql_errors`: total count, up to eight + allowlisted provider codes and query paths of at most eight components, with + explicit truncation markers. Unknown codes/path components are null. Messages, + arbitrary field values and identities are excluded from this error metadata. + Every rejection also has private `cause` metadata: allowlisted category/type, + guard or validation code, HTTP status when available, up to eight request or + pagination stages, and numeric quota/reset evidence for quota guards. The + primary error chain is bounded to sixteen links with explicit truncation. + Wrapping a transport or guard error as a validation failure retains this cause + without changing the existing error class, retry or acceptance behavior. No + error messages, URLs, tokens, headers, bodies, identities or certificate subjects + are copied into this diagnostic metadata. Unknown types remain `unclassified`. + Older receipts are not rewritten to infer a cause from later provider reads. +- `analytics_retries(repository, number, operation, first_seen_at, last_seen_at, + next_attempt_at, attempts, last_attempt_id, resolved_at)` is keyed by + `(repository, number, operation)`. Resolved entries and attempt history remain. + Operations include `graphql_history` (core traversal), `review_state` + (targeted enrichment), `discover_issues`, and + `discover_pullRequests`. Recovery queue rows start with zero attempts. + +`analytics --status --json` reads bounded recent receipts, outstanding retry count, +discovery coverage and review-state scan progress without credentials or collection. +Timestamped logs distinguish `github_update_complete`, `github_update_failed`, +`github_coverage_pending` (core), and `review_state_progress` (enrichment). +`review_state_quota` records fresh limit/remaining/reset/reserve and admitted wave +size; `review_state_cost` records actual provider points per recovery query. +The older successful-only `sync_runs` table is not a +complete failure ledger. Existing embeddings are reused; this command does not +generate embeddings or reinterpret empty review bodies as missing replies. + +The permanent `runner.lock` coordinates ownership with the full-history backfill +supervisor. Do not unlink it or start a second supervisor against the same archive. +A finished backfill should be disabled as an automatic startup job when ongoing +collection takes ownership. The token-command result is held only in memory and +refreshed before its expected expiry, preserving one credential across quota +reservation and dispatch. Existing provider-rate protections remain active. + +See [sync](/sync/), [configuration](/configuration/) and the [command reference](/commands/). + +## Owner-directed removal + +Source schema 16 adds local owner exclusions. An exclusion is an operator policy, +not a provider deletion, an empty review membership, or a successful recovery. +Plan an exact set of issue/PR numbers before applying it: + +```sh +gitcrawl --config SOURCE_CONFIG purge-threads owner/repo --numbers 123,456 --json +gitcrawl --config SOURCE_CONFIG purge-threads owner/repo --numbers 123,456 \ + --apply --request-id owner-request-reference --json +``` + +The default is a read-only metadata/count plan. Apply requires the existing +collector `runner.lock` to be idle, opens the writer only after taking that lock, +and atomically records exclusions while removing the selected content, revisions, +review memberships, derived documents/vectors, content-node identity evidence, +queued retries and per-target receipts. Shared actor profiles, unrelated blobs, +unrelated conversations and peers' mixed-batch diagnostic receipts stay intact. +Blob-backed targets and linked workflow reservations require separate owner repair +and are refused. Repositories with retained workflow-run snapshots are also +refused: this bounded command cannot establish exclusive ownership of their +current or historical payloads. Shared cluster relationships are refused. +Missing/ambiguous targets and deletions that could allow SQLite to +reuse a removed native integer ID are also refused. There is no automatic tail-ID +renumbering, backup copy, provider request, or permission change. + +`thread_exclusions` has primary key `(repository TEXT, number INTEGER)` and stores +`kind`, `original_thread_id`, `github_id`, `excluded_at`, fixed +`reason='owner_requested'`, and `request_id`. `thread_excluded_nodes` has primary +key `node_id TEXT` and links to the policy's repository/number. These are local +operational policy tables. They contain no content bodies. The policy follows the +explicit collector `owner/repo` namespace, which normal sync retains even when +provider metadata reports a different canonical name. Changing the configured +target requires reviewing and carrying its exclusion policy; this command does +not introduce repository rename/transfer migration. Normal discovery, +explicit sync, retry scheduling and identity enrichment respect the exclusions; +store guards prevent reinsertion. Existing `analytics_coverage.complete` still +expresses core collection health. Review coverage has an independent +`owner_excluded_items INTEGER` count and stays incomplete when this count is +positive, even when `pending_items=0`. + +For an **existing managed portable reader mirror**, use its checkout configuration +and add `--runtime-mirror` to both commands. This resolves and locks the established +mirror without downloading, recreating, migrating to the full archive schema, or +editing publisher Git data. It uses the existing writable-mirror policy: scheduled +portable refresh preserves local bytes instead of replacing them from upstream. +Consequently that compact mirror no longer advances with upstream snapshots; the +independent full-history collector continues normal updates. A direct portable +DB alias is refused because it cannot bind the publisher's ownership lock. Older +sparse formats with dangling blob references are refused. Portable publication +from an archive containing local owner exclusions is refused to avoid dropping +suppression or publishing local request metadata. Keep the exclusion policy and +native removal receipt with the archive when moving it. + +After applying, verify zero selected content/history/queue rows, retained peer +hashes, the exact exclusion count, and an advancing ordinary core cycle. Downstream +consumers must fence capture and permanently suppress the exact removed native +keys before source removal. Treat this as logical archive removal, not a claim of +forensic disk erasure or deletion from GitHub. diff --git a/docs/commands.md b/docs/commands.md index aa4f11ef..56525b26 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -43,6 +43,7 @@ These work on every command. | Command | Purpose | Docs | | --- | --- | --- | | `gitcrawl sync owner/repo [--state --since --numbers --limit --include-comments --include-pr-details --with pr-details --graphql-history --force --progress-file --json]` | Sync issues and PRs from GitHub into local SQLite | [Sync](/sync/) | +| `gitcrawl analytics owner/repo [--apply --enrich --watch --once --json]` | Audit/repair source publication dates, enrich stable identities, and maintain GraphQL updates | [Analytics source preparation](/analytics-source/) | | `gitcrawl sync-failures owner/repo [--include-resolved --limit N --json]` | List failed issue, comment, and PR hydration attempts and optional resolved history | [Sync](/sync/#hydration-depth) | | `gitcrawl coverage [owner/repo \| --repos owner/a,owner/b] [--min-missing-pr-details N --json]` | Report archive, PR-detail, and enrichment coverage/freshness | — | | `gitcrawl fill-pr-details owner/repo [--limit --order --batch-size --reserve-rate-limit --include-comments --json-progress --json]` | Hydrate locally missing pull request detail rows in bounded batches | — | diff --git a/docs/sync.md b/docs/sync.md index f3cc291d..828655b1 100644 --- a/docs/sync.md +++ b/docs/sync.md @@ -369,3 +369,10 @@ gitcrawl sync owner/repo --numbers "$NUMS" --with pr-details - [Refresh and embed](/refresh-and-embed/) — the wrapper that runs sync, embed, and cluster end to end - [gh shim migration](/gh-shim/) — Octopool owns pooled `gh` reads now - [Portable stores](/portable-stores/) — sharing the synced cache across machines + +## Analytics source preparation + +The [analytics source command](/analytics-source/) repairs retained review publication +timestamps without rewriting raw evidence and maintains a full-history archive +through incremental GraphQL collection. Its actor IDs and coverage receipts are +source evidence for downstream analytics; account classifications remain derived. diff --git a/internal/cli/analytics.go b/internal/cli/analytics.go new file mode 100644 index 00000000..8d29a5bd --- /dev/null +++ b/internal/cli/analytics.go @@ -0,0 +1,392 @@ +package cli + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "github.com/openclaw/gitcrawl/internal/config" + gh "github.com/openclaw/gitcrawl/internal/github" + "github.com/openclaw/gitcrawl/internal/store" + "github.com/openclaw/gitcrawl/internal/syncer" + "io" + "os" + "path/filepath" + "sync" + "time" +) + +func (a *App) analyticsClient(ctx context.Context, cfg config.Config) (*gh.Client, error) { + token := a.resolveGitHubToken(ctx, cfg) + var provider func(context.Context) (string, error) + var e error + if a.githubTokenCommand != nil { + provider, e = githubTokenProvider(*a.githubTokenCommand) + if e != nil { + return nil, e + } + } + if provider != nil { + fetch := provider + var mu sync.Mutex + var cached string + var expires time.Time + provider = func(ctx context.Context) (string, error) { + mu.Lock() + defer mu.Unlock() + if cached != "" && time.Now().Before(expires) { + return cached, nil + } + value, e := fetch(ctx) + if e != nil { + return "", e + } + cached = value + expires = time.Now().Add(45 * time.Minute) + return value, nil + } + a.analyticsTokenProvider = provider + } + if provider == nil && token.Value == "" { + return nil, fmt.Errorf("missing GitHub credential") + } + return gh.New(gh.Options{Token: token.Value, TokenProvider: provider, BaseURL: githubBaseURL(), RateLimit: a.observeGitHubRateLimit(ctx), RateLimitReserve: analyticsCoreReserve}), nil +} +func (a *App) runAnalytics(ctx context.Context, args []string) error { + for _, arg := range args { + if arg == "--help" || arg == "-h" { + _, err := fmt.Fprintln(a.Stdout, "Usage: gitcrawl [--config SOURCE_CONFIG] [--github-token-command TOKEN_HELPER] analytics owner/repo [--status|--apply|--enrich] [--watch|--once] [--json]\nWithout action flags: read-only publication audit. --status reads bounded failure/recovery status; --apply repairs retained timestamps; --enrich collects actor evidence; --watch maintains GraphQL updates.") + return err + } + } + + fs := flag.NewFlagSet("analytics", flag.ContinueOnError) + fs.SetOutput(io.Discard) + apply := fs.Bool("apply", false, "apply source publication corrections") + enrich := fs.Bool("enrich", false, "collect missing provider identities and actor profiles") + watch := fs.Bool("watch", false, "maintain GraphQL updates every two minutes") + once := fs.Bool("once", false, "run one GraphQL update cycle") + status := fs.Bool("status", false, "read bounded collection, failure and recovery status") + fs.Bool("json", false, "JSON output") + if e := fs.Parse(normalizeCommandArgs(args, nil)); e != nil { + return e + } + if fs.NArg() != 1 { + return fmt.Errorf("analytics requires owner/repo") + } + owner, repo, e := parseOwnerRepo(fs.Arg(0)) + if e != nil { + return e + } + a.format = FormatJSON + cfg, e := config.LoadRuntime(a.configPath) + if e != nil { + return e + } + if *status { + if *apply || *enrich || *watch || *once { + return fmt.Errorf("--status cannot be combined with collection actions") + } + rt, err := a.openLocalRuntimeReadOnly(ctx) + if err != nil { + return err + } + defer rt.Store.Close() + result, err := rt.Store.AnalyticsIntegrityStatus(ctx, owner+"/"+repo) + if err != nil { + return err + } + return a.writeOutput("analytics_status", result, false) + } + if !*apply && !*enrich && !*watch && !*once { + rt, e := a.openLocalRuntimeReadOnly(ctx) + if e != nil { + return e + } + defer rt.Store.Close() + r, e := rt.Store.RepairPublication(ctx, false) + if e != nil { + return e + } + return a.writeOutput("analytics_repair_dry_run", r, false) + } + lock, e := os.OpenFile(filepath.Join(filepath.Dir(cfg.DBPath), "runner.lock"), os.O_CREATE|os.O_RDWR, 0600) + if e != nil { + return e + } + defer lock.Close() + if e = lockPortableFile(lock); e != nil { + return fmt.Errorf("collector ownership lock busy: %w", e) + } + rt, e := a.openLocalRuntime(ctx) + if e != nil { + return e + } + defer rt.Store.Close() + if *apply { + r, e := rt.Store.RepairPublication(ctx, true) + if e != nil { + return e + } + if e = a.writeOutput("analytics_repair", r, false); e != nil { + return e + } + } + if !*enrich && !*watch && !*once { + return nil + } + client, e := a.analyticsClient(ctx, cfg) + if e != nil { + return e + } + if *watch || *once { + if e = rt.Store.SeedAnalyticsNodes(ctx, true); e != nil { + return e + } + var existing int + if e = rt.Store.DB().QueryRowContext(ctx, "SELECT count(*) FROM analytics_coverage WHERE repository=?", owner+"/"+repo).Scan(&existing); e != nil { + return e + } + if existing == 0 { + b, err := os.ReadFile(filepath.Join(filepath.Dir(a.configPath), "status.json")) + if err == nil { + var status struct { + Phase string `json:"phase"` + Discovery []struct { + Kind string `json:"kind"` + Done int `json:"done"` + Total int `json:"total"` + Updated string `json:"updated_at"` + } `json:"discovery"` + } + if json.Unmarshal(b, &status) == nil && status.Phase == "complete" && len(status.Discovery) == 2 { + through := "" + issues, prs := 0, 0 + valid := true + for _, d := range status.Discovery { + valid = valid && d.Done == 1 + if through == "" || d.Updated < through { + through = d.Updated + } + if d.Kind == "issues" { + issues = d.Total + } else if d.Kind == "pullRequests" { + prs = d.Total + } else { + valid = false + } + } + if valid && issues > 0 && prs > 0 { + if e = rt.Store.SaveAnalyticsCoverage(ctx, owner+"/"+repo, through, issues, prs); e != nil { + return e + } + } + } + } + } + } + // Independent guarded clients permit disjoint evidence batches to overlap; + // each preserves the normal quota reservation and uses the same cached token. + actorClients := make([]*gh.Client, 8) + for i := range actorClients { + token := a.resolveGitHubToken(ctx, cfg) + actorClients[i] = gh.New(gh.Options{Token: token.Value, TokenProvider: a.analyticsTokenProvider, BaseURL: githubBaseURL(), RateLimit: a.observeGitHubRateLimit(ctx), RateLimitReserve: analyticsCoreReserve}) + } + parallelWatch := *watch && *enrich + if parallelWatch { + pollCtx, cancel := context.WithCancel(ctx) + done := make(chan struct{}) + defer func() { cancel(); <-done }() + go func() { + defer close(done) + for { + nextPoll := time.Now().Add(analyticsPollInterval) + pollErr := a.analyticsCycle(pollCtx, rt.Store, client, owner, repo) + a.analyticsUpdateLog(pollErr) + select { + case <-pollCtx.Done(): + return + case <-time.After(time.Until(nextPoll)): + } + } + }() + } + if *enrich { + for _, profiles := range []bool{true, false} { + if e = rt.Store.SeedAnalyticsNodes(ctx, profiles); e != nil { + return e + } + } + e = maintainAnalyticsEnrichment(ctx, parallelWatch, 2*time.Minute, func() error { + for _, profiles := range []bool{true, false, true} { + for { + var ids []string + if profiles { + ids, e = rt.Store.AnalyticsProfileNodes(ctx, 800) + } else { + ids, e = rt.Store.AnalyticsIdentityNodes(ctx, 800) + } + if e != nil { + return e + } + if len(ids) == 0 { + break + } + var group sync.WaitGroup + var failures []error + var failureMu sync.Mutex + for offset := 0; offset < len(ids); offset += 100 { + part := append([]string(nil), ids[offset:min(offset+100, len(ids))]...) + worker := actorClients[offset/100] + group.Add(1) + go func() { + defer group.Done() + nodes, err := worker.AnalyticsNodes(ctx, part, profiles) + if err == nil { + at := time.Now().UTC().Format(time.RFC3339Nano) + if profiles { + err = rt.Store.SaveActorProfiles(ctx, nodes, at) + } else { + err = rt.Store.SaveActorEvidence(ctx, nodes, at) + } + } + if err != nil { + failureMu.Lock() + failures = append(failures, err) + failureMu.Unlock() + } + }() + } + group.Wait() + if len(failures) > 0 { + fmt.Fprintf(a.Stderr, "{\"event\":\"actor_enrichment_retry\",\"failed_batches\":%d}\n", len(failures)) + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(30 * time.Second): + } + continue + } + + fmt.Fprintf(a.Stderr, "{\"event\":\"actor_enrichment\",\"profiles\":%t,\"nodes\":%d}\n", profiles, len(ids)) + + } + } + return nil + }) + if e != nil { + return e + } + } + + if parallelWatch { + <-ctx.Done() + return ctx.Err() + } + for *watch || *once { + nextPoll := time.Now().Add(analyticsPollInterval) + e = a.analyticsCycle(ctx, rt.Store, client, owner, repo) + if e != nil { + if !*watch { + return e + } + fmt.Fprintf(a.Stderr, "{\"event\":\"github_update_failed\",\"error\":%q}\n", e.Error()) + } else { + fmt.Fprintln(a.Stderr, "{\"event\":\"github_update_complete\"}") + } + if !*watch { + return nil + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(time.Until(nextPoll)): + } + } + return nil +} + +type updateCheckpoint struct { + Started string `json:"started"` + Since string `json:"since"` + Kind int `json:"kind"` + Cursor string `json:"cursor"` + Issues int `json:"issues"` + PRs int `json:"prs"` +} + +// Smaller requests prevent high-fanout conversation queries exhausting GitHub's +// execution deadline. Split a persistently failing transient batch without +// accepting partial conversation evidence or changing transports. +func (a *App) syncAnalyticsBatch(ctx context.Context, s *store.Store, owner, repo string, numbers []int, operation string) error { + cfg, err := config.LoadRuntime(a.configPath) + if err != nil { + return err + } + token := a.resolveGitHubToken(ctx, cfg) + reserve := analyticsCoreReserve + var responseLimit int64 + var responseBytes, points, providerMillis int64 + var reporter gh.Reporter + if operation == "review_state" { + reserve = analyticsReviewReserve + responseLimit = 32 << 20 + // This call owns its client, history session and reporter. Synchronous + // callbacks are never shared with the other analyticsNumbers workers. + var effectiveRemaining, effectiveReset int + reporter = func(message string) { + var bytes, callNumber, millis int64 + if _, err := fmt.Sscanf(message, "[github] graphql bytes %d", &bytes); err == nil { + responseBytes += bytes + return + } + if _, err := fmt.Sscanf(message, "[github] graphql timing %d %d", &callNumber, &millis); err == nil { + providerMillis += millis + return + } + var providerRemaining, providerReset int + if _, err := fmt.Sscanf(message, "[github] graphql quota provider_remaining %d provider_reset %d effective_remaining %d effective_reset %d", &providerRemaining, &providerReset, &effectiveRemaining, &effectiveReset); err == nil { + return + } + var call, cost, remaining, reset int + if _, err := fmt.Sscanf(message, "[github] graphql cost %d %d remaining %d reset %d", &call, &cost, &remaining, &reset); err == nil { + points += int64(cost) + fmt.Fprintf(a.Stderr, "{\"event\":\"review_state_cost\",\"at\":%q,\"points\":%d,\"remaining\":%d,\"reset_unix\":%d,\"provider_remaining\":%d,\"provider_reset_unix\":%d}\n", time.Now().UTC().Format(time.RFC3339Nano), cost, effectiveRemaining, effectiveReset, remaining, reset) + } + } + } + client := gh.New(gh.Options{Token: token.Value, TokenProvider: a.analyticsTokenProvider, BaseURL: githubBaseURL(), RateLimit: a.observeGitHubRateLimit(ctx), RateLimitReserve: reserve, GraphQLResponseLimit: responseLimit, GraphQLQuotaGuard: operation == "review_state"}) + // The watch owner has already validated and opened this store. Reopening it + // for every two threads repeats full-archive migration audits and serializes + // otherwise independent network work. Native transactions still own writes. + stats, err := syncer.New(client, s).Sync(ctx, syncer.Options{Owner: owner, Repo: repo, GraphQLHistory: true, ReviewStateOnly: operation == "review_state", ReceiptOperation: operation, State: "all", Numbers: numbers, IncludeComments: true, IncludePRMetadata: true, Reporter: reporter}) + if metrics, ok := ctx.Value(analyticsMetricsKey{}).(*analyticsWorkMetrics); ok { + metrics.bytes.Add(responseBytes) + metrics.points.Add(points) + metrics.providerMillis.Add(providerMillis) + metrics.dbMillis.Add(stats.PersistMillis) + metrics.attempted.Add(int64(len(numbers))) + if err == nil { + metrics.recovered.Add(int64(stats.ThreadsSynced)) + } + } + return err +} + +// A drained queue is not the end of a watch: new actors and stale profiles +// become eligible later. Initial source scanning happens outside this loop. +func maintainAnalyticsEnrichment(ctx context.Context, watch bool, interval time.Duration, drain func() error) error { + for { + if err := drain(); err != nil { + return err + } + if !watch { + return nil + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(interval): + } + } +} diff --git a/internal/cli/analytics_integrity.go b/internal/cli/analytics_integrity.go new file mode 100644 index 00000000..b1dfc1c8 --- /dev/null +++ b/internal/cli/analytics_integrity.go @@ -0,0 +1,538 @@ +package cli + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "sync" + "sync/atomic" + "time" + + gh "github.com/openclaw/gitcrawl/internal/github" + "github.com/openclaw/gitcrawl/internal/store" +) + +var errAnalyticsIncomplete = errors.New("analytics coverage remains incomplete") + +const ( + analyticsPollInterval = 2 * time.Minute + analyticsCoreReserve = 1500 + // Keep another 1500 points available to ordinary capture above its floor. + analyticsReviewReserve = 3000 + analyticsReviewWorkers = 32 + analyticsReviewBatch = 8 + analyticsReviewWave = analyticsReviewWorkers * analyticsReviewBatch + analyticsReviewWaveBytes = 16 << 20 +) + +type analyticsMetricsKey struct{} +type analyticsWorkMetrics struct { + busy, maxBusy, bytes, points, attempted, recovered, providerMillis, dbMillis, workerMillis atomic.Int64 +} + +func (m *analyticsWorkMetrics) enter() func() { + start := time.Now() + busy := m.busy.Add(1) + for old := m.maxBusy.Load(); busy > old; old = m.maxBusy.Load() { + if m.maxBusy.CompareAndSwap(old, busy) { + break + } + } + return func() { m.workerMillis.Add(time.Since(start).Milliseconds()); m.busy.Add(-1) } +} + +func (a *App) analyticsUpdateLog(err error) { + event := "github_update_complete" + fields := map[string]any{"at": time.Now().UTC().Format(time.RFC3339Nano)} + if err != nil { + event = "github_update_failed" + if errors.Is(err, errAnalyticsIncomplete) { + event = "github_coverage_pending" + fields["error"] = err.Error() + } else { + class, message, _ := gh.HistoryFailureDetails(err) + fields["error_class"] = class + fields["error"] = message + } + } + fields["event"] = event + encoded, _ := json.Marshal(fields) + fmt.Fprintln(a.Stderr, string(encoded)) +} + +func migrateAnalyticsLanes(ctx context.Context, s *store.Store, repository string) error { + // runAnalytics initializes coverage from a verified phase=complete discovery + // receipt before any cycle. A partial checkpoint alone cannot certify a + // historical baseline; missing baseline evidence must remain an error. + marker := "independent_lanes:" + repository + value, err := s.AnalyticsState(ctx, marker) + if err != nil || value != "" { + return err + } + value, err = s.AnalyticsState(ctx, "updates:"+repository) + if err != nil { + return err + } + var legacy updateCheckpoint + if value != "" { + if err = json.Unmarshal([]byte(value), &legacy); err != nil { + return err + } + } + through, err := s.AnalyticsState(ctx, "through:"+repository) + if err != nil { + return err + } + if through == "" { + if err = s.DB().QueryRowContext(ctx, "SELECT through FROM analytics_coverage WHERE repository=?", repository).Scan(&through); err != nil { + return fmt.Errorf("verified historical coverage watermark required: %w", err) + } + } + var verifiedThrough string + if err = s.DB().QueryRowContext(ctx, "SELECT through FROM analytics_coverage WHERE repository=?", repository).Scan(&verifiedThrough); err != nil { + return err + } + if _, err = time.Parse(time.RFC3339Nano, verifiedThrough); err != nil { + return fmt.Errorf("invalid verified historical watermark: %w", err) + } + return s.WithTx(ctx, func(tx *store.Store) error { + // through retains the previous verified baseline even when a transient + // post-upgrade failure temporarily clears complete before migration. + if err := tx.SetAnalyticsState(ctx, "core_baseline_verified:"+repository, "1"); err != nil { + return err + } + for i, kind := range []string{"issues", "pullRequests"} { + cp := updateCheckpoint{} + laneThrough := through + if legacy.Started != "" { + if legacy.Kind > i { + laneThrough = legacy.Started + total := legacy.Issues + if i == 1 { + total = legacy.PRs + } + encodedTotal, _ := json.Marshal(total) + if err := tx.SetAnalyticsState(ctx, "total:"+repository+":"+kind, string(encodedTotal)); err != nil { + return err + } + } else { + cp = legacy + cp.Kind = i + if legacy.Kind < i { + cp.Cursor = "" + } + } + } + encoded := "" + if cp.Started != "" { + b, _ := json.Marshal(cp) + encoded = string(b) + } + if err := tx.SetAnalyticsState(ctx, "updates:"+repository+":"+kind, encoded); err != nil { + return err + } + if err := tx.SetAnalyticsState(ctx, "through:"+repository+":"+kind, laneThrough); err != nil { + return err + } + } + // The original receipt/checkpoint is retained for historical evidence. + return tx.SetAnalyticsState(ctx, marker, time.Now().UTC().Format(time.RFC3339Nano)) + }) +} + +func (a *App) analyticsCycle(ctx context.Context, s *store.Store, c *gh.Client, owner, repo string) error { + nextCore := time.Now().Add(analyticsPollInterval) + repository := owner + "/" + repo + if err := migrateAnalyticsLanes(ctx, s, repository); err != nil { + return err + } + // Ordinary capture and its retry obligations always go first. Historical + // review enrichment uses only the remaining time before the next core poll. + due, err := s.DueAnalyticsRetries(ctx, repository, time.Now().UTC().Format(time.RFC3339Nano), 8, "graphql_history") + if err != nil { + return err + } + if err = a.analyticsNumbers(ctx, s, owner, repo, due, false, "graphql_history"); err != nil { + return err + } + var failures []error + for i, kind := range []string{"issues", "pullRequests"} { + if err = a.analyticsLane(ctx, s, c, owner, repo, kind, i); err != nil { + failures = append(failures, err) + } + if ctx.Err() != nil { + return errors.Join(append(failures, ctx.Err())...) + } + } + var through string + baseline, err := s.AnalyticsState(ctx, "core_baseline_verified:"+repository) + if err != nil { + return err + } + complete := baseline == "1" && len(failures) == 0 + var totals [2]int + // Preserve last verified totals until this lane obtains a new provider count. + _ = s.DB().QueryRowContext(ctx, "SELECT issues,pull_requests FROM analytics_coverage WHERE repository=?", repository).Scan(&totals[0], &totals[1]) + for i, kind := range []string{"issues", "pullRequests"} { + at, e := s.AnalyticsState(ctx, "through:"+repository+":"+kind) + if e != nil { + return e + } + if through == "" || at < through { + through = at + } + value, e := s.AnalyticsState(ctx, "total:"+repository+":"+kind) + if e != nil { + return e + } + if value != "" { + if e = json.Unmarshal([]byte(value), &totals[i]); e != nil { + return e + } + } + } + outstanding, err := s.AnalyticsCoreOutstanding(ctx, repository) + if err != nil { + return err + } + complete = complete && outstanding == 0 + if err = s.WithTx(ctx, func(tx *store.Store) error { + if e := tx.SaveAnalyticsCoverage(ctx, repository, through, totals[0], totals[1]); e != nil { + return e + } + if e := tx.SetAnalyticsCoverageComplete(ctx, repository, complete); e != nil { + return e + } + return tx.SetAnalyticsState(ctx, "through:"+repository, through) + }); err != nil { + return err + } + reviewErr := a.analyticsReviewRecovery(ctx, s, c, owner, repo, nextCore.Add(-5*time.Second)) + if len(failures) > 0 { + return errors.Join(append(failures, reviewErr)...) + } + if !complete { + return errors.Join(fmt.Errorf("%w: core_unresolved=%d", errAnalyticsIncomplete, outstanding), reviewErr) + } + return reviewErr +} + +// Fill the time formerly spent idle with bounded, quota-checked waves through +// the existing executor. Each scan chunk and item receipt remains durable. +func (a *App) analyticsReviewRecovery(ctx context.Context, s *store.Store, c *gh.Client, owner, repo string, deadline time.Time) (resultErr error) { + window, cancel := context.WithDeadline(ctx, deadline) + defer cancel() + yield := func(err error) error { + if ctx.Err() == nil && window.Err() == context.DeadlineExceeded && analyticsCancellationOnly(err) { + fmt.Fprintf(a.Stderr, "{\"event\":\"review_state_yield\",\"at\":%q}\n", time.Now().UTC().Format(time.RFC3339Nano)) + return nil + } + return err + } + repository := owner + "/" + repo + var progress store.ReviewStateRecovery + bytesPerItem := int64(64 << 10) + haveProgress, quotaBlocked := false, false + defer func() { + if !haveProgress { + return + } + // Cancellation cannot erase the last committed scan or completed items. + receiptCtx, stop := context.WithTimeout(context.Background(), 10*time.Second) + defer stop() + if err := s.SaveReviewStateCoverage(receiptCtx, repository, progress); err != nil { + resultErr = errors.Join(resultErr, err) + return + } + var pending int + if err := s.DB().QueryRowContext(receiptCtx, "SELECT pending_items FROM analytics_review_state_coverage WHERE repository=?", repository).Scan(&pending); err != nil { + resultErr = errors.Join(resultErr, err) + return + } + encoded, _ := json.Marshal(map[string]any{"event": "review_state_progress", "at": time.Now().UTC().Format(time.RFC3339Nano), "scanned": progress.Scanned, "ceiling": progress.Ceiling, "queued": progress.Queued, "pending_items": pending, "scan_complete": progress.Done, "complete": progress.Done && pending == 0}) + fmt.Fprintln(a.Stderr, string(encoded)) + }() + for time.Until(deadline) > 5*time.Second { + if err := window.Err(); err != nil { + return yield(err) + } + next, err := s.SeedReviewStateRecovery(window, repository, 5000) + if err != nil { + return yield(err) + } + progress, haveProgress = next, true + // Persist progress even if quota is exhausted or the process is stopped. + if err = s.SaveReviewStateCoverage(window, repository, progress); err != nil { + return yield(err) + } + if quotaBlocked { + if progress.Done { + return nil + } + continue + } + due, err := s.DueReviewStateWork(window, repository, time.Now().UTC().Format(time.RFC3339Nano), analyticsReviewWave) + if err != nil { + return yield(err) + } + if len(due) == 0 { + if progress.Done { + return nil + } + continue + } + observedQuota, rawQuota, err := c.AnalyticsRateLimit(window) + if err != nil { + if window.Err() != nil { + return yield(err) + } + class, message, _ := gh.HistoryFailureDetails(err) + fmt.Fprintf(a.Stderr, "{\"event\":\"review_state_quota_deferred\",\"at\":%q,\"error_class\":%q,\"error\":%q}\n", time.Now().UTC().Format(time.RFC3339Nano), class, message) + quotaBlocked = true + continue + } + budget, quota, err := analyticsReviewBudget([]gh.RateLimitSnapshot{observedQuota}, time.Now()) + if err != nil { + fmt.Fprintf(a.Stderr, "{\"event\":\"review_state_quota_deferred\",\"at\":%q,\"error\":%q}\n", time.Now().UTC().Format(time.RFC3339Nano), err.Error()) + quotaBlocked = true + continue + } + budget = min(budget, int(analyticsReviewWaveBytes/bytesPerItem)) + if budget > 0 && budget < len(due) { + // Retry fairness is relative to actual admitted work, including + // a smaller point/byte budget, not the maximum wave size. + due, err = s.DueReviewStateWork(window, repository, time.Now().UTC().Format(time.RFC3339Nano), budget) + if err != nil { + return yield(err) + } + } + encoded, _ := json.Marshal(map[string]any{"event": "review_state_quota", "at": time.Now().UTC().Format(time.RFC3339Nano), "limit": quota.Limit, "remaining": quota.Remaining, "reset_at": quota.ResetAt, "provider_remaining": rawQuota.Remaining, "provider_reset_at": rawQuota.ResetAt, "reserve": analyticsReviewReserve, "wave_items": min(len(due), budget)}) + fmt.Fprintln(a.Stderr, string(encoded)) + if budget == 0 { + quotaBlocked = true + continue + } + // Item-level reserve failures are durably queued and absorbed by + // analyticsIsolatedBatch; the next wave reprobes quota and keeps scanning. + // Remaining errors include unrecorded storage failures, not safe deferrals. + metrics := &analyticsWorkMetrics{} + waveStarted := time.Now() + err = a.analyticsNumbers(context.WithValue(window, analyticsMetricsKey{}, metrics), s, owner, repo, due[:min(len(due), budget)], false, "review_state") + if metrics.attempted.Load() > 0 { + bytesPerItem = max(bytesPerItem, metrics.bytes.Load()/metrics.attempted.Load()) + } + measured, _ := json.Marshal(map[string]any{"event": "review_state_wave", "at": time.Now().UTC().Format(time.RFC3339Nano), "elapsed_ms": time.Since(waveStarted).Milliseconds(), "workers": analyticsReviewWorkers, "max_busy_workers": metrics.maxBusy.Load(), "busy_worker_ms": metrics.workerMillis.Load(), "attempted_items": metrics.attempted.Load(), "recovered_items": metrics.recovered.Load(), "response_bytes": metrics.bytes.Load(), "reported_points": metrics.points.Load(), "provider_ms": metrics.providerMillis.Load(), "db_ms": metrics.dbMillis.Load(), "byte_budget": analyticsReviewWaveBytes}) + fmt.Fprintln(a.Stderr, string(measured)) + if err != nil { + return yield(err) + } + } + return ctx.Err() +} + +// Joined storage/receipt errors must never disappear behind a window timeout. +func analyticsCancellationOnly(err error) bool { + if err == nil { + return false + } + if joined, ok := err.(interface{ Unwrap() []error }); ok { + for _, child := range joined.Unwrap() { + if !analyticsCancellationOnly(child) { + return false + } + } + return true + } + if wrapped, ok := err.(interface{ Unwrap() error }); ok { + return analyticsCancellationOnly(wrapped.Unwrap()) + } + return err == context.Canceled || err == context.DeadlineExceeded +} + +func analyticsReviewBudget(limits []gh.RateLimitSnapshot, now time.Time) (int, gh.RateLimitSnapshot, error) { + for _, quota := range limits { + if quota.Resource != "graphql" { + continue + } + if quota.Limit <= 0 || quota.Remaining < 0 || !quota.ResetAt.After(now) { + return 0, quota, fmt.Errorf("fresh GraphQL quota required for review recovery") + } + // A conservative admission margin limits in-flight overshoot. Native + // request guards recheck actual remaining quota before every request. + return min(analyticsReviewWave, max(0, quota.Remaining-analyticsReviewReserve)/32), quota, nil + } + return 0, gh.RateLimitSnapshot{}, fmt.Errorf("GraphQL quota unavailable for review recovery") +} + +func (a *App) analyticsNumbers(ctx context.Context, s *store.Store, owner, repo string, numbers []int, discovery bool, operation string) (resultErr error) { + var err error + numbers, err = s.FilterExcludedNumbers(ctx, owner+"/"+repo, numbers) + if err != nil { + return err + } + var wg sync.WaitGroup + var failures []error + // Every return, including cancelled admission, waits for durable receipts. + defer func() { + wg.Wait() + resultErr = errors.Join(append(failures, resultErr)...) + }() + workers, batchSize := 8, 2 + if operation == "review_state" { + workers, batchSize = analyticsReviewWorkers, analyticsReviewBatch + } + slots := make(chan struct{}, workers) + var mu sync.Mutex + for i := 0; i < len(numbers); i += batchSize { + if err := ctx.Err(); err != nil { + return err + } + var part []int + for _, n := range numbers[i:min(i+batchSize, len(numbers))] { + if discovery { + // Review-only recovery must not defer a newly discovered core edit. + // If this core attempt fails, it creates graphql_history retry state + // and every later overlap skips it until the bounded scheduler retries. + deferred, e := s.AnalyticsItemQueued(ctx, owner+"/"+repo, n) + if e != nil { + return e + } + if deferred { + continue + } + } + part = append(part, n) + } + if len(part) == 0 { + continue + } + select { + case slots <- struct{}{}: + case <-ctx.Done(): + return ctx.Err() + } + if err := ctx.Err(); err != nil { + <-slots + return err + } + wg.Add(1) + go func(part []int) { + defer wg.Done() + defer func() { <-slots }() + if metrics, ok := ctx.Value(analyticsMetricsKey{}).(*analyticsWorkMetrics); ok { + defer metrics.enter()() + } + if e := a.analyticsIsolatedBatch(ctx, s, owner, repo, part, operation); e != nil { + mu.Lock() + failures = append(failures, e) + mu.Unlock() + } + }(part) + } + return nil +} + +func (a *App) analyticsIsolatedBatch(ctx context.Context, s *store.Store, owner, repo string, numbers []int, operation string) error { + bounded, cancel := context.WithTimeout(ctx, 2*time.Minute) + err := a.syncAnalyticsBatch(bounded, s, owner, repo, numbers, operation) + cancel() + if err == nil { + return nil + } + if ctx.Err() != nil { + return err + } + // Persisted failure receipts allow splitting all rejected batches, including + // partial/invalid connections. A poison item cannot block its healthy peer. + if len(numbers) > 1 { + var failures []error + for _, n := range numbers { + if e := a.analyticsIsolatedBatch(ctx, s, owner, repo, []int{n}, operation); e != nil { + failures = append(failures, e) + } + } + return errors.Join(failures...) + } + queued, e := s.AnalyticsItemQueued(ctx, owner+"/"+repo, numbers[0], operation) + if e != nil { + return errors.Join(err, e) + } + if !queued { + return err + } // Never advance past an unrecorded failure. + return nil +} + +func (a *App) analyticsLane(ctx context.Context, s *store.Store, c *gh.Client, owner, repo, kind string, index int) error { + repository := owner + "/" + repo + key := "updates:" + repository + ":" + kind + value, err := s.AnalyticsState(ctx, key) + if err != nil { + return err + } + var cp updateCheckpoint + if value != "" { + if err = json.Unmarshal([]byte(value), &cp); err != nil { + return err + } + } + if cp.Started == "" { + through, e := s.AnalyticsState(ctx, "through:"+repository+":"+kind) + if e != nil { + return e + } + at, e := time.Parse(time.RFC3339Nano, through) + if e != nil { + return e + } + cp = updateCheckpoint{Started: time.Now().UTC().Format(time.RFC3339Nano), Since: at.Add(-5 * time.Minute).Format(time.RFC3339Nano), Kind: index} + } + since, err := time.Parse(time.RFC3339Nano, cp.Since) + if err != nil { + return err + } + for pages := 0; pages < 2; pages++ { + started := time.Now().UTC().Format(time.RFC3339Nano) + page, e := c.UpdatedNumbers(ctx, owner, repo, kind, cp.Cursor, since) + attempt := store.AnalyticsAttempt{Repository: repository, Operation: "discover_" + kind, StartedAt: started, FinishedAt: time.Now().UTC().Format(time.RFC3339Nano), Status: "success", Evidence: json.RawMessage(`{}`)} + if e != nil { + attempt.Status = "failed" + attempt.ErrorClass, attempt.ErrorText, attempt.Evidence = gh.HistoryFailureDetails(e) + } + receiptCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + writeErr := s.RecordAnalyticsAttempt(receiptCtx, attempt) + cancel() + if e != nil || writeErr != nil { + return errors.Join(e, writeErr) + } + if e = a.analyticsNumbers(ctx, s, owner, repo, page.Numbers, true, "graphql_history"); e != nil { + return e + } + done := !page.More || (!page.Oldest.IsZero() && page.Oldest.Before(since)) + if !done { + cp.Cursor = page.Cursor + } + if e = s.WithTx(ctx, func(tx *store.Store) error { + count, _ := json.Marshal(page.Total) + if e := tx.SetAnalyticsState(ctx, "total:"+repository+":"+kind, string(count)); e != nil { + return e + } + if done { + if e := tx.SetAnalyticsState(ctx, "through:"+repository+":"+kind, cp.Started); e != nil { + return e + } + return tx.SetAnalyticsState(ctx, key, "") + } + encoded, _ := json.Marshal(cp) + return tx.SetAnalyticsState(ctx, key, string(encoded)) + }); e != nil { + return e + } + fmt.Fprintf(a.Stderr, "{\"event\":\"github_update_page\",\"at\":%q,\"kind\":%q,\"threads\":%d}\n", time.Now().UTC().Format(time.RFC3339Nano), kind, len(page.Numbers)) + if done { + return nil + } + } + return nil +} diff --git a/internal/cli/analytics_integrity_test.go b/internal/cli/analytics_integrity_test.go new file mode 100644 index 00000000..39122e9e --- /dev/null +++ b/internal/cli/analytics_integrity_test.go @@ -0,0 +1,284 @@ +package cli + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync/atomic" + "testing" + "time" + + gh "github.com/openclaw/gitcrawl/internal/github" + "github.com/openclaw/gitcrawl/internal/store" +) + +func TestAnalyticsPoisonItemAndDiscoveryLaneDoNotStarvePeers(t *testing.T) { + for _, brokenDiscovery := range []bool{false, true} { + t.Run(fmt.Sprint(brokenDiscovery), func(t *testing.T) { + ctx := context.Background() + dir := t.TempDir() + path := filepath.Join(dir, "archive.db") + s, err := store.Open(ctx, path) + if err != nil { + t.Fatal(err) + } + defer func() { s.Close() }() + at := time.Now().UTC().Add(-time.Minute).Format(time.RFC3339Nano) + baseline := time.Now().UTC().Add(-10 * time.Minute).Format(time.RFC3339Nano) + if err = s.SetAnalyticsState(ctx, "through:fixture/repo", baseline); err != nil { + t.Fatal(err) + } + if err = s.SaveAnalyticsCoverage(ctx, "fixture/repo", baseline, 1, 1); err != nil { + t.Fatal(err) + } + var broken atomic.Bool + broken.Store(true) + var revised atomic.Bool + conn := func(nodes ...any) map[string]any { + if nodes == nil { + nodes = []any{} + } + return map[string]any{"nodes": nodes, "totalCount": len(nodes), "pageInfo": map[string]any{"hasNextPage": false, "endCursor": "end"}} + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + if r.URL.Path == "/rate_limit" { + fmt.Fprint(w, `{"resources":{"graphql":{"limit":50000,"remaining":49000,"reset":4102444800},"core":{"limit":50000,"remaining":49000,"reset":4102444800}}}`) + return + } + if r.URL.Path != "/graphql" { + t.Errorf("unexpected fallback: %s", r.URL.Path) + http.Error(w, "no fallback", 400) + return + } + var req struct{ Query string } + if decodeErr := json.NewDecoder(r.Body).Decode(&req); decodeErr != nil { + t.Error(decodeErr) + return + } + data := map[string]any{"rateLimit": map[string]any{"cost": 1, "remaining": 48000, "resetAt": "2099-01-01T00:00:00Z"}} + if strings.Contains(req.Query, "orderBy") { + kind, n := "issues", 1 + if strings.Contains(req.Query, "pullRequests(first:") { + kind, n = "pullRequests", 2 + } + page := conn(map[string]any{"number": n, "updatedAt": at}) + if brokenDiscovery && broken.Load() && n == 1 { + delete(page, "pageInfo") + } + data["repository"] = map[string]any{kind: page} + } else if strings.Contains(req.Query, "issueOrPullRequest") { + n, typ := 1, "Issue" + if strings.Contains(req.Query, "number:2)") { + n, typ = 2, "PullRequest" + } + node := map[string]any{"id": fmt.Sprint("node-", n), "fullDatabaseId": fmt.Sprint(n), "__typename": typ, "number": n, "title": "fixture", "body": "retained body", "state": "OPEN", "createdAt": "2026-01-01T00:00:00Z", "updatedAt": at, "url": fmt.Sprintf("https://github.com/fixture/repo/issues/%d", n), "repository": map[string]any{"nameWithOwner": "fixture/repo"}, "author": map[string]any{"id": "actor", "login": "fixture", "__typename": "User"}, "labels": conn(), "assignees": conn(), "comments": conn()} + if revised.Load() && n == 2 { + node["body"] = "fresh core body" + } + if n == 2 { + node["reviews"] = conn() + node["reviewThreads"] = conn() + } + if !brokenDiscovery && broken.Load() && n == 1 { + node["comments"].(map[string]any)["totalCount"] = 1 + } + data["repository"] = map[string]any{"id": "repo", "databaseId": 1, "nameWithOwner": "fixture/repo", "n0": node} + } + json.NewEncoder(w).Encode(map[string]any{"data": data}) + })) + defer server.Close() + t.Setenv("GITCRAWL_GITHUB_BASE_URL", server.URL) + t.Setenv("GITHUB_TOKEN", "test-token-placeholder") + a := New() + a.Stderr = io.Discard + a.configPath = writeDoctorTestConfig(t, dir, path) + client := gh.New(gh.Options{BaseURL: server.URL, Token: "test-token-placeholder"}) + reviewRetry := store.AnalyticsAttempt{Repository: "fixture/repo", Number: 1, Operation: "review_state", StartedAt: "2099-01-01T00:00:00Z", FinishedAt: "2099-01-01T00:00:01Z", Status: "failed", Evidence: json.RawMessage(`{}`)} + if !brokenDiscovery { + if err = s.RecordAnalyticsAttempt(ctx, reviewRetry); err != nil { + t.Fatal(err) + } + } else { + // A transient core failure can precede first lane migration. Its + // incomplete flag must not erase the verified historical baseline. + failure := store.AnalyticsAttempt{Repository: "fixture/repo", Operation: "discover_issues", StartedAt: baseline, FinishedAt: baseline, Status: "failed", Evidence: json.RawMessage(`{}`)} + if err = s.RecordAnalyticsAttempt(ctx, failure); err != nil { + t.Fatal(err) + } + var complete int + if err = s.DB().QueryRow("SELECT complete FROM analytics_coverage WHERE repository='fixture/repo'").Scan(&complete); err != nil || complete != 0 { + t.Fatalf("failure did not clear core coverage: complete=%d err=%v", complete, err) + } + } + if !brokenDiscovery { + // Exercise the real first-watch entry point with no coverage row or + // completed-update watermark: the completed discovery receipt owns it. + if _, err = s.DB().Exec("DELETE FROM analytics_coverage"); err != nil { + t.Fatal(err) + } + s.SetAnalyticsState(ctx, "through:fixture/repo", "") + receipt, _ := json.Marshal(map[string]any{"phase": "complete", "discovery": []map[string]any{{"kind": "issues", "done": 1, "total": 1, "updated_at": baseline}, {"kind": "pullRequests", "done": 1, "total": 1, "updated_at": baseline}}}) + if err = os.WriteFile(filepath.Join(dir, "status.json"), receipt, 0600); err != nil { + t.Fatal(err) + } + a.Stdout = io.Discard + err = a.runAnalytics(ctx, []string{"fixture/repo", "--once", "--json"}) + } else { + err = a.analyticsCycle(ctx, s, client, "fixture", "repo") + } + if err == nil { + t.Fatal("incomplete coverage reported complete") + } + if !brokenDiscovery && !errors.Is(err, errAnalyticsIncomplete) { + t.Fatal(err) + } + var count int + s.DB().QueryRow("SELECT count(*) FROM threads WHERE number=2").Scan(&count) + if count != 1 { + t.Fatal("independent PR never persisted") + } + s.DB().QueryRow("SELECT complete FROM analytics_coverage WHERE repository='fixture/repo'").Scan(&count) + if count != 0 { + t.Fatal("poison item hidden by complete=true") + } + if !brokenDiscovery { + var before, after int + s.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts WHERE number=1 AND operation='graphql_history'").Scan(&before) + if err = a.analyticsCycle(ctx, s, client, "fixture", "repo"); !errors.Is(err, errAnalyticsIncomplete) { + t.Fatalf("unexpected repeat outcome %v", err) + } + s.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts WHERE number=1 AND operation='graphql_history'").Scan(&after) + if before != after { + t.Fatal("review-only failure bypassed core retry backoff repeatedly") + } + } + broken.Store(false) + // Simulate a due retry after a process restart, without dropping its history. + if _, err = s.DB().Exec("UPDATE analytics_retries SET next_attempt_at='2000-01-01T00:00:00Z'"); err != nil { + t.Fatal(err) + } + s.Close() + s, err = store.Open(ctx, path) + if err != nil { + t.Fatal(err) + } + if err = a.analyticsCycle(ctx, s, client, "fixture", "repo"); err != nil { + t.Fatal(err) + } + if outstanding, err := s.AnalyticsOutstanding(ctx, "fixture/repo"); err != nil || outstanding != 0 { + t.Fatalf("not recovered: %d %v", outstanding, err) + } + if err = s.DB().QueryRow("SELECT complete FROM analytics_coverage WHERE repository='fixture/repo'").Scan(&count); err != nil || count != 1 { + t.Fatalf("verified core baseline did not recover: complete=%d err=%v", count, err) + } + s.DB().QueryRow("SELECT count(*) FROM threads").Scan(&count) + if count != 2 { + t.Fatalf("wrong recovered content count %d", count) + } + s.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts WHERE status='failed'").Scan(&count) + if count < 1 { + t.Fatal("failure history erased") + } + reviewRetry.Number = 2 + if err = s.RecordAnalyticsAttempt(ctx, reviewRetry); err != nil { + t.Fatal(err) + } + revised.Store(true) + if err = a.analyticsCycle(ctx, s, client, "fixture", "repo"); err != nil { + t.Fatal(err) + } + var body string + s.DB().QueryRow("SELECT body FROM threads WHERE number=2").Scan(&body) + if body != "fresh core body" { + t.Fatal("review recovery backoff hid a newly discovered core update") + } + }) + } +} + +func TestAnalyticsLaneMigrationRetainsInflightCursorAndLegacyReceipt(t *testing.T) { + ctx := context.Background() + s, err := store.Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + legacy := `{"started":"2026-01-01T12:00:00Z","since":"2026-01-01T10:55:00Z","kind":1,"cursor":"opaque-provider-cursor","issues":10,"prs":20}` + if err = s.SaveAnalyticsCoverage(ctx, "fixture/repo", "2026-01-01T11:00:00Z", 10, 20); err != nil { + t.Fatal(err) + } + s.SetAnalyticsState(ctx, "through:fixture/repo", "2026-01-01T11:00:00Z") + s.SetAnalyticsState(ctx, "updates:fixture/repo", legacy) + if err = migrateAnalyticsLanes(ctx, s, "fixture/repo"); err != nil { + t.Fatal(err) + } + value, err := s.AnalyticsState(ctx, "updates:fixture/repo:pullRequests") + if err != nil { + t.Fatal(err) + } + var cp updateCheckpoint + if err = json.Unmarshal([]byte(value), &cp); err != nil { + t.Fatal(err) + } + if cp.Cursor != "opaque-provider-cursor" || cp.Since != "2026-01-01T10:55:00Z" { + t.Fatal("in-flight provider cursor reset") + } + total, _ := s.AnalyticsState(ctx, "total:fixture/repo:issues") + if total != "10" { + t.Fatalf("completed lane total lost: %s", total) + } + old, _ := s.AnalyticsState(ctx, "updates:fixture/repo") + if old != legacy { + t.Fatal("legacy evidence overwritten") + } + s.SetAnalyticsState(ctx, "updates:fixture/repo:pullRequests", "new-progress") + if err = migrateAnalyticsLanes(ctx, s, "fixture/repo"); err != nil { + t.Fatal(err) + } + value, _ = s.AnalyticsState(ctx, "updates:fixture/repo:pullRequests") + if value != "new-progress" { + t.Fatal("restart reseeded an existing checkpoint") + } +} + +func TestAnalyticsDiscoveryLeavesDueItemsToBoundedRetryScheduler(t *testing.T) { + ctx := context.Background() + s, err := store.Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + for i := 1; i <= 24; i++ { + a := store.AnalyticsAttempt{Repository: "fixture/repo", Number: i, Operation: "graphql_history", StartedAt: "2026-01-01T00:00:00Z", FinishedAt: "2026-01-01T00:00:01Z", Status: "failed", Evidence: json.RawMessage(`{}`)} + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + } + due, err := s.DueAnalyticsRetries(ctx, "fixture/repo", "2099-01-01T00:00:00Z", 8, "graphql_history") + if err != nil || len(due) != 8 { + t.Fatalf("retry bound: %v %v", due, err) + } + // This app has no usable configuration. Any attempted collection would fail; + // discovery must skip even due items not selected by the retry budget. + a := New() + numbers := []int{} + for i := 1; i <= 24; i++ { + numbers = append(numbers, i) + } + if err = a.analyticsNumbers(ctx, s, "fixture", "repo", numbers, true, "graphql_history"); err != nil { + t.Fatal(err) + } + var attempts int + s.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts").Scan(&attempts) + if attempts != 24 { + t.Fatal("discovery bypassed the retry scheduler") + } +} diff --git a/internal/cli/analytics_partial_response_test.go b/internal/cli/analytics_partial_response_test.go new file mode 100644 index 00000000..853f52e2 --- /dev/null +++ b/internal/cli/analytics_partial_response_test.go @@ -0,0 +1,167 @@ +package cli + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/http/httptest" + "path/filepath" + "regexp" + "strconv" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/openclaw/gitcrawl/internal/store" +) + +func TestAnalyticsPairedPartialRejectionIsolatesPeerAndPreservesUnavailableHistory(t *testing.T) { + ctx := context.Background() + dir := t.TempDir() + dbpath := filepath.Join(dir, "archive.db") + s, err := store.Open(ctx, dbpath) + if err != nil { + t.Fatal(err) + } + defer s.Close() + var reject atomic.Bool + beforeAt := time.Now().UTC().Add(-time.Minute).Format(time.RFC3339Nano) + afterAt := time.Now().UTC().Format(time.RFC3339Nano) + conn := func(nodes ...any) map[string]any { + if nodes == nil { + nodes = []any{} + } + return map[string]any{"totalCount": len(nodes), "nodes": nodes, "pageInfo": map[string]any{"hasNextPage": false, "endCursor": "end"}} + } + aliases := regexp.MustCompile(`n([0-9]+): issueOrPullRequest\(number:([0-9]+)\)`) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + fmt.Fprint(w, `{"resources":{"graphql":{"limit":20000,"remaining":19000,"reset":4102444800},"core":{"limit":20000,"remaining":19000,"reset":4102444800}}}`) + return + } + var req struct{ Query string } + if e := json.NewDecoder(r.Body).Decode(&req); e != nil { + t.Error(e) + return + } + data := map[string]any{"rateLimit": map[string]any{"cost": 1, "limit": 20000, "remaining": 19000, "resetAt": "2099-01-01T00:00:00Z"}} + response := map[string]any{"data": data} + repo := map[string]any{"id": "repo", "databaseId": 1, "nameWithOwner": "fixture/repo"} + var failures []any + matches := aliases.FindAllStringSubmatch(req.Query, -1) + for _, m := range matches { + n, _ := strconv.Atoi(m[2]) + alias := "n" + m[1] + if reject.Load() && n == 16945 && len(matches) == 1 { + var retainedBody string + if e := s.DB().QueryRow("SELECT body FROM threads WHERE number=16945").Scan(&retainedBody); e != nil || retainedBody != "retained original" { + t.Errorf("paired partial response was applied before isolated success: %q %v", retainedBody, e) + } + } + if reject.Load() && n == 16944 { + repo[alias] = nil + failures = append(failures, map[string]any{"type": "NOT_FOUND", "path": []any{"repository", alias}, "message": "private provider prose"}) + continue + } + at, body := beforeAt, "retained original" + if reject.Load() { + at, body = afterAt, "isolated success" + } + comment := map[string]any{"id": fmt.Sprint("C", n), "__typename": "IssueComment", "fullDatabaseId": fmt.Sprint(n + 1000000), "body": "retained comment", "createdAt": beforeAt, "updatedAt": beforeAt, "publishedAt": beforeAt, "author": map[string]any{"id": "actor", "login": "fixture", "__typename": "User"}, "url": fmt.Sprintf("https://github.com/fixture/repo/pull/%d#comment", n)} + repo[alias] = map[string]any{"id": fmt.Sprint("PR", n), "fullDatabaseId": fmt.Sprint(n), "__typename": "PullRequest", "number": n, "title": "fixture", "body": body, "state": "CLOSED", "createdAt": "2026-01-01T00:00:00Z", "updatedAt": at, "url": fmt.Sprintf("https://github.com/fixture/repo/pull/%d", n), "repository": map[string]any{"nameWithOwner": "fixture/repo"}, "author": map[string]any{"id": "actor", "login": "fixture", "__typename": "User"}, "labels": conn(), "assignees": conn(), "comments": conn(comment), "reviews": conn(), "reviewThreads": conn()} + } + if len(repo) > 3 { + data["repository"] = repo + } + if len(failures) > 0 { + response["errors"] = failures + } + json.NewEncoder(w).Encode(response) + })) + defer server.Close() + t.Setenv("GITCRAWL_GITHUB_BASE_URL", server.URL) + t.Setenv("GITHUB_TOKEN", "test-token-placeholder") + a := New() + a.Stderr = io.Discard + a.configPath = writeDoctorTestConfig(t, dir, dbpath) + if err = a.syncAnalyticsBatch(ctx, s, "fixture", "repo", []int{16944, 16945}, "graphql_history"); err != nil { + t.Fatal(err) + } + if err = s.SaveAnalyticsCoverage(ctx, "fixture/repo", beforeAt, 0, 2); err != nil { + t.Fatal(err) + } + // Reproduce a retained legacy PR with unknown review membership. + if _, err = s.DB().Exec("DELETE FROM pull_request_review_thread_syncs WHERE thread_id=(SELECT id FROM threads WHERE number=16944)"); err != nil { + t.Fatal(err) + } + snapshot := func() string { + t.Helper() + tables := []string{"threads", "thread_revisions", "comments", "comment_revisions"} + out := map[string][][]any{} + for _, table := range tables { + where := "thread_id=(SELECT id FROM threads WHERE number=16944)" + if table == "threads" { + where = "number=16944" + } else if table == "comment_revisions" { + where = "comment_id IN (SELECT id FROM comments WHERE thread_id=(SELECT id FROM threads WHERE number=16944))" + } + rows, e := s.DB().Query("SELECT * FROM " + table + " WHERE " + where + " ORDER BY id") + if e != nil { + t.Fatal(e) + } + cols, _ := rows.Columns() + values := [][]any{} + for rows.Next() { + row := make([]any, len(cols)) + dest := make([]any, len(cols)) + for i := range row { + dest[i] = &row[i] + } + if e = rows.Scan(dest...); e != nil { + t.Fatal(e) + } + values = append(values, row) + } + if e = rows.Err(); e != nil { + t.Fatal(e) + } + rows.Close() + out[table] = values + } + b, _ := json.Marshal(out) + return string(b) + } + retained := snapshot() + reject.Store(true) + if err = a.analyticsIsolatedBatch(ctx, s, "fixture", "repo", []int{16944, 16945}, "review_state"); err != nil { + t.Fatal(err) + } + if snapshot() != retained { + t.Fatal("unavailable PR history changed") + } + var unknown, complete int + s.DB().QueryRow("SELECT count(*) FROM pull_request_review_thread_syncs WHERE thread_id=(SELECT id FROM threads WHERE number=16944)").Scan(&unknown) + s.DB().QueryRow("SELECT complete FROM analytics_coverage WHERE repository='fixture/repo'").Scan(&complete) + if unknown != 0 || complete != 1 { + t.Fatalf("fabricated membership or invalidated core: %d %d", unknown, complete) + } + var evidence string + if err = s.DB().QueryRow("SELECT evidence_json FROM analytics_fetch_attempts WHERE number=16944 AND operation='review_state' ORDER BY id DESC LIMIT 1").Scan(&evidence); err != nil { + t.Fatal(err) + } + if !strings.Contains(evidence, `"type":"NOT_FOUND"`) || !strings.Contains(evidence, `"path":["repository","n0"]`) || strings.Contains(evidence, "private provider prose") { + t.Fatal("missing or unsafe durable cause", evidence) + } + var pending, peerResolved, success int + s.DB().QueryRow("SELECT count(*) FROM analytics_retries WHERE number=16944 AND operation='review_state' AND resolved_at IS NULL AND attempts=2").Scan(&pending) + s.DB().QueryRow("SELECT count(*) FROM analytics_retries WHERE number=16945 AND operation='review_state' AND resolved_at IS NOT NULL").Scan(&peerResolved) + s.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts WHERE number=16945 AND operation='review_state' AND status='success'").Scan(&success) + var membership, body string + s.DB().QueryRow("SELECT x.review_thread_ids_json,t.body FROM threads t JOIN pull_request_review_thread_syncs x ON x.thread_id=t.id WHERE t.number=16945").Scan(&membership, &body) + if pending != 1 || peerResolved != 1 || success != 1 || membership != "[]" || body != "retained original" { + t.Fatalf("isolation/retry proof failed: %d %d %d %s %s", pending, peerResolved, success, membership, body) + } +} diff --git a/internal/cli/analytics_readonly_test.go b/internal/cli/analytics_readonly_test.go new file mode 100644 index 00000000..d1ecffba --- /dev/null +++ b/internal/cli/analytics_readonly_test.go @@ -0,0 +1,148 @@ +package cli + +import ( + "bytes" + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync/atomic" + "testing" + + "github.com/openclaw/gitcrawl/internal/store" +) + +func TestAnalyticsCommandsRespectOwnershipAndExplicitRepair(t *testing.T) { + ctx := context.Background() + dir := t.TempDir() + path := filepath.Join(dir, "source.db") + s, err := store.Open(ctx, path) + if err != nil { + t.Fatal(err) + } + defer s.Close() + repoID, err := s.UpsertRepository(ctx, store.Repository{Owner: "fixture", Name: "repo", FullName: "fixture/repo", RawJSON: `{}`, UpdatedAt: "2026-01-01T00:00:00Z"}) + if err != nil { + t.Fatal(err) + } + tid, err := s.UpsertThread(ctx, store.Thread{RepoID: repoID, GitHubID: "1", Number: 1, Kind: "pull_request", State: "open", Title: "retained", HTMLURL: "https://github.com/fixture/repo/pull/1", LabelsJSON: "[]", AssigneesJSON: "[]", RawJSON: `{}`, ContentHash: "h", UpdatedAt: "2026-01-01T00:00:00Z"}) + if err != nil { + t.Fatal(err) + } + _, err = s.UpsertComment(ctx, store.Comment{ThreadID: tid, GitHubID: "review", CommentType: "pull_review", RawJSON: `{"submitted_at":"2026-01-02T00:00:00Z"}`, CreatedAtGitHub: "2026-01-01T00:00:00Z"}) + if err != nil { + t.Fatal(err) + } + if _, err = s.DB().ExecContext(ctx, "UPDATE comments SET submitted_at_gh=NULL,publication_at_gh=NULL; UPDATE comment_revisions SET submitted_at_gh=NULL,publication_at_gh=NULL"); err != nil { + t.Fatal(err) + } + if err = s.SaveAnalyticsCoverage(ctx, "fixture/repo", "2026-01-01T00:00:00Z", 1, 1); err != nil { + t.Fatal(err) + } + if err = s.RecordAnalyticsAttempt(ctx, store.AnalyticsAttempt{Repository: "fixture/repo", Number: 1, Operation: "review_state", Status: "failed", ErrorClass: "validation", ErrorText: "private-rejection-sentinel", Evidence: json.RawMessage(`{"private":"private-rejection-sentinel"}`), StartedAt: "2026-01-01T00:00:00Z", FinishedAt: "2026-01-01T00:00:01Z"}); err != nil { + t.Fatal(err) + } + cfg := writeDoctorTestConfig(t, dir, path) + lock, err := os.OpenFile(filepath.Join(dir, "runner.lock"), os.O_CREATE|os.O_RDWR, 0600) + if err != nil { + t.Fatal(err) + } + defer lock.Close() + if err = lockPortableFile(lock); err != nil { + t.Fatal(err) + } + var requests atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests.Add(1) + http.Error(w, "unexpected provider request", 500) + })) + defer server.Close() + t.Setenv("GITCRAWL_GITHUB_BASE_URL", server.URL) + run := func(args ...string) (map[string]any, error) { + t.Helper() + a := New() + var out bytes.Buffer + a.Stdout = &out + a.Stderr = io.Discard + err := a.Run(ctx, append([]string{"--config", cfg, "analytics", "fixture/repo", "--json"}, args...)) + if err != nil { + return nil, err + } + if strings.Contains(out.String(), "private-rejection-sentinel") { + t.Fatal("status exposed private rejection evidence") + } + var payload map[string]any + if err = json.Unmarshal(out.Bytes(), &payload); err != nil { + t.Fatal(err) + } + return payload, nil + } + status, err := run("--status") + if err != nil { + t.Fatal(err) + } + coverage, ok := status["coverage"].(map[string]any) + if !ok || coverage["complete"] != true || status["unresolved_retries"] != float64(1) || status["core_unresolved_retries"] != float64(0) { + t.Fatalf("review failure hid core coverage: %+v", status) + } + audit, err := run() + if err != nil { + t.Fatal(err) + } + if audit["would_change"] != float64(2) || audit["changed"] != float64(0) { + t.Fatalf("audit mutated or missed repair: %+v", audit) + } + if _, err = run("--apply"); err == nil || !strings.Contains(err.Error(), "ownership lock busy") { + t.Fatalf("mutation bypassed collector owner: %v", err) + } + if _, err = run("--status", "--once"); err == nil || !strings.Contains(err.Error(), "cannot be combined") { + t.Fatalf("mixed read/write mode accepted: %v", err) + } + var modified, attempts int + if err = s.DB().QueryRowContext(ctx, "SELECT (SELECT count(*) FROM comments WHERE publication_at_gh IS NOT NULL)+(SELECT count(*) FROM comment_revisions WHERE publication_at_gh IS NOT NULL)").Scan(&modified); err != nil || modified != 0 { + t.Fatalf("audit applied publication repair: %d %v", modified, err) + } + if err = s.DB().QueryRowContext(ctx, "SELECT count(*) FROM analytics_fetch_attempts").Scan(&attempts); err != nil || attempts != 1 { + t.Fatalf("read created a collection receipt: %d %v", attempts, err) + } + // An explicit repair succeeds only after the fixture collector releases its + // lock. It normalizes retained evidence without inventing a new revision. + var rawBefore, recordedBefore string + if err = s.DB().QueryRowContext(ctx, "SELECT raw_json,recorded_at FROM comment_revisions").Scan(&rawBefore, &recordedBefore); err != nil { + t.Fatal(err) + } + if err = lock.Close(); err != nil { + t.Fatal(err) + } + repaired, err := run("--apply") + if err != nil || repaired["changed"] != float64(2) { + t.Fatalf("explicit repair=%+v err=%v", repaired, err) + } + generation, err := s.AnalyticsState(ctx, "publication_repair_generation") + if err != nil || generation == "" { + t.Fatalf("missing repair generation: %q %v", generation, err) + } + var rawAfter, recordedAfter, published string + var revisions int + if err = s.DB().QueryRowContext(ctx, "SELECT count(*),raw_json,recorded_at,publication_at_gh FROM comment_revisions").Scan(&revisions, &rawAfter, &recordedAfter, &published); err != nil { + t.Fatal(err) + } + if revisions != 1 || rawBefore != rawAfter || recordedBefore != recordedAfter || published != "2026-01-02T00:00:00Z" { + t.Fatal("repair changed retained source history or publication time") + } + replay, err := run("--apply") + if err != nil || replay["changed"] != float64(0) { + t.Fatalf("repair replay=%+v err=%v", replay, err) + } + again, err := s.AnalyticsState(ctx, "publication_repair_generation") + if err != nil || generation != again { + t.Fatalf("idempotent replay changed repair generation: %q %q %v", generation, again, err) + } + if requests.Load() != 0 { + t.Fatalf("read-only modes contacted provider %d times", requests.Load()) + } +} diff --git a/internal/cli/analytics_test.go b/internal/cli/analytics_test.go new file mode 100644 index 00000000..5fd55cf4 --- /dev/null +++ b/internal/cli/analytics_test.go @@ -0,0 +1,28 @@ +package cli + +import ( + "context" + "errors" + "testing" + "time" +) + +func TestAnalyticsEnrichmentContinuesAfterDraining(t *testing.T) { + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + calls := 0 + err := maintainAnalyticsEnrichment(ctx, true, time.Millisecond, func() error { + calls++ + if calls == 2 { + cancel() + } + return nil + }) + if calls != 2 || !errors.Is(err, context.Canceled) { + t.Fatalf("calls=%d err=%v", calls, err) + } + calls = 0 + if err := maintainAnalyticsEnrichment(context.Background(), false, time.Millisecond, func() error { calls++; return nil }); err != nil || calls != 1 { + t.Fatalf("one-shot calls=%d err=%v", calls, err) + } +} diff --git a/internal/cli/analytics_throughput_test.go b/internal/cli/analytics_throughput_test.go new file mode 100644 index 00000000..8b0f2976 --- /dev/null +++ b/internal/cli/analytics_throughput_test.go @@ -0,0 +1,340 @@ +package cli + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "path/filepath" + "regexp" + "strconv" + "strings" + "sync/atomic" + "testing" + "time" + + gh "github.com/openclaw/gitcrawl/internal/github" + "github.com/openclaw/gitcrawl/internal/store" +) + +func TestAnalyticsRecoveryUsesQuotaAfterCoreAndResumesCancellation(t *testing.T) { + for _, mode := range []string{"available", "reserved", "quota_drops", "quota_races", "concurrent", "cancel", "narrow_fair"} { + t.Run(mode, func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + dir := t.TempDir() + s, err := store.Open(ctx, filepath.Join(dir, "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + _, err = s.DB().Exec(`INSERT INTO repositories(id,owner,name,full_name,github_repo_id,raw_json,updated_at) VALUES(1,'fixture','repo','fixture/repo',1,'{}','2026-01-01')`) + if err != nil { + t.Fatal(err) + } + oldItems := 48 + if mode == "quota_drops" || mode == "concurrent" { + oldItems = 300 + } + if mode == "quota_races" { + oldItems = 6001 + } + tx, err := s.DB().BeginTx(ctx, nil) + if err != nil { + t.Fatal(err) + } + for n := 1; n <= oldItems; n++ { + _, err = tx.Exec(`INSERT INTO threads(id,repo_id,github_id,number,kind,state,title,html_url,labels_json,assignees_json,raw_json,content_hash,updated_at,last_pulled_at) VALUES(?,1,?,?,'pull_request','open','','','[]','[]','{}','fixture','2026-01-01','2026-01-01T00:00:00Z')`, n, fmt.Sprint(n), n) + if err != nil { + t.Fatal(err) + } + } + if err = tx.Commit(); err != nil { + t.Fatal(err) + } + if mode == "narrow_fair" { + for n := 1; n <= 16; n++ { + if _, err = s.DB().Exec("INSERT INTO analytics_retries(repository,number,operation,first_seen_at,last_seen_at,next_attempt_at,attempts) VALUES('fixture/repo',?,'review_state','2000-01-01T00:00:00Z','2000-01-01T00:00:00Z','2000-01-01T00:00:00Z',1)", n); err != nil { + t.Fatal(err) + } + } + } + baseline := time.Now().UTC().Add(-time.Minute).Format(time.RFC3339Nano) + if err = s.SaveAnalyticsCoverage(ctx, "fixture/repo", baseline, 0, oldItems); err != nil { + t.Fatal(err) + } + ready := make(chan struct{}) + var concurrentBatches atomic.Int64 + var coreSeen, cancelled atomic.Bool + var recovered, probes, recoveryProbes atomic.Int64 + quota := func() int { + if mode == "narrow_fair" { + if recovered.Load() >= 16 { + return 3020 + } + return 3530 + } + + if mode == "quota_races" && recoveryProbes.Load() > 1 { + return 2999 + } + if mode == "reserved" || (mode == "quota_drops" && recovered.Load() >= 16) { + return 3020 + } + return 19000 + } + conn := func(nodes ...any) map[string]any { + if nodes == nil { + nodes = []any{} + } + return map[string]any{"totalCount": len(nodes), "nodes": nodes, "pageInfo": map[string]any{"hasNextPage": false, "endCursor": "end"}} + } + numbers := regexp.MustCompile(`n([0-9]+): issueOrPullRequest\(number:([0-9]+)\)`) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + if r.URL.Path == "/rate_limit" { + probes.Add(1) + if coreSeen.Load() { + recoveryProbes.Add(1) + } + fmt.Fprintf(w, `{"resources":{"graphql":{"limit":20000,"remaining":%d,"reset":4102444800},"core":{"limit":20000,"remaining":19999,"reset":4102444800}}}`, quota()) + return + } + if r.URL.Path != "/graphql" { + t.Errorf("unexpected endpoint %s", r.URL.Path) + http.Error(w, "unexpected", 400) + return + } + var req struct{ Query string } + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Error(err) + return + } + if coreSeen.Load() && !strings.Contains(req.Query, "orderBy") && !strings.Contains(req.Query, "issueOrPullRequest") { + recoveryProbes.Add(1) + } + data := map[string]any{"rateLimit": map[string]any{"cost": 1, "remaining": quota(), "limit": 20000, "resetAt": "2099-01-01T00:00:00Z"}} + if strings.Contains(req.Query, "orderBy") { + kind, page := "issues", conn() + if strings.Contains(req.Query, "pullRequests(first:") { + kind, page = "pullRequests", conn(map[string]any{"number": 10000, "updatedAt": time.Now().UTC().Format(time.RFC3339Nano)}) + } + data["repository"] = map[string]any{kind: page} + } else if matches := numbers.FindAllStringSubmatch(req.Query, -1); len(matches) > 0 { + if mode == "concurrent" && !strings.Contains(req.Query, "number:10000)") { + if concurrentBatches.Add(1) == 32 { + close(ready) + } + select { + case <-ready: + case <-time.After(10 * time.Second): + t.Error("32 workers were not active") + return + } + } + repo := map[string]any{"id": "repo", "databaseId": 1, "nameWithOwner": "fixture/repo"} + for _, m := range matches { + n, _ := strconv.Atoi(m[2]) + if n == 10000 { + coreSeen.Store(true) + } else { + if !coreSeen.Load() { + t.Error("recovery ran before ordinary capture") + } + var complete int + if e := s.DB().QueryRow("SELECT complete FROM analytics_coverage WHERE repository='fixture/repo'").Scan(&complete); e != nil || complete != 1 { + t.Errorf("core coverage unavailable during review: %d %v", complete, e) + } + if mode == "cancel" && cancelled.CompareAndSwap(false, true) { + cancel() + <-r.Context().Done() + return + } + recovered.Add(1) + } + node := map[string]any{"id": fmt.Sprint("PR-", n), "fullDatabaseId": fmt.Sprint(n), "__typename": "PullRequest", "number": n, "title": "fixture", "body": "retained", "state": "OPEN", "createdAt": "2026-01-01T00:00:00Z", "updatedAt": baseline, "url": fmt.Sprintf("https://github.com/fixture/repo/pull/%d", n), "repository": map[string]any{"nameWithOwner": "fixture/repo"}, "author": map[string]any{"id": "actor", "login": "fixture", "__typename": "User"}, "labels": conn(), "assignees": conn(), "comments": conn(), "reviews": conn(), "reviewThreads": conn()} + repo["n"+m[1]] = node + } + data["repository"] = repo + } + json.NewEncoder(w).Encode(map[string]any{"data": data}) + })) + defer server.Close() + t.Setenv("GITCRAWL_GITHUB_BASE_URL", server.URL) + t.Setenv("GITHUB_TOKEN", "test-token-placeholder") + a := New() + a.Stderr = io.Discard + a.configPath = writeDoctorTestConfig(t, dir, filepath.Join(dir, "archive.db")) + client := gh.New(gh.Options{BaseURL: server.URL, Token: "test-token-placeholder"}) + err = a.analyticsCycle(ctx, s, client, "fixture", "repo") + if mode == "cancel" { + if !errors.Is(err, context.Canceled) { + t.Fatalf("cancellation lost: %v", err) + } + var failed int + if e := s.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts WHERE operation='review_state' AND status='failed'").Scan(&failed); e != nil || failed == 0 { + t.Fatalf("missing cancellation receipts: %d %v", failed, e) + } + if _, err = s.DB().Exec("UPDATE analytics_retries SET next_attempt_at='2000-01-01T00:00:00Z' WHERE resolved_at IS NULL"); err != nil { + t.Fatal(err) + } + // Resume from the real committed cursor and retry rows; no reset/reseed. + err = a.analyticsCycle(context.Background(), s, client, "fixture", "repo") + } + if err != nil { + t.Fatal(err) + } + var resolved, pending, scanned, complete int + if err = s.DB().QueryRow("SELECT count(*) FROM analytics_retries WHERE operation='review_state' AND resolved_at IS NOT NULL").Scan(&resolved); err != nil { + t.Fatal(err) + } + if err = s.DB().QueryRow("SELECT pending_items,scanned,complete FROM analytics_review_state_coverage").Scan(&pending, &scanned, &complete); err != nil { + t.Fatal(err) + } + want := oldItems + if mode == "reserved" || mode == "quota_races" { + want = 0 + } + if mode == "quota_drops" { + want = analyticsReviewWave + } + if mode == "narrow_fair" { + want = 16 + var fresh int + if err = s.DB().QueryRow("SELECT count(*) FROM analytics_retries WHERE operation='review_state' AND resolved_at IS NOT NULL AND attempts=0").Scan(&fresh); err != nil || fresh != 12 { + t.Fatalf("shrunken wave starved fresh work: %d %v", fresh, err) + } + } + if resolved != want || pending != oldItems-want || scanned != oldItems+1 || (complete == 1) != (want == oldItems) { + t.Fatalf("resolved=%d pending=%d scanned=%d complete=%d, want recovered%d", resolved, pending, scanned, complete, want) + } + if mode == "quota_races" { + var deferred int + if e := s.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts WHERE error_class='rate_limit'").Scan(&deferred); e != nil || deferred == 0 { + t.Fatalf("native reserve guard not exercised: %d %v", deferred, e) + } + } + if probes.Load() == 0 { + t.Fatal("never obtained actual quota") + } + if err = s.DB().QueryRow("SELECT complete FROM analytics_coverage").Scan(&complete); err != nil || complete != 1 { + t.Fatalf("core coverage affected by recovery: %d %v", complete, err) + } + }) + } +} + +func TestAnalyticsReviewBudgetRejectsMissingAndExpiredQuota(t *testing.T) { + now := time.Now() + for _, limits := range [][]gh.RateLimitSnapshot{nil, {{Resource: "core", Limit: 20000, Remaining: 19000, ResetAt: now.Add(time.Hour)}}, {{Resource: "graphql", Limit: 20000, Remaining: 19000, ResetAt: now.Add(-time.Second)}}} { + if _, _, err := analyticsReviewBudget(limits, now); err == nil { + t.Fatal("invalid quota admitted recovery") + } + } +} + +func TestAnalyticsRecoveryScansPastOneChunkWhenQuotaIsReserved(t *testing.T) { + for _, quotaJSON := range []string{`{"data":{"rateLimit":{"cost":1,"limit":20000,"remaining":3020,"resetAt":"2099-01-01T00:00:00Z"}}}`, `{"data":{}}`, `{"data":{"rateLimit":{"cost":1,"limit":20000,"remaining":19000,"resetAt":"2000-01-01T00:00:00Z"}}}`} { + ctx := context.Background() + s, err := store.Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + _, err = s.DB().Exec(`INSERT INTO repositories(id,owner,name,full_name,github_repo_id,raw_json,updated_at) VALUES(1,'fixture','repo','fixture/repo',1,'{}','2026-01-01'); + WITH RECURSIVE n(x) AS (VALUES(1) UNION ALL SELECT x+1 FROM n WHERE x<6001) + INSERT INTO threads(id,repo_id,github_id,number,kind,state,title,html_url,labels_json,assignees_json,raw_json,content_hash,updated_at,last_pulled_at) + SELECT x,1,printf('%d',x),x,'pull_request','open','','','[]','[]','{}','fixture','2026-01-01','2026-01-01T00:00:00Z' FROM n`) + if err != nil { + t.Fatal(err) + } + var requests atomic.Int64 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests.Add(1) + if r.URL.Path != "/graphql" { + t.Errorf("provider work admitted inside reserve: %s", r.URL.Path) + } + fmt.Fprint(w, quotaJSON) + })) + defer server.Close() + a := New() + a.Stderr = io.Discard + c := gh.New(gh.Options{BaseURL: server.URL}) + if err = a.analyticsReviewRecovery(ctx, s, c, "fixture", "repo", time.Now().Add(time.Minute)); err != nil { + t.Fatal(err) + } + var scanned, pending, done, complete int + if err = s.DB().QueryRow("SELECT scanned,pending_items,scan_complete,complete FROM analytics_review_state_coverage").Scan(&scanned, &pending, &done, &complete); err != nil { + t.Fatal(err) + } + if scanned != 6001 || pending != 6001 || done != 1 || complete != 0 || requests.Load() != 1 { + t.Fatalf("scan stalled or reserve breached: %d %d %d %d requests=%d", scanned, pending, done, complete, requests.Load()) + } + } +} + +func TestAnalyticsRecoveryDeadlineYieldsWithDurableReceipt(t *testing.T) { + ctx := context.Background() + dir := t.TempDir() + s, err := store.Open(ctx, filepath.Join(dir, "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + _, err = s.DB().Exec(`INSERT INTO repositories(id,owner,name,full_name,github_repo_id,raw_json,updated_at) VALUES(1,'fixture','repo','fixture/repo',1,'{}','2026-01-01'); + INSERT INTO threads(id,repo_id,github_id,number,kind,state,title,html_url,labels_json,assignees_json,raw_json,content_hash,updated_at,last_pulled_at) VALUES(1,1,'1',1,'pull_request','open','','','[]','[]','{}','fixture','2026-01-01','2026-01-01T00:00:00Z')`) + if err != nil { + t.Fatal(err) + } + if err = s.SaveAnalyticsCoverage(ctx, "fixture/repo", "2026-01-01T00:00:00Z", 0, 1); err != nil { + t.Fatal(err) + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + fmt.Fprint(w, `{"resources":{"graphql":{"limit":20000,"remaining":19000,"reset":4102444800}}}`) + return + } + var req struct{ Query string } + if e := json.NewDecoder(r.Body).Decode(&req); e != nil { + t.Error(e) + return + } + if !strings.Contains(req.Query, "issueOrPullRequest") { + fmt.Fprint(w, `{"data":{"rateLimit":{"cost":1,"limit":20000,"remaining":19000,"resetAt":"2099-01-01T00:00:00Z"}}}`) + return + } + <-r.Context().Done() + })) + defer server.Close() + t.Setenv("GITCRAWL_GITHUB_BASE_URL", server.URL) + t.Setenv("GITHUB_TOKEN", "test-token-placeholder") + a := New() + a.Stderr = io.Discard + a.configPath = writeDoctorTestConfig(t, dir, filepath.Join(dir, "archive.db")) + c := gh.New(gh.Options{BaseURL: server.URL, Token: "test-token-placeholder"}) + if err = a.analyticsReviewRecovery(ctx, s, c, "fixture", "repo", time.Now().Add(6*time.Second)); err != nil { + t.Fatal("window yield failed core cycle:", err) + } + var failed, pending, core int + if err = s.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts WHERE status='failed' AND error_class='cancelled'").Scan(&failed); err != nil { + t.Fatal(err) + } + if err = s.DB().QueryRow("SELECT pending_items FROM analytics_review_state_coverage").Scan(&pending); err != nil { + t.Fatal(err) + } + if err = s.DB().QueryRow("SELECT complete FROM analytics_coverage").Scan(&core); err != nil { + t.Fatal(err) + } + if failed != 1 || pending != 1 || core != 1 { + t.Fatalf("lost cancellation state: failed=%d pending=%d core=%d", failed, pending, core) + } + if analyticsCancellationOnly(errors.Join(context.DeadlineExceeded, errors.New("receipt failed"))) { + t.Fatal("storage error hidden as yield") + } + if !analyticsCancellationOnly(errors.Join(fmt.Errorf("request: %w", context.DeadlineExceeded), context.Canceled)) { + t.Fatal("normal cancellation not recognized") + } +} diff --git a/internal/cli/app.go b/internal/cli/app.go index 0a390fc7..f79f5097 100644 --- a/internal/cli/app.go +++ b/internal/cli/app.go @@ -25,11 +25,12 @@ const ( ) type App struct { - githubTokenCommand *string - githubTokenMu sync.Mutex - observedGitHubToken string - Stdout io.Writer - Stderr io.Writer + analyticsTokenProvider func(context.Context) (string, error) + githubTokenCommand *string + githubTokenMu sync.Mutex + observedGitHubToken string + Stdout io.Writer + Stderr io.Writer configPath string format OutputFormat @@ -126,6 +127,8 @@ func (a *App) Run(ctx context.Context, args []string) error { return a.runDoctor(ctx, rest[1:]) case "status": return a.runStatus(ctx, rest[1:]) + case "analytics": + return a.runAnalytics(ctx, rest[1:]) case "sync": return a.runSync(ctx, rest[1:]) case "fill-pr-details": @@ -134,6 +137,8 @@ func (a *App) Run(ctx context.Context, args []string) error { return a.runThreads(ctx, rest[1:]) case "capture": return a.runCapture(ctx, rest[1:]) + case "purge-threads": + return a.runPurgeThreads(ctx, rest[1:]) case "close-thread": return a.runCloseThread(ctx, rest[1:]) case "reopen-thread": diff --git a/internal/cli/app_test.go b/internal/cli/app_test.go index 1a3c3fdd..467cde2b 100644 --- a/internal/cli/app_test.go +++ b/internal/cli/app_test.go @@ -4344,10 +4344,10 @@ func TestDoctorJSONReportsCurrentSchemaDiagnosticsWithoutMutation(t *testing.T) if got := schema["state"]; got != "current" { t.Fatalf("db_schema.state = %#v, payload=%#v", got, schema) } - if got := schema["current_version"]; got != float64(13) { + if got := schema["current_version"]; got != float64(16) { t.Fatalf("db_schema.current_version = %#v, payload=%#v", got, schema) } - if got := schema["supported_version"]; got != float64(13) { + if got := schema["supported_version"]; got != float64(16) { t.Fatalf("db_schema.supported_version = %#v, payload=%#v", got, schema) } if got := schema["child_observation_reservations"]; got != true { @@ -4609,7 +4609,7 @@ func TestDoctorJSONReportsLegacyPendingSchemaWithoutMutation(t *testing.T) { t.Fatalf("pr_details.duplicate_path_files_supported = %#v, payload=%#v", got, prDetails) } pending := doctorStringList(t, schema, "pending_migrations") - if !doctorListContains(pending, "schema_version_3_to_13") || + if !doctorListContains(pending, "schema_version_3_to_16") || !doctorListContains(pending, "pull_request_files_position_key") || !doctorListContains(pending, "thread_child_observation_reservations_table") { t.Fatalf("pending_migrations = %#v", pending) diff --git a/internal/cli/gh_search_test.go b/internal/cli/gh_search_test.go index aeb79dcc..650c8160 100644 --- a/internal/cli/gh_search_test.go +++ b/internal/cli/gh_search_test.go @@ -265,8 +265,8 @@ func TestGHSearchSyncIfStaleMigratesFreshPortableRuntime(t *testing.T) { if err := rt.Store.DB().QueryRowContext(ctx, `pragma user_version`).Scan(&schemaVersion); err != nil { t.Fatalf("read runtime schema version: %v", err) } - if schemaVersion != 13 { - t.Fatalf("runtime schema version = %d, want 13", schemaVersion) + if schemaVersion != 16 { + t.Fatalf("runtime schema version = %d, want 16", schemaVersion) } var tableName string if err := rt.Store.DB().QueryRowContext(ctx, `select name from sqlite_schema where type = 'table' and name = 'sync_runs'`).Scan(&tableName); err != nil { diff --git a/internal/cli/help.go b/internal/cli/help.go index a9d9c4d1..189fb96a 100644 --- a/internal/cli/help.go +++ b/internal/cli/help.go @@ -53,6 +53,7 @@ Core commands: init create config, optionally from a portable store doctor check config, token, and database readiness sync sync GitHub issue and pull request metadata + analytics repair publication dates, enrich identities, or watch GraphQL updates sync-failures list failed sync hydration attempts coverage report local archive PR-detail completeness fill-pr-details hydrate locally missing pull request detail rows @@ -63,6 +64,7 @@ Core commands: capture export a stable code-free conversation snapshot code index index tracked text files from a local Git checkout cluster build durable clusters from local thread vectors + purge-threads plan/apply owner-directed removal and durable collection exclusion close-thread locally hide one issue or pull request row reopen-thread clear a local hide for one issue or pull request row close-cluster locally hide one durable cluster @@ -248,6 +250,19 @@ Usage: Usage: gitcrawl runs owner/repo [--kind sync|summary|embedding|cluster] [--limit N] [--json] +`, + "purge-threads": `gitcrawl purge-threads plans owner-directed local removal, never a GitHub deletion. + +Usage: + gitcrawl [--config SOURCE_CONFIG] purge-threads owner/repo --numbers 1,2 [--runtime-mirror] [--apply --request-id OWNER_REQUEST] [--json] + +Default is a read-only metadata/count plan. Apply permanently excludes the exact +repository numbers and removes their native content/history and retry work. +Requires the native archive and its idle collector lock, or explicit +--runtime-mirror with its checkout config and existing portable owner lease. +A purged mirror stays locally preserved across scheduled upstream refresh. +Blob-backed targets, workflow dependencies and shared cluster state require +a separate source-owned repair. `, "close-thread": `gitcrawl close-thread locally hides one issue or pull request row. diff --git a/internal/cli/sync.go b/internal/cli/sync.go index 16a8b1d7..ee53ccfe 100644 --- a/internal/cli/sync.go +++ b/internal/cli/sync.go @@ -400,6 +400,9 @@ func (a *App) syncRepository(ctx context.Context, owner, repo string, options sy return command(ctx) } } + if a.analyticsTokenProvider != nil { + provider = a.analyticsTokenProvider + } if provider == nil && token.Value == "" { return syncer.Stats{}, dbTargetInfo{}, fmt.Errorf("missing GitHub token: set %s or authenticate gh", cfg.GitHub.TokenEnv) } diff --git a/internal/cli/thread_purge.go b/internal/cli/thread_purge.go new file mode 100644 index 00000000..93a5252b --- /dev/null +++ b/internal/cli/thread_purge.go @@ -0,0 +1,157 @@ +package cli + +import ( + "context" + "flag" + "fmt" + "io" + "os" + "path/filepath" + "strings" + + "github.com/openclaw/gitcrawl/internal/config" + "github.com/openclaw/gitcrawl/internal/store" +) + +func (a *App) runPurgeThreads(ctx context.Context, args []string) error { + fs := flag.NewFlagSet("purge-threads", flag.ContinueOnError) + fs.SetOutput(io.Discard) + raw := fs.String("numbers", "", "explicit issue/PR numbers") + mirror := fs.Bool("runtime-mirror", false, "purge only the existing managed portable runtime mirror and preserve local ownership") + apply := fs.Bool("apply", false, "apply owner-directed purge and durable exclusion") + requestID := fs.String("request-id", "", "owner request identifier, required for apply") + jsonOut := fs.Bool("json", false, "JSON metadata/count audit") + if err := fs.Parse(normalizeCommandArgs(args, map[string]bool{"numbers": true, "request-id": true})); err != nil { + return usageErr(err) + } + if fs.NArg() != 1 { + return usageErr(fmt.Errorf("purge-threads requires owner/repo")) + } + owner, repo, err := parseOwnerRepo(fs.Arg(0)) + if err != nil { + return usageErr(err) + } + numbers, err := parseOptionalThreadNumberList(*raw, owner+"/"+repo) + if err != nil { + return usageErr(err) + } + if len(numbers) == 0 || len(numbers) > 100 { + return usageErr(fmt.Errorf("purge-threads requires 1..100 explicit --numbers")) + } + if *apply && (strings.TrimSpace(*requestID) == "" || len(*requestID) > 128) { + return usageErr(fmt.Errorf("--apply requires a nonempty --request-id of at most 128 bytes")) + } + a.applyCommandJSON(*jsonOut) + cfg, err := config.LoadRuntime(a.configPath) + if err != nil { + return err + } + if cfg.Remote.Enabled() { + return fmt.Errorf("purge-threads requires a native local archive") + } + dbPath := cfg.DBPath + root, portable, err := portableStoreRoot(ctx, cfg.DBPath) + if err != nil { + return err + } + if *mirror { + if !portable { + return fmt.Errorf("--runtime-mirror requires a config pointing to its portable checkout") + } + var release func() + ctx, release, err = acquirePortableOwner(ctx, root) + if err != nil { + return err + } + defer release() + dbPath, err = a.portableRuntimeDBPath(ctx, cfg.DBPath) + if err != nil { + return err + } + dbPath, err = canonicalPortablePath(dbPath) + if err != nil { + return err + } + canonicalRoot, err := canonicalPortablePath(root) + if err != nil { + return err + } + if pathWithin(canonicalRoot, dbPath) || pathWithin(filepath.Dir(dbPath), canonicalRoot) { + return fmt.Errorf("runtime mirror must be outside portable checkout") + } + // Inspect an existing mirror only: never materialize, refresh or fetch. + snapshot, err := inspectPortableMirror(ctx, dbPath, cfg.DBPath) + if err != nil { + return err + } + if !snapshot.exists { + return fmt.Errorf("managed runtime mirror does not exist") + } + if err = snapshot.recheck(ctx); err != nil { + return err + } + } else if portable { + return fmt.Errorf("portable checkout requires explicit --runtime-mirror; publisher data is never purged") + } + probe, err := store.OpenReadOnly(ctx, dbPath) + if err != nil { + return err + } + var hasPortable bool + err = probe.DB().QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM sqlite_schema WHERE name='portable_metadata' AND type='table')").Scan(&hasPortable) + if err != nil { + probe.Close() + return err + } + if hasPortable && !*mirror { + probe.Close() + return fmt.Errorf("portable mirror requires its checkout config and --runtime-mirror") + } + plan, err := probe.PlanThreadPurge(ctx, owner+"/"+repo, numbers) + closeErr := probe.Close() + if err != nil { + return err + } + if closeErr != nil { + return closeErr + } + if !*apply { + return a.writeOutput("thread_purge_plan", plan, true) + } + var st *store.Store + if *mirror { + + st, err = store.OpenThreadPurgeMirror(ctx, dbPath) + } else { + // The permanent collector owner lock must precede writer open/migration. + lock, e := os.OpenFile(filepath.Join(filepath.Dir(dbPath), "runner.lock"), os.O_CREATE|os.O_RDWR, 0600) + if e != nil { + return e + } + defer lock.Close() + if e = lockPortableFile(lock); e != nil { + return fmt.Errorf("collector ownership lock busy: %w", e) + } + st, err = store.Open(ctx, dbPath) + } + if err != nil { + return err + } + defer st.Close() + result, err := st.PurgeThreads(ctx, owner+"/"+repo, numbers, *requestID) + if err != nil { + return err + } + if *mirror { + // The portable owner lease fences refresh through commit and this stamp. + // If interrupted after commit, changed bytes or a live WAL already make + // inspectPortableMirror preserve the local DB under the existing policy. + statePath := portableStoreRefreshStatePath(dbPath) + state := readPortableStoreRefreshState(statePath) + state.MirrorWritable = true + if err = writePortableStoreRefreshState(statePath, state); err != nil { + return fmt.Errorf("owner purge committed; persist writable mirror stamp: %w", err) + } + } + return a.writeOutput("thread_purge", result, true) +} diff --git a/internal/cli/thread_purge_test.go b/internal/cli/thread_purge_test.go new file mode 100644 index 00000000..d4dc8dff --- /dev/null +++ b/internal/cli/thread_purge_test.go @@ -0,0 +1,222 @@ +package cli + +import ( + "bytes" + "context" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/openclaw/gitcrawl/internal/store" +) + +func TestThreadPurgeCLIPlansThenRequiresIdleOwner(t *testing.T) { + ctx := context.Background() + dir := t.TempDir() + db := filepath.Join(dir, "archive.db") + cfg := filepath.Join(dir, "config.toml") + if err := os.WriteFile(cfg, []byte(fmt.Sprintf("db_path=%q\n", db)), 0600); err != nil { + t.Fatal(err) + } + s, err := store.Open(ctx, db) + if err != nil { + t.Fatal(err) + } + _, err = s.DB().Exec(`INSERT INTO repositories(id,owner,name,full_name,raw_json,updated_at) VALUES(1,'fixture','repo','fixture/repo','{}','2026-01-01'); + INSERT INTO threads(id,repo_id,github_id,number,kind,state,title,html_url,labels_json,assignees_json,raw_json,content_hash,updated_at) VALUES + (1,1,'one',10,'pull_request','open','PRIVATE_SENTINEL','','[]','[]','{}','h','2026-01-01'), + (2,1,'two',20,'pull_request','open','keeper','','[]','[]','{}','h','2026-01-01')`) + if err != nil { + t.Fatal(err) + } + s.Close() + run := func(args ...string) (string, error) { + a := New() + var out, stderr bytes.Buffer + a.Stdout = &out + a.Stderr = &stderr + err := a.Run(ctx, append([]string{"--config", cfg, "purge-threads", "fixture/repo", "--numbers", "10", "--json"}, args...)) + return out.String(), err + } + out, err := run() + if err != nil || !strings.Contains(out, `"applied": false`) || strings.Contains(out, "PRIVATE_SENTINEL") { + t.Fatalf("plan=%s err=%v", out, err) + } + if _, err = run("--apply"); err == nil { + t.Fatal("missing owner request accepted") + } + lock, err := os.OpenFile(filepath.Join(dir, "runner.lock"), os.O_CREATE|os.O_RDWR, 0600) + if err != nil { + t.Fatal(err) + } + if err = lockPortableFile(lock); err != nil { + t.Fatal(err) + } + _, err = run("--apply", "--request-id", "fixture-request") + if err == nil || !strings.Contains(err.Error(), "ownership lock busy") { + t.Fatalf("owner bypass: %v", err) + } + lock.Close() + out, err = run("--apply", "--request-id", "fixture-request") + if err != nil || !strings.Contains(out, `"applied": true`) { + t.Fatalf("apply=%s %v", out, err) + } + out, err = run() + if err != nil || !strings.Contains(out, `"already_excluded": true`) { + t.Fatalf("repeat plan=%s %v", out, err) + } +} + +func TestThreadPurgeManagedMirrorSurvivesRefresh(t *testing.T) { + ctx := context.Background() + fixture := newPortableRefreshFixture(t, false) + fixture.advance(t, false) + if _, err := fixture.refresh(t); err != nil { + t.Fatal(err) + } + // The fixture publisher is a native archive; compact only the disposable + // runtime copy to exercise the live sparse portable schema. + mirrorStore, e := store.Open(ctx, fixture.mirror) + if e != nil { + t.Fatal(e) + } + if _, e = mirrorStore.PrunePortablePayloads(ctx, store.PortablePruneOptions{BodyChars: 32, RetainSanitizedPayloadColumns: true}); e != nil { + t.Fatal(e) + } + mirrorStore.Close() + beforeHead := portableTestGit(t, fixture.checkout, "rev-parse", "HEAD") + beforeSource, err := fileSHA256(filepath.Join(fixture.checkout, fixture.relative)) + if err != nil { + t.Fatal(err) + } + run := func(cfg string, extra ...string) (string, error) { + a := New() + var out, stderr bytes.Buffer + a.Stdout = &out + a.Stderr = &stderr + args := []string{"--config", cfg, "purge-threads", "openclaw/openclaw", "--numbers", "1", "--json"} + err := a.Run(ctx, append(args, extra...)) + return out.String(), err + } + if _, err = run(fixture.configPath); err == nil { + t.Fatal("publisher path accepted without mirror flag") + } + alias := filepath.Join(t.TempDir(), "alias.toml") + if err = os.WriteFile(alias, []byte(fmt.Sprintf("db_path=%q\n", fixture.mirror)), 0600); err != nil { + t.Fatal(err) + } + if _, err = run(alias); err == nil { + t.Fatal("portable alias accepted without managed owner") + } + if _, err = run(alias, "--runtime-mirror"); err == nil { + t.Fatal("unbound mirror accepted") + } + out, err := run(fixture.configPath, "--runtime-mirror") + if err != nil || !strings.Contains(out, `"applied": false`) { + t.Fatalf("plan: %s %v", out, err) + } + out, err = run(fixture.configPath, "--runtime-mirror", "--apply", "--request-id", "mirror-fixture") + if err != nil || !strings.Contains(out, `"applied": true`) { + t.Fatalf("apply: %s %v", out, err) + } + if !readPortableStoreRefreshState(portableStoreRefreshStatePath(fixture.mirror)).MirrorWritable { + t.Fatal("mirror replacement not fenced") + } + if got := portableTestGit(t, fixture.checkout, "rev-parse", "HEAD"); got != beforeHead { + t.Fatal("purge changed publisher checkout") + } + if got, e := fileSHA256(filepath.Join(fixture.checkout, fixture.relative)); e != nil || got != beforeSource { + t.Fatal("purge changed source bytes", e) + } + result, err := fixture.refresh(t) + if err != nil || result.MirrorResult != "preserved-local" { + t.Fatalf("refresh lost owner policy: %+v %v", result, err) + } + s, err := store.OpenReadOnly(ctx, fixture.mirror) + if err != nil { + t.Fatal(err) + } + defer s.Close() + var removed, kept, excluded int + s.DB().QueryRow("SELECT count(*) FROM threads WHERE number=1").Scan(&removed) + s.DB().QueryRow("SELECT count(*) FROM threads WHERE number=2").Scan(&kept) + s.DB().QueryRow("SELECT count(*) FROM thread_exclusions WHERE number=1 AND reason='owner_requested'").Scan(&excluded) + if removed != 0 || kept != 1 || excluded != 1 { + t.Fatalf("refresh restored target or lost peer: %d %d %d", removed, kept, excluded) + } +} + +func TestThreadPurgeFailedMirrorApplyDoesNotFreezeRefresh(t *testing.T) { + ctx := context.Background() + f := newPortableRefreshFixture(t, false) + f.advance(t, false) + if _, err := f.refresh(t); err != nil { + t.Fatal(err) + } + s, err := store.Open(ctx, f.mirror) + if err != nil { + t.Fatal(err) + } + if _, err = s.PrunePortablePayloads(ctx, store.PortablePruneOptions{BodyChars: 32, RetainSanitizedPayloadColumns: true}); err != nil { + t.Fatal(err) + } + _, err = s.DB().Exec(`CREATE TRIGGER fixture_purge_failure BEFORE DELETE ON threads WHEN OLD.number=1 BEGIN SELECT RAISE(ABORT,'fixture purge rollback'); END`) + if err != nil { + t.Fatal(err) + } + s.Close() + before, err := fileSHA256(f.mirror) + if err != nil { + t.Fatal(err) + } + statePath := portableStoreRefreshStatePath(f.mirror) + state := readPortableStoreRefreshState(statePath) + state.MirrorWritable = false + state.MirrorHealthSourceSHA256 = fmt.Sprintf("%x", before) + if err = writePortableStoreRefreshState(statePath, state); err != nil { + t.Fatal(err) + } + run := func(request string) error { + a := New() + a.Stdout = &bytes.Buffer{} + a.Stderr = &bytes.Buffer{} + return a.Run(ctx, []string{"--config", f.configPath, "purge-threads", "openclaw/openclaw", "--numbers", "1", "--runtime-mirror", "--apply", "--request-id", request, "--json"}) + } + for _, request := range []string{strings.Repeat("x", 129), "rollback"} { + if err = run(request); err == nil { + t.Fatal("failed request accepted") + } + if readPortableStoreRefreshState(statePath).MirrorWritable { + t.Fatal("failed apply froze mirror") + } + if after, e := fileSHA256(f.mirror); e != nil || after != before { + t.Fatal("failed apply changed mirror bytes", e) + } + inspect, e := inspectPortableMirror(ctx, f.mirror, filepath.Join(f.checkout, f.relative)) + if e != nil || inspect.preserve { + t.Fatal("failed apply blocks ordinary replacement", e) + } + } + s, err = store.OpenThreadPurgeMirror(ctx, f.mirror) + if err != nil { + t.Fatal(err) + } + if _, err = s.DB().Exec("DROP TRIGGER fixture_purge_failure"); err != nil { + t.Fatal(err) + } + s.Close() + if err = run("success"); err != nil { + t.Fatal(err) + } + // Simulate interruption between SQLite commit and the advisory state stamp: + // refresh must preserve committed owner policy based on changed bytes/WAL. + if err = writePortableStoreRefreshState(statePath, state); err != nil { + t.Fatal(err) + } + inspect, err := inspectPortableMirror(ctx, f.mirror, filepath.Join(f.checkout, f.relative)) + if err != nil || !inspect.preserve { + t.Fatal("commit-to-stamp interruption can restore removed content", err) + } +} diff --git a/internal/github/analytics.go b/internal/github/analytics.go new file mode 100644 index 00000000..f6f13500 --- /dev/null +++ b/internal/github/analytics.go @@ -0,0 +1,139 @@ +package github + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/http" + "time" +) + +// AnalyticsNodes reads public provider evidence for the requested native node IDs. +func (c *Client) AnalyticsNodes(ctx context.Context, ids []string, profiles bool) ([]map[string]any, error) { + fields := `... on Issue {author{login __typename ... on Node{id}}} ... on PullRequest {author{login __typename ... on Node{id}}} ... on IssueComment {author{login __typename ... on Node{id}}} ... on PullRequestReview {author{login __typename ... on Node{id}}} ... on PullRequestReviewComment {author{login __typename ... on Node{id}}}` + if profiles { + fields = `... on User {login name bio url createdAt} ... on Bot {login url createdAt} ... on Mannequin {login url createdAt}` + } + payload, e := json.Marshal(graphqlEnvelope{Query: `query($ids:[ID!]!){rateLimit{cost remaining limit used resetAt} nodes(ids:$ids){id __typename ` + fields + `}}`, Variables: map[string]any{"ids": ids}}) + if e != nil { + return nil, e + } + var envelope struct { + Data json.RawMessage `json:"data"` + Errors []struct { + Message string `json:"message"` + Type string `json:"type"` + Path []any `json:"path"` + } `json:"errors"` + } + if e = c.doJSON(ctx, http.MethodPost, c.graphQLURL, bytes.NewReader(payload), nil, &envelope); e != nil { + return nil, e + } + for _, failure := range envelope.Errors { + if failure.Type != "NOT_FOUND" || len(failure.Path) < 2 || failure.Path[0] != "nodes" { + return nil, fmt.Errorf("actor evidence GraphQL error: %s", failure.Message) + } + } + var data map[string]any + if e = json.Unmarshal(envelope.Data, &data); e != nil { + return nil, e + } + + nodes, ok := data["nodes"].([]any) + if !ok || len(nodes) != len(ids) { + return nil, fmt.Errorf("incomplete actor identity response") + } + out := make([]map[string]any, 0, len(ids)) + for i, v := range nodes { + n, ok := v.(map[string]any) + if !ok { + n = map[string]any{"id": ids[i], "__typename": "Unavailable", "unavailable": true} + } + if n["id"] != ids[i] { + return nil, fmt.Errorf("actor node identity mismatch") + } + out = append(out, n) + } + return out, nil +} + +type UpdatedPage struct { + Numbers []int + Cursor string + More bool + Total int + Oldest time.Time +} + +// AnalyticsRateLimit reads the same GraphQL balance charged by history queries. +// REST resource counters can differ and must not admit recovery on that basis. +func (c *Client) AnalyticsRateLimit(ctx context.Context) (effective, observed RateLimitSnapshot, err error) { + // Reuse the same credential-bound reserve state without changing ordinary + // content clients' existing transport policy. + quotaClient := *c + quotaClient.graphQLQuotaGuard = true + h := historySession{client: "aClient, remaining: 20000} + data, err := h.quota(ctx) + if err != nil { + return RateLimitSnapshot{}, RateLimitSnapshot{}, err + } + rate := historyMap(data["rateLimit"]) + limit, ok := historyInt(rate["limit"]) + if !ok || limit <= 0 { + return RateLimitSnapshot{}, RateLimitSnapshot{}, fmt.Errorf("GraphQL quota limit unavailable") + } + remaining, _ := historyInt(rate["remaining"]) + reset, _ := time.Parse(time.RFC3339, historyString(rate["resetAt"])) + observed = RateLimitSnapshot{Resource: "graphql", Limit: limit, Remaining: remaining, ResetAt: reset} + return c.reserve.observeGraphQL(observed, time.Now()), observed, nil +} + +func (c *Client) UpdatedNumbers(ctx context.Context, owner, repo, kind, after string, since time.Time) (UpdatedPage, error) { + if kind != "issues" && kind != "pullRequests" { + return UpdatedPage{}, fmt.Errorf("invalid discovery kind") + } + h := historySession{client: c, remaining: 20000} + var cursor any + if after != "" { + cursor = after + } + data, e := h.request(ctx, `query($owner:String!,$repo:String!,$after:String){rateLimit{cost remaining limit used resetAt} repository(owner:$owner,name:$repo){`+kind+`(first:100,after:$after,orderBy:{field:UPDATED_AT,direction:DESC}){totalCount pageInfo{hasNextPage endCursor} nodes{number updatedAt}}}}`, map[string]any{"owner": owner, "repo": repo, "after": cursor}, 1) + if e != nil { + return UpdatedPage{}, e + } + r := historyMap(historyMap(data["repository"])[kind]) + p := UpdatedPage{} + var valid bool + p.Total, valid = historyInt(r["totalCount"]) + if !valid || p.Total < 0 { + return p, historyFailure("discovery_validation", 0, r, fmt.Errorf("missing update-discovery count")) + } + info := historyMap(r["pageInfo"]) + var hasPageFlag bool + p.More, hasPageFlag = info["hasNextPage"].(bool) + nodes, hasNodes := r["nodes"].([]any) + // totalCount covers the whole connection. A resumed final page can become + // empty after deletion/reordering; it must not pin its cursor forever. + if !hasPageFlag || !hasNodes || (len(nodes) == 0 && p.Total > 0 && after == "") || len(nodes) > p.Total { + return p, historyFailure("discovery_validation", 0, r, fmt.Errorf("incomplete update-discovery page")) + } + p.Cursor = historyString(info["endCursor"]) + for _, n := range historyNodes(historyMap(data["repository"]), kind) { + number, ok := historyInt(n["number"]) + at, e := time.Parse(time.RFC3339Nano, historyString(n["updatedAt"])) + if !ok || number < 1 || e != nil { + return p, historyFailure("discovery_validation", 0, r, fmt.Errorf("invalid update-discovery evidence")) + } + if !at.Before(since) { + p.Numbers = append(p.Numbers, number) + } + if p.Oldest.IsZero() || at.Before(p.Oldest) { + p.Oldest = at + } + } + if p.More && (p.Cursor == "" || p.Cursor == after || p.Oldest.IsZero()) { + return p, historyFailure("discovery_validation", 0, r, fmt.Errorf("update cursor did not advance")) + } + return p, nil +} diff --git a/internal/github/analytics_test.go b/internal/github/analytics_test.go new file mode 100644 index 00000000..60b36cac --- /dev/null +++ b/internal/github/analytics_test.go @@ -0,0 +1,188 @@ +package github + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +func TestAnalyticsDiscoveryFiltersOldRowsAndRejectsBadCursors(t *testing.T) { + more := false + cursor := "end" + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"rateLimit": map[string]any{"cost": 1, "remaining": 19000, "limit": 20000, "used": 1000, "resetAt": time.Now().UTC().Add(time.Hour).Format(time.RFC3339)}, "repository": map[string]any{"issues": map[string]any{"totalCount": 2, "pageInfo": map[string]any{"hasNextPage": more, "endCursor": cursor}, "nodes": []any{map[string]any{"number": 2, "updatedAt": "2026-09-24T00:00:00Z"}, map[string]any{"number": 1, "updatedAt": "2026-09-01T00:00:00Z"}}}}}}) + })) + defer server.Close() + c := New(Options{Token: "fixture", BaseURL: server.URL}) + since, _ := time.Parse(time.RFC3339, "2026-09-23T00:00:00Z") + p, e := c.UpdatedNumbers(context.Background(), "fixture", "repo", "issues", "", since) + if e != nil || len(p.Numbers) != 1 || p.Numbers[0] != 2 || p.Total != 2 { + t.Fatalf("%+v %v", p, e) + } + more = true + cursor = "same" + if _, e = c.UpdatedNumbers(context.Background(), "fixture", "repo", "issues", "same", since); e == nil { + t.Fatal("non-advancing cursor accepted") + } +} +func TestAnalyticsDiscoveryEmptyContinuation(t *testing.T) { + for _, tc := range []struct { + name, after string + more, wantError bool + }{ + {"final continuation", "previous", false, false}, + {"incomplete initial page", "", false, true}, + {"empty advancing page", "previous", true, true}, + } { + t.Run(tc.name, func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"rateLimit": map[string]any{"cost": 1, "remaining": 19000, "resetAt": "2099-01-01T00:00:00Z"}, "repository": map[string]any{"issues": map[string]any{"totalCount": 100, "pageInfo": map[string]any{"hasNextPage": tc.more, "endCursor": "next"}, "nodes": []any{}}}}}) + })) + defer server.Close() + c := New(Options{Token: "test-token-placeholder", BaseURL: server.URL}) + p, err := c.UpdatedNumbers(context.Background(), "fixture", "repo", "issues", tc.after, time.Time{}) + if (err != nil) != tc.wantError { + t.Fatalf("page=%+v error=%v, wantError=%v", p, err, tc.wantError) + } + if err == nil && (p.More || len(p.Numbers) != 0 || p.Total != 100) { + t.Fatalf("incorrect final page: %+v", p) + } + }) + } +} + +func TestAnalyticsUnavailableNodesAreNotAuthorizationSuccess(t *testing.T) { + typ := "NOT_FOUND" + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"nodes": []any{map[string]any{"id": "one", "__typename": "User", "login": "fixture"}, nil}}, "errors": []any{map[string]any{"type": typ, "message": "fixture unavailable", "path": []any{"nodes", 1}}}}) + })) + defer server.Close() + c := New(Options{Token: "fixture", BaseURL: server.URL}) + nodes, e := c.AnalyticsNodes(context.Background(), []string{"one", "two"}, true) + if e != nil || len(nodes) != 2 || nodes[1]["id"] != "two" || nodes[1]["__typename"] != "Unavailable" { + t.Fatalf("%+v %v", nodes, e) + } + typ = "FORBIDDEN" + if _, e = c.AnalyticsNodes(context.Background(), []string{"one", "two"}, true); e == nil { + t.Fatal("authorization failure became missing-data evidence") + } +} + +func TestAnalyticsQuotaAndHistoryReserveUseActualGraphQLBalance(t *testing.T) { + var contentRequests int + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + fmt.Fprint(w, `{"resources":{"graphql":{"limit":20000,"remaining":19999,"reset":4102444800}}}`) + return + } + var req struct{ Query string } + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Error(err) + return + } + if strings.Contains(req.Query, "issueOrPullRequest") { + contentRequests++ + } + fmt.Fprint(w, `{"data":{"rateLimit":{"cost":1,"limit":20000,"remaining":2990,"resetAt":"2099-01-01T00:00:00Z"}}}`) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, Token: "test-token-placeholder", RateLimitReserve: 3000}) + quota, _, err := c.AnalyticsRateLimit(context.Background()) + if err != nil || quota.Remaining != 2990 || quota.Limit != 20000 { + t.Fatalf("REST counter admitted work: %+v %v", quota, err) + } + if _, err = c.FetchGraphQLHistory(context.Background(), "fixture", "repo", []int{1}, nil); err == nil || !strings.Contains(err.Error(), "quota reserve reached") { + t.Fatalf("actual GraphQL reserve ignored: %v", err) + } + if contentRequests != 0 { + t.Fatal("content dispatched inside actual GraphQL reserve") + } +} + +func TestAnalyticsQuotaDoesNotIncreaseOnAnomalousGraphQLSample(t *testing.T) { + reset := time.Now().UTC().Add(time.Hour).Truncate(time.Second) + balance := 3070 + shifted := reset + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + fmt.Fprintf(w, `{"resources":{"graphql":{"limit":20000,"remaining":19999,"reset":%d}}}`, reset.Unix()) + return + } + json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"rateLimit": map[string]any{"cost": 1, "limit": 20000, "remaining": balance, "resetAt": shifted.Format(time.RFC3339)}}}) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, Token: "test-token-placeholder", RateLimitReserve: 1500}) + first, _, err := c.AnalyticsRateLimit(context.Background()) + if err != nil { + t.Fatal(err) + } + balance = 19985 + shifted = reset.Add(4 * time.Second) + high, raw, err := c.AnalyticsRateLimit(context.Background()) + if err != nil || first.Remaining != 3070 || high.Remaining != 3070 || raw.Remaining != 19985 || high.ResetAt != shifted { + t.Fatalf("anomaly increased admission: first=%+v effective=%+v raw=%+v err=%v", first, high, raw, err) + } + balance = 3050 + shifted = reset + low, _, err := c.AnalyticsRateLimit(context.Background()) + if err != nil || low.Remaining != 3050 || !low.ResetAt.Equal(reset.Add(4*time.Second)) { + t.Fatalf("lower balance/reset boundary lost: %+v %v", low, err) + } + // REST refreshes cannot replace this GraphQL evidence. Only a real rollover + // after the later observed reset boundary can replenish the balance. + c.reserve.replace([]RateLimitSnapshot{{Resource: "graphql", Remaining: 19999, ResetAt: reset.Add(time.Hour)}}) + same := c.reserve.observeGraphQL(RateLimitSnapshot{Resource: "graphql", Remaining: 19900, ResetAt: reset.Add(time.Hour)}, reset.Add(2*time.Second)) + if same.Remaining != 3050 || !same.ResetAt.Equal(reset.Add(4*time.Second)) { + t.Fatal("early rollover increased balance") + } + next := c.reserve.observeGraphQL(RateLimitSnapshot{Resource: "graphql", Remaining: 19900, ResetAt: reset.Add(time.Hour)}, reset.Add(5*time.Second)) + if next.Remaining != 19900 { + t.Fatal("actual rollover could not refill budget") + } +} + +func TestGraphQLQuotaJitterAfterPreviousBoundaryCannotRefill(t *testing.T) { + now := time.Now().UTC() + r := newRateLimitReserve(3000, nil) + r.observeGraphQL(RateLimitSnapshot{Resource: "graphql", Remaining: 3050, ResetAt: now.Add(time.Minute)}, now) + held := r.observeGraphQL(RateLimitSnapshot{Resource: "graphql", Remaining: 19985, ResetAt: now.Add(time.Minute + 4*time.Second)}, now.Add(time.Minute+time.Second)) + if held.Remaining != 3050 || !held.ResetAt.Equal(now.Add(time.Minute+4*time.Second)) { + t.Fatalf("post-boundary jitter refilled: %+v", held) + } + renewed := r.observeGraphQL(RateLimitSnapshot{Resource: "graphql", Remaining: 19980, ResetAt: now.Add(time.Hour)}, now.Add(time.Minute+5*time.Second)) + if renewed.Remaining != 19980 { + t.Fatal("genuine rollover remained clamped") + } +} + +func TestGraphQLHistoryQuotaLogLabelsProviderAndEffectiveBalance(t *testing.T) { + reset := time.Now().UTC().Add(time.Hour).Truncate(time.Second) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + fmt.Fprintf(w, `{"resources":{"graphql":{"limit":20000,"remaining":19999,"reset":%d}}}`, reset.Unix()) + return + } + fmt.Fprintf(w, `{"data":{"rateLimit":{"cost":1,"limit":20000,"remaining":19985,"resetAt":%q}}}`, reset.Add(4*time.Second).Format(time.RFC3339)) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, RateLimitReserve: 3000}) + c.reserve.observeGraphQL(RateLimitSnapshot{Resource: "graphql", Remaining: 3070, ResetAt: reset}, time.Now()) + var messages []string + h := historySession{client: c, remaining: 20000, reporter: func(message string) { messages = append(messages, message) }} + data, err := h.request(context.Background(), "query { rateLimit { cost limit remaining resetAt } }", nil, 1) + if err != nil { + t.Fatal(err) + } + if raw, ok := historyInt(historyMap(data["rateLimit"])["remaining"]); !ok || raw != 19985 { + t.Fatal("raw provider evidence rewritten") + } + log := strings.Join(messages, "\n") + if !strings.Contains(log, "provider_remaining 19985") || !strings.Contains(log, "effective_remaining 3070") { + t.Fatal("raw and effective quota not labeled", log) + } +} diff --git a/internal/github/client.go b/internal/github/client.go index 612e0e92..310f924e 100644 --- a/internal/github/client.go +++ b/internal/github/client.go @@ -20,19 +20,23 @@ import ( type Reporter func(message string) type Client struct { - httpClient *http.Client - baseURL string - graphQLURL string - token string - tokenProvider func(context.Context) (string, error) - userAgent string - pageDelay time.Duration - rateLimit RateLimitObserver - reserve *rateLimitReserve + httpClient *http.Client + baseURL string + graphQLURL string + token string + tokenProvider func(context.Context) (string, error) + userAgent string + pageDelay time.Duration + rateLimit RateLimitObserver + reserve *rateLimitReserve + graphQLResponseLimit int64 + graphQLQuotaGuard bool } type Options struct { - Token string + GraphQLResponseLimit int64 + GraphQLQuotaGuard bool + Token string // TokenProvider exclusively selects credentials immediately before dispatch. TokenProvider func(context.Context) (string, error) BaseURL string @@ -41,7 +45,8 @@ type Options struct { PageDelay time.Duration RateLimit RateLimitObserver // RateLimitReserve preserves a best-effort observed floor for the shared - // token. Guarded requests refresh /rate_limit before dispatch so other token + // token. Unless GraphQLQuotaGuard uses explicit observed GraphQL quota, + // guarded requests refresh /rate_limit before dispatch so other token // consumers are observed, but unrelated consumers cannot be locked between // that probe and dispatch. RateLimitReserve int @@ -79,6 +84,45 @@ type rateLimitReserve struct { mu sync.Mutex reserve int snapshots map[string]RateLimitSnapshot + // GraphQL response evidence must survive REST snapshot replacement and + // upward provider anomalies until the observed reset boundary has passed. + graphqlObserved RateLimitSnapshot + graphqlToken string +} + +type graphQLQuotaProbeKey struct{} +type graphQLRequestEstimateKey struct{} + +func (r *rateLimitReserve) bindGraphQLToken(token string) { + if r == nil { + return + } + r.mu.Lock() + defer r.mu.Unlock() + if r.graphqlToken != token { + r.graphqlObserved = RateLimitSnapshot{} + r.graphqlToken = token + } +} + +func (r *rateLimitReserve) beforeObservedGraphQL(token string, cost int) error { + if r == nil { + return requestFailureAt("dispatch_guard", "quota_guard_missing", fmt.Errorf("observed GraphQL quota guard required")) + } + r.mu.Lock() + defer r.mu.Unlock() + q := r.graphqlObserved + if token != r.graphqlToken { + return requestFailureAt("dispatch_guard", "credential_changed", fmt.Errorf("GraphQL credential changed; quota probe required")) + } + if q.Resource != "graphql" || !q.ResetAt.After(time.Now()) { + return requestFailureAt("dispatch_guard", "quota_observation_stale", fmt.Errorf("fresh observed GraphQL quota required")) + } + if q.Remaining-cost < r.reserve { + return &RateLimitReserveError{RateLimit: q, Reserve: r.reserve} + } + r.graphqlObserved.Remaining -= cost + return nil } type rateLimitRequestLockKey struct{} @@ -133,14 +177,16 @@ func New(options Options) *Client { userAgent = "gitcrawl" } client := &Client{ - httpClient: httpClient, - baseURL: baseURL, - graphQLURL: graphQLURLForBaseURL(baseURL), - token: options.Token, - tokenProvider: options.TokenProvider, - userAgent: userAgent, - pageDelay: options.PageDelay, - rateLimit: options.RateLimit, + httpClient: httpClient, + baseURL: baseURL, + graphQLURL: graphQLURLForBaseURL(baseURL), + token: options.Token, + tokenProvider: options.TokenProvider, + userAgent: userAgent, + pageDelay: options.PageDelay, + graphQLResponseLimit: options.GraphQLResponseLimit, + graphQLQuotaGuard: options.GraphQLQuotaGuard, + rateLimit: options.RateLimit, } if options.RateLimitReserve > 0 { client.reserve = newRateLimitReserve(options.RateLimitReserve, options.InitialRateLimits) @@ -165,7 +211,7 @@ func (r *rateLimitReserve) beforeRequest(resource string, cost int) error { defer r.mu.Unlock() snapshot, ok := r.snapshots[resource] if !ok { - return fmt.Errorf("github %s rate limit status unavailable; cannot preserve reserve %d", resource, r.reserve) + return requestFailureAt("dispatch_guard", "quota_snapshot_missing", fmt.Errorf("github %s rate limit status unavailable; cannot preserve reserve %d", resource, r.reserve)) } if !snapshot.ResetAt.IsZero() && !time.Now().UTC().Before(snapshot.ResetAt) { return &rateLimitStatusExpiredError{RateLimit: snapshot} @@ -187,6 +233,28 @@ func (r *rateLimitReserve) observe(snapshot RateLimitSnapshot) { r.snapshots[snapshot.Resource] = snapshot } +func (r *rateLimitReserve) observeGraphQL(snapshot RateLimitSnapshot, now time.Time) RateLimitSnapshot { + if r == nil || snapshot.Remaining < 0 || !snapshot.ResetAt.After(now) { + return snapshot + } + r.mu.Lock() + defer r.mu.Unlock() + previous := r.graphqlObserved + nearbyLaterReset := !previous.ResetAt.IsZero() && snapshot.ResetAt.After(previous.ResetAt) && snapshot.ResetAt.Sub(previous.ResetAt) <= time.Minute + if previous.ResetAt.After(now) || nearbyLaterReset { + snapshot.Remaining = min(snapshot.Remaining, previous.Remaining) + // A shifted reset timestamp before the prior boundary is not a new + // window. Wait through both boundaries before accepting a refill. + // An early sample of the next hourly window must not postpone the + // current boundary by another hour. Only nearby reset jitter extends it. + if previous.ResetAt.After(snapshot.ResetAt) || snapshot.ResetAt.Sub(previous.ResetAt) > time.Minute { + snapshot.ResetAt = previous.ResetAt + } + } + r.graphqlObserved = snapshot + return snapshot +} + func (r *rateLimitReserve) replace(snapshots []RateLimitSnapshot) { if r == nil { return @@ -246,7 +314,7 @@ func (c *Client) getRateLimits(ctx context.Context, reporter Reporter, selectedT } defer resp.Body.Close() if err := json.NewDecoder(resp.Body).Decode(&payload); err != nil { - return nil, "", fmt.Errorf("decode github response: %w", err) + return nil, "", requestFailureAt("rest_quota_decode", "", fmt.Errorf("decode github response: %w", err)) } token := strings.TrimPrefix(resp.Request.Header.Get("Authorization"), "Bearer ") host := rateLimitHostForBaseURL(c.baseURL) @@ -525,7 +593,7 @@ func (c *Client) doOnce(ctx context.Context, method, path string, body io.Reader if targetErr != nil || originErr != nil || target.User != nil || origin.Host == "" || (target.Scheme != "https" && target.Scheme != "http") || !strings.EqualFold(target.Scheme, origin.Scheme) || !strings.EqualFold(target.Host, origin.Host) { - return nil, errors.New("GitHub token provider requires the configured API origin") + return nil, requestFailureAt("dispatch_guard", "origin_mismatch", errors.New("GitHub token provider requires the configured API origin")) } } resource, cost := c.requestRateLimit(method, fullURL) @@ -538,11 +606,12 @@ func (c *Client) doOnce(ctx context.Context, method, path string, body io.Reader } } var probeToken string - if c.reserve != nil && cost > 0 { + observedGraphQL := c.graphQLQuotaGuard && resource == "graphql" + if c.reserve != nil && cost > 0 && !observedGraphQL { var err error _, probeToken, err = c.getRateLimits(ctx, reporter, nil) if err != nil { - return nil, fmt.Errorf("refresh GitHub rate limit status: %w", err) + return nil, requestFailureAt("rest_preflight", "", fmt.Errorf("refresh GitHub rate limit status: %w", err)) } } token := c.token @@ -555,34 +624,50 @@ func (c *Client) doOnce(ctx context.Context, method, path string, body io.Reader return nil, err } } - if c.tokenProvider != nil && c.reserve != nil && cost > 0 && token != probeToken { + if c.tokenProvider != nil && c.reserve != nil && cost > 0 && !observedGraphQL && token != probeToken { // A new token cannot spend the previous token's quota. Allow one new // probe, then reject further rotation before the protected request. _, probeToken, err := c.getRateLimits(ctx, reporter, &token) if err != nil { - return nil, fmt.Errorf("refresh GitHub rate limit status: %w", err) + return nil, requestFailureAt("rest_preflight", "", fmt.Errorf("refresh GitHub rate limit status: %w", err)) } token, err = c.requestToken(ctx) if err != nil { return nil, err } if token != probeToken { - return nil, errors.New("GitHub token changed during rate limit reservation") + return nil, requestFailureAt("dispatch_guard", "credential_changed", errors.New("GitHub token changed during rate limit reservation")) } } - if err := c.reserve.beforeRequest(resource, cost); err != nil { - var expired *rateLimitStatusExpiredError - if c.tokenProvider != nil || !errors.As(err, &expired) { + if observedGraphQL { + probe, _ := ctx.Value(graphQLQuotaProbeKey{}).(bool) + if probe { + c.reserve.bindGraphQLToken(token) + } else if err := c.reserve.beforeObservedGraphQL(token, cost); err != nil { return nil, err } - _, refreshErr := c.GetRateLimits(ctx, reporter) - if refreshErr != nil { - return nil, fmt.Errorf("refresh GitHub rate limit status: %w", refreshErr) - } - if err := c.reserve.beforeRequest(resource, cost); err != nil { - return nil, err + } else if err := c.reserve.beforeRequest(resource, cost); err != nil { + var expired *rateLimitStatusExpiredError + if errors.As(err, &expired) && resource == "graphql" { + if err := c.refreshExpiredGraphQLQuota(ctx, token, cost); err != nil { + return nil, requestFailureAt("graphql_quota_refresh", "", err) + } + } else { + if c.tokenProvider != nil || !errors.As(err, &expired) { + return nil, requestFailureAt("dispatch_guard", "", err) + } + _, refreshErr := c.GetRateLimits(ctx, reporter) + if refreshErr != nil { + return nil, requestFailureAt("rest_preflight", "", fmt.Errorf("refresh GitHub rate limit status: %w", refreshErr)) + } + if err := c.reserve.beforeRequest(resource, cost); err != nil { + return nil, err + } } } + if resource == "graphql" && !observedGraphQL { + c.reserve.bindGraphQLToken(token) + } req, err := http.NewRequestWithContext(ctx, method, fullURL, body) if err != nil { return nil, err @@ -599,7 +684,7 @@ func (c *Client) doOnce(ctx context.Context, method, path string, body io.Reader reporter.Printf("[github] request %s %s", method, path) resp, err := c.guardedHTTPClient().Do(req) if err != nil { - return nil, fmt.Errorf("github request: %w", err) + return nil, requestFailureAt("transport", "", fmt.Errorf("github request: %w", err)) } responseResource, responseCost := c.requestRateLimit(resp.Request.Method, resp.Request.URL.String()) responseToken := strings.TrimPrefix(resp.Request.Header.Get("Authorization"), "Bearer ") @@ -620,6 +705,33 @@ func (c *Client) doOnce(ctx context.Context, method, path string, body io.Reader } } +// refreshExpiredGraphQLQuota runs only after an expired REST GraphQL snapshot, +// with requestMu already held. A quota-only probe may cross that stale boundary; +// content may not. Reuse the authoritative probe and conservative same-window +// accounting, pinning its credential until the protected dispatch is rechecked. +func (c *Client) refreshExpiredGraphQLQuota(ctx context.Context, token string, cost int) error { + quotaClient := *c + quotaClient.tokenProvider = func(context.Context) (string, error) { return token, nil } + _, observed, err := quotaClient.AnalyticsRateLimit(ctx) + if err != nil { + return err + } + if !observed.ResetAt.After(time.Now()) { + return &rateLimitStatusExpiredError{RateLimit: observed} + } + current, err := c.requestToken(ctx) + if err != nil { + return err + } + if current != token { + return requestFailureAt("dispatch_guard", "credential_changed", errors.New("GitHub token changed during quota refresh")) + } + // A history session checked its estimate before entering transport, using + // the prior observation. Recheck against the newly refreshed balance too. + estimate, _ := ctx.Value(graphQLRequestEstimateKey{}).(int) + return c.reserve.beforeObservedGraphQL(token, max(cost, estimate)) +} + func (c *Client) guardedHTTPClient() *http.Client { if c.reserve == nil && c.tokenProvider == nil { return c.httpClient @@ -649,7 +761,7 @@ func (c *Client) requestToken(ctx context.Context) (string, error) { if ctx.Err() != nil { return "", ctx.Err() } - return "", errors.New("GitHub token provider failed") + return "", requestFailureAt("credential", "credential_provider_failed", errors.New("GitHub token provider failed")) } return token, nil } diff --git a/internal/github/history.go b/internal/github/history.go index b0012a27..99610e21 100644 --- a/internal/github/history.go +++ b/internal/github/history.go @@ -16,26 +16,30 @@ import ( ) type HistoryItem struct { - Thread, Pull map[string]any - Comments, Reviews, ReviewComments []map[string]any + Thread, Pull map[string]any + Comments, Reviews, ReviewComments, ReviewThreads []map[string]any } type HistoryBatch struct { Repository map[string]any Items []HistoryItem } -const historyActor = `author { login __typename url }` +const historyActor = `author { login __typename url ... on Node { id } }` const historyComment = `id __typename fullDatabaseId body ` + historyActor + ` authorAssociation createdAt updatedAt publishedAt url isMinimized minimizedReason` const historyInline = historyComment + ` path diffHunk line startLine originalLine originalStartLine position originalPosition state subjectType outdated commit { oid } originalCommit { oid } replyTo { id fullDatabaseId } pullRequestReview { id fullDatabaseId }` var historyReview = historyComment + ` state submittedAt commit { oid } ` + historyConnection("comments", historyInline, "") -var historyReviewThread = `id __typename ` + historyConnection("comments", historyInline, "") +var historyReviewThread = `id __typename path line startLine isResolved isOutdated viewerCanResolve viewerCanUnresolve viewerCanReply ` + historyConnection("comments", historyInline, "") var historyCommon = `id __typename fullDatabaseId number title body ` + historyActor + ` authorAssociation createdAt updatedAt closedAt url state locked activeLockReason repository { nameWithOwner } milestone { number title state dueOn createdAt updatedAt url } ` + historyConnection("labels", `id name color description`, "") + " " + historyConnection("assignees", `id login __typename url`, "") + " " + historyConnection("comments", historyComment, "") var historyIssue = historyCommon + ` stateReason` var historyPull = historyCommon + ` isDraft merged mergedAt mergedBy { login __typename url } mergeCommit { oid } mergeable mergeStateStatus maintainerCanModify additions deletions changedFiles headRefName headRefOid baseRefName baseRefOid headRepository { nameWithOwner } baseRepository { nameWithOwner } commits { totalCount } ` + historyConnection("reviews", historyReview, "") + " " + historyConnection("reviewThreads", historyReviewThread, "") func historyConnection(name, fields, after string) string { - return name + `(first:20` + after + `) { totalCount pageInfo { hasNextPage endCursor } nodes { ` + fields + ` } }` + size := "20" + if after != "" && name != "reviews" && name != "reviewThreads" { + size = "100" + } + return name + `(first:` + size + after + `) { totalCount pageInfo { hasNextPage endCursor } nodes { ` + fields + ` } }` } type historySession struct { @@ -46,6 +50,10 @@ type historySession struct { retrySleep func(context.Context, time.Duration) error } +func (h *historySession) quota(ctx context.Context) (map[string]any, error) { + return h.request(context.WithValue(ctx, graphQLQuotaProbeKey{}, true), `query { rateLimit {cost remaining limit used resetAt} }`, nil, 1) +} + func (h *historySession) request(ctx context.Context, query string, variables map[string]any, estimate int) (map[string]any, error) { for attempt := 0; ; attempt++ { data, err := h.requestOnce(ctx, query, variables, estimate) @@ -103,17 +111,23 @@ func sleepHistoryRetry(ctx context.Context, duration time.Duration) error { func (h *historySession) requestOnce(ctx context.Context, query string, variables map[string]any, estimate int) (map[string]any, error) { if h.calls >= 1000 { - return nil, fmt.Errorf("GraphQL history pagination budget exceeded") + return nil, requestFailureAt("graphql_response", "pagination_budget", fmt.Errorf("GraphQL history pagination budget exceeded")) } - if h.remaining < 500+estimate { - return nil, fmt.Errorf("GraphQL history quota reserve reached") + reserve := 500 + if h.client.reserve != nil { + reserve = max(reserve, h.client.reserve.reserve) + } + // Retain the configured floor against actual GraphQL responses as well as + // the existing REST quota guard, including every pagination request. + if h.remaining < reserve+estimate { + return nil, fmt.Errorf("GraphQL history quota reserve reached: %w", &RateLimitReserveError{RateLimit: RateLimitSnapshot{Resource: "graphql", Remaining: h.remaining}, Reserve: reserve}) } h.calls++ // An unanswered request is charged conservatively by external supervisors. h.reporter.Printf("[github] graphql budget %d %d", h.calls, estimate) var data map[string]any started := time.Now() - err := h.client.doGraphQL(ctx, query, variables, h.reporter, &data) + err := h.client.doGraphQL(context.WithValue(ctx, graphQLRequestEstimateKey{}, estimate), query, variables, h.reporter, &data) h.reporter.Printf("[github] graphql timing %d %d", h.calls, time.Since(started).Milliseconds()) if err != nil { return nil, err @@ -121,17 +135,19 @@ func (h *historySession) requestOnce(ctx context.Context, query string, variable rate := historyMap(data["rateLimit"]) cost, ok := historyInt(rate["cost"]) if !ok || cost < 0 { - return nil, fmt.Errorf("GraphQL history missing cost") + return nil, requestFailureAt("graphql_response", "quota_cost_missing", fmt.Errorf("GraphQL history missing cost")) } remaining, ok := historyInt(rate["remaining"]) if !ok || remaining < 0 { - return nil, fmt.Errorf("GraphQL history missing remaining quota") + return nil, requestFailureAt("graphql_response", "quota_remaining_missing", fmt.Errorf("GraphQL history missing remaining quota")) } reset, err := time.Parse(time.RFC3339, historyString(rate["resetAt"])) if err != nil { - return nil, fmt.Errorf("GraphQL history invalid reset") + return nil, requestFailureAt("graphql_response", "quota_reset_invalid", fmt.Errorf("GraphQL history invalid reset")) } - h.remaining = min(h.remaining-cost, remaining) + effective := h.client.reserve.observeGraphQL(RateLimitSnapshot{Resource: "graphql", Remaining: remaining, ResetAt: reset}, time.Now()) + h.remaining = min(h.remaining-cost, effective.Remaining) + h.reporter.Printf("[github] graphql quota provider_remaining %d provider_reset %d effective_remaining %d effective_reset %d", remaining, reset.Unix(), h.remaining, effective.ResetAt.Unix()) h.reporter.Printf("[github] graphql cost %d %d remaining %d reset %d", h.calls, cost, remaining, reset.Unix()) return data, nil } @@ -178,10 +194,10 @@ func (c *Client) FetchGraphQLHistory(ctx context.Context, owner, repo string, nu node := historyMap(r[fmt.Sprintf("n%d", i)]) got, _ := historyInt(node["number"]) if got != n || !strings.EqualFold(historyString(historyMap(node["repository"])["nameWithOwner"]), owner+"/"+repo) || historyString(node["id"]) == "" { - return result, fmt.Errorf("GraphQL history item #%d unavailable or moved", n) + return result, historyFailure("identity", n, node, fmt.Errorf("GraphQL history item #%d unavailable or moved", n)) } if err := h.hydrate(ctx, node); err != nil { - return result, fmt.Errorf("GraphQL history #%d: %w", n, err) + return result, historyFailure("validation", n, node, fmt.Errorf("GraphQL history #%d: %w", n, err)) } item, err := historyItem(node) if err != nil { @@ -213,6 +229,13 @@ func historyFields(typ, key string) (string, error) { func (h *historySession) hydrate(ctx context.Context, node map[string]any) error { typ := historyString(node["__typename"]) + if typ == "PullRequestReviewThread" { + for _, field := range []string{"isResolved", "isOutdated", "viewerCanResolve", "viewerCanUnresolve", "viewerCanReply"} { + if _, ok := node[field].(bool); !ok { + return fmt.Errorf("missing or invalid review-thread %s", field) + } + } + } var required []string switch typ { case "Issue": @@ -232,6 +255,11 @@ func (h *historySession) hydrate(ctx context.Context, node map[string]any) error return fmt.Errorf("missing provider identity") } } + return h.hydrateConnections(ctx, node) +} + +func (h *historySession) hydrateConnections(ctx context.Context, node map[string]any) error { + typ := historyString(node["__typename"]) for _, key := range []string{"labels", "assignees", "comments", "reviews", "reviewThreads"} { connection, exists := node[key] if !exists { @@ -239,7 +267,7 @@ func (h *historySession) hydrate(ctx context.Context, node map[string]any) error } conn := historyMap(connection) if conn == nil { - return fmt.Errorf("missing %s connection", key) + return requestFailureAt("pagination_"+key, "connection_shape", fmt.Errorf("missing %s connection", key)) } fields, err := historyFields(typ, key) if err != nil { @@ -250,50 +278,50 @@ func (h *historySession) hydrate(ctx context.Context, node map[string]any) error page := historyMap(conn["pageInfo"]) next, ok := page["hasNextPage"].(bool) if !ok { - return fmt.Errorf("missing %s pageInfo", key) + return requestFailureAt("pagination_"+key, "connection_shape", fmt.Errorf("missing %s pageInfo", key)) } if !next { break } cursor := historyString(page["endCursor"]) if cursor == "" || seen[cursor] { - return fmt.Errorf("nonadvancing %s cursor", key) + return requestFailureAt("pagination_"+key, "connection_cursor", fmt.Errorf("nonadvancing %s cursor", key)) } seen[cursor] = true q := `query($id:ID!,$after:String!){node(id:$id){id ... on ` + typ + `{` + historyConnection(key, fields, `,after:$after`) + `}} rateLimit{cost remaining limit used resetAt}}` data, err := h.request(ctx, q, map[string]any{"id": node["id"], "after": cursor}, 2) if err != nil { - return err + return requestFailureAt("pagination_"+key, "", err) } parent := historyMap(data["node"]) if parent["id"] != node["id"] { - return fmt.Errorf("history pagination identity mismatch") + return requestFailureAt("pagination_"+key, "connection_identity", fmt.Errorf("history pagination identity mismatch")) } nxt := historyMap(parent[key]) a, ok := conn["nodes"].([]any) if !ok { - return fmt.Errorf("missing history nodes") + return requestFailureAt("pagination_"+key, "connection_shape", fmt.Errorf("missing history nodes")) } b, ok := nxt["nodes"].([]any) if !ok || len(b) == 0 { - return fmt.Errorf("empty history continuation") + return requestFailureAt("pagination_"+key, "connection_shape", fmt.Errorf("empty history continuation")) } conn["nodes"] = append(a, b...) conn["pageInfo"] = nxt["pageInfo"] } children, ok := conn["nodes"].([]any) if !ok { - return fmt.Errorf("missing history nodes") + return requestFailureAt("pagination_"+key, "connection_shape", fmt.Errorf("missing history nodes")) } if total, ok := historyInt(conn["totalCount"]); !ok || total != len(children) { - return fmt.Errorf("incomplete history %s count", key) + return requestFailureAt("pagination_"+key, "connection_count", fmt.Errorf("incomplete history %s count", key)) } ids := map[string]bool{} for _, child := range children { m := historyMap(child) id := historyString(m["id"]) if id == "" || ids[id] { - return fmt.Errorf("missing or duplicate history child identity") + return requestFailureAt("pagination_"+key, "connection_identity", fmt.Errorf("missing or duplicate history child identity")) } ids[id] = true if err := h.hydrate(ctx, m); err != nil { @@ -374,6 +402,7 @@ func historyItem(node map[string]any) (HistoryItem, error) { // A comment's review association is nullable. Threads independently // supply standalone comments; review bodies and their metadata stay above. for _, thread := range historyNodes(node, "reviewThreads") { + item.ReviewThreads = append(item.ReviewThreads, thread) for _, comment := range historyNodes(thread, "comments") { id := historyString(comment["id"]) if _, exists := inlineByID[id]; exists { diff --git a/internal/github/history_cause.go b/internal/github/history_cause.go new file mode 100644 index 00000000..4257108c --- /dev/null +++ b/internal/github/history_cause.go @@ -0,0 +1,145 @@ +package github + +import ( + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "io" + "net" + "net/url" +) + +// requestFailure adds source-owned diagnostic labels without changing the error +// text, unwrap chain, retry policy or acceptance decision. +type requestFailure struct { + cause error + stage, code string +} + +func (e *requestFailure) Error() string { return e.cause.Error() } +func (e *requestFailure) Unwrap() error { return e.cause } +func requestFailureAt(stage, code string, err error) error { + if err == nil { + return nil + } + return &requestFailure{cause: err, stage: stage, code: code} +} + +// All emitted strings are fixed labels. Never serialize Error(), URL, address, +// field/type names from JSON errors, certificate subjects, headers or tokens. +func safeHistoryCause(err error) map[string]any { + out := map[string]any{"category": "unknown", "type": "unclassified"} + stages := []string{} + stage := func(value string) { + switch value { + case "graphql_quota_refresh", "rest_preflight", "rest_quota_decode", "dispatch_guard", "credential", "transport", "graphql_request", "graphql_response", "validation", "identity", "partial_response", "response_decode", "response_size", "missing_data", "discovery_validation", "pagination_labels", "pagination_assignees", "pagination_comments", "pagination_reviews", "pagination_reviewThreads": + if len(stages) < 8 && (len(stages) == 0 || stages[len(stages)-1] != value) { + stages = append(stages, value) + } + } + } + set := func(category, typ, code string) { out["category"] = category; out["type"] = typ; out["code"] = code } + quota := func(q RateLimitSnapshot, reserve int) { + v := map[string]any{"remaining": q.Remaining, "limit": q.Limit} + switch q.Resource { + case "graphql", "core", "search": + v["resource"] = q.Resource + } + if !q.ResetAt.IsZero() { + v["reset_at"] = q.ResetAt.UTC().Format("2006-01-02T15:04:05Z07:00") + } + if reserve > 0 { + v["reserve"] = reserve + } + out["quota"] = v + } + for depth := 0; err != nil && depth < 16; depth++ { + switch e := err.(type) { + case *requestFailure: + stage(e.stage) + switch e.code { + case "quota_guard_missing", "quota_snapshot_missing", "quota_observation_stale", "credential_changed", "origin_mismatch": + set("guard", "native_guard", e.code) + case "credential_provider_failed": + set("credential", "native_provider", e.code) + case "quota_cost_missing", "quota_remaining_missing", "quota_reset_invalid", "pagination_budget": + set("validation", "native_validation", e.code) + case "connection_shape", "connection_cursor", "connection_identity", "connection_count": + set("validation", "native_validation", e.code) + } + case *HistoryFailure: + stage(e.Stage) + case *rateLimitStatusExpiredError: + set("guard", "quota_snapshot", "quota_snapshot_expired") + quota(e.RateLimit, 0) + case *RateLimitReserveError: + set("guard", "quota_reserve", "quota_reserve_reached") + quota(e.RateLimit, e.Reserve) + case *RequestError: + set("http", "github_http", "http_status") + if e.Status >= 100 && e.Status <= 599 { + out["http_status"] = e.Status + } + case *json.SyntaxError: + set("decode", "json_syntax", "invalid_json") + case *json.UnmarshalTypeError: + set("decode", "json_type", "invalid_json_type") + case *net.DNSError: + set("network", "dns", "dns_error") + out["timeout"] = e.Timeout() + case *net.OpError: + set("network", "net_operation", "network_error") + out["timeout"] = e.Timeout() + case *url.Error: + set("network", "url_request", "network_error") + out["timeout"] = e.Timeout() + case *tls.CertificateVerificationError: + set("tls", "certificate_verification", "certificate_invalid") + case x509.UnknownAuthorityError: + set("tls", "unknown_authority", "certificate_invalid") + case x509.HostnameError: + set("tls", "hostname", "certificate_invalid") + case x509.CertificateInvalidError: + set("tls", "certificate", "certificate_invalid") + } + switch err { + case context.Canceled: + set("cancelled", "context", "cancelled") + case context.DeadlineExceeded: + set("cancelled", "context", "deadline") + case io.EOF: + set("transport", "io", "eof") + case io.ErrUnexpectedEOF: + set("transport", "io", "unexpected_eof") + } + // A receipt describes its primary cause, not an unbounded joined error tree. + switch e := err.(type) { + case interface{ Unwrap() error }: + err = e.Unwrap() + case interface{ Unwrap() []error }: + children := e.Unwrap() + err = nil + if len(children) > 0 { + err = children[0] + } + default: + err = nil + } + } + if err != nil { + out["chain_truncated"] = true + } + out["stages"] = stages + return out +} + +func historyEvidenceWithCause(evidence json.RawMessage, err error) json.RawMessage { + var value map[string]json.RawMessage + if json.Unmarshal(evidence, &value) != nil || value == nil { + value = map[string]json.RawMessage{} + } + value["cause"], _ = json.Marshal(safeHistoryCause(err)) + out, _ := json.Marshal(value) + return out +} diff --git a/internal/github/history_cause_test.go b/internal/github/history_cause_test.go new file mode 100644 index 00000000..032cd099 --- /dev/null +++ b/internal/github/history_cause_test.go @@ -0,0 +1,243 @@ +package github + +import ( + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "net/http" + "net/http/httptest" + "net/url" + "reflect" + "strings" + "testing" + "time" +) + +func TestHistoryCauseKeepsOnlyBoundedTypedMetadata(t *testing.T) { + canary := "private-token-host-body-identity" + cases := []struct { + name string + err error + category, typ, code string + }{ + {"unknown", errors.New(canary), "unknown", "unclassified", ""}, + {"cancel", context.Canceled, "cancelled", "context", "cancelled"}, + {"deadline", context.DeadlineExceeded, "cancelled", "context", "deadline"}, + {"eof", io.EOF, "transport", "io", "eof"}, + {"short", io.ErrUnexpectedEOF, "transport", "io", "unexpected_eof"}, + {"json syntax", &json.SyntaxError{}, "decode", "json_syntax", "invalid_json"}, + {"json type", &json.UnmarshalTypeError{Value: canary, Field: canary, Struct: canary, Type: reflect.TypeOf(0)}, "decode", "json_type", "invalid_json_type"}, + {"dns", &net.DNSError{Name: canary, Server: canary, Err: canary, IsTimeout: true}, "network", "dns", "dns_error"}, + {"operation", &net.OpError{Op: canary, Net: canary, Err: errors.New(canary)}, "network", "net_operation", "network_error"}, + {"url", &url.Error{Op: canary, URL: "https://" + canary, Err: errors.New(canary)}, "network", "url_request", "network_error"}, + {"tls", &tls.CertificateVerificationError{Err: errors.New(canary)}, "tls", "certificate_verification", "certificate_invalid"}, + {"authority", x509.UnknownAuthorityError{}, "tls", "unknown_authority", "certificate_invalid"}, + {"hostname", x509.HostnameError{Host: canary}, "tls", "hostname", "certificate_invalid"}, + {"certificate", x509.CertificateInvalidError{Detail: canary}, "tls", "certificate", "certificate_invalid"}, + {"http", &RequestError{Method: canary, URL: canary, Status: 401, Body: canary, Headers: http.Header{"Authorization": []string{canary}}}, "http", "github_http", "http_status"}, + {"expired", &rateLimitStatusExpiredError{RateLimit: RateLimitSnapshot{Resource: "graphql", Remaining: 19999, ResetAt: time.Unix(100, 0)}}, "guard", "quota_snapshot", "quota_snapshot_expired"}, + {"reserve", &RateLimitReserveError{RateLimit: RateLimitSnapshot{Resource: "graphql", Remaining: 1500}, Reserve: 1500}, "guard", "quota_reserve", "quota_reserve_reached"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := safeHistoryCause(fmt.Errorf("%s: %w", canary, tc.err)) + if got["category"] != tc.category || got["type"] != tc.typ { + t.Fatalf("%+v", got) + } + if tc.code != "" && got["code"] != tc.code { + t.Fatalf("code=%v", got["code"]) + } + b, _ := json.Marshal(got) + if strings.Contains(string(b), canary) || len(b) > 2048 { + t.Fatal("unsafe/unbounded diagnostic", string(b)) + } + }) + } + for code, category := range map[string]string{"quota_guard_missing": "guard", "quota_snapshot_missing": "guard", "quota_observation_stale": "guard", "credential_changed": "guard", "origin_mismatch": "guard", "credential_provider_failed": "credential", "quota_cost_missing": "validation", "quota_remaining_missing": "validation", "quota_reset_invalid": "validation", "pagination_budget": "validation", "connection_shape": "validation", "connection_cursor": "validation", "connection_identity": "validation", "connection_count": "validation"} { + base := errors.New(canary) + wrapped := requestFailureAt("dispatch_guard", code, base) + got := safeHistoryCause(wrapped) + if got["code"] != code || got["category"] != category || !errors.Is(wrapped, base) || wrapped.Error() != base.Error() { + t.Fatalf("diagnostic changed behavior: %s %+v", code, got) + } + } + var long error = errors.New(canary) + for i := 0; i < 30; i++ { + stage := "rest_preflight" + if i%2 == 0 { + stage = "graphql_request" + } + long = requestFailureAt(stage, "", long) + } + got := safeHistoryCause(long) + if got["chain_truncated"] != true || len(got["stages"].([]string)) != 8 { + t.Fatal("unbounded cause chain", got) + } + unknown := safeHistoryCause(requestFailureAt(canary, canary, errors.New(canary))) + b, _ := json.Marshal(unknown) + if strings.Contains(string(b), canary) { + t.Fatal("untrusted label leaked") + } + joined := safeHistoryCause(errors.Join(requestFailureAt("rest_preflight", "", io.ErrUnexpectedEOF), errors.New(canary))) + if joined["code"] != "unexpected_eof" { + t.Fatal("primary joined cause lost") + } + if requestFailureAt("transport", "", nil) != nil { + t.Fatal("nil error manufactured") + } + // Retry eligibility must stay identical after diagnostic wrapping. + for _, e := range []error{io.EOF, io.ErrUnexpectedEOF, &RequestError{Status: 503}, &RequestError{Status: 401}, context.Canceled} { + if transientHistoryError(e) != transientHistoryError(requestFailureAt("graphql_request", "", e)) { + t.Fatal("retry semantics changed") + } + } +} + +func TestHistoryCauseDiagnosesNativePreflightWithoutBypassingGuards(t *testing.T) { + for _, mode := range []string{"expired", "missing", "decode", "rotation", "http", "missing_cost", "missing_remaining", "invalid_reset"} { + t.Run(mode, func(t *testing.T) { + gql, credentials := 0, 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + if mode == "decode" { + fmt.Fprint(w, `{"resources":private-response`) + return + } + if mode == "missing" { + fmt.Fprint(w, `{"resources":{}}`) + return + } + if mode == "http" { + http.Error(w, "private-response", 401) + return + } + reset := time.Now().Add(time.Hour).Unix() + if mode == "expired" { + reset = time.Now().Add(-time.Minute).Unix() + } + fmt.Fprintf(w, `{"resources":{"graphql":{"limit":20000,"remaining":19000,"reset":%d}}}`, reset) + return + } + gql++ + rate := map[string]any{"cost": 1, "limit": 20000, "remaining": 19000, "resetAt": time.Now().UTC().Add(time.Hour).Format(time.RFC3339)} + if mode == "expired" { + rate["resetAt"] = time.Now().UTC().Add(-time.Minute).Format(time.RFC3339) + } + if mode == "missing_cost" { + delete(rate, "cost") + } + if mode == "missing_remaining" { + delete(rate, "remaining") + } + if mode == "invalid_reset" { + rate["resetAt"] = "private-response" + } + json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"rateLimit": rate}}) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, RateLimitReserve: 1500, TokenProvider: func(context.Context) (string, error) { + credentials++ + if mode == "rotation" { + return fmt.Sprint("test-token-", credentials), nil + } + return "test-token-placeholder", nil + }}) + batch, err := c.FetchGraphQLHistory(context.Background(), "fixture", "repo", []int{1}, nil) + if err == nil || len(batch.Items) != 0 { + t.Fatal("failed evidence accepted") + } + class, _, evidence := HistoryFailureDetails(err) + var record struct { + Cause struct { + Code string `json:"code"` + Status int `json:"http_status"` + Stages []string `json:"stages"` + } `json:"cause"` + } + if e := json.Unmarshal(evidence, &record); e != nil { + t.Fatal(e) + } + want := map[string]string{"expired": "quota_snapshot_expired", "missing": "quota_snapshot_missing", "decode": "invalid_json", "rotation": "credential_changed", "http": "http_status", "missing_cost": "quota_cost_missing", "missing_remaining": "quota_remaining_missing", "invalid_reset": "quota_reset_invalid"}[mode] + if record.Cause.Code != want { + t.Fatalf("mode=%s class=%s evidence=%s", mode, class, evidence) + } + if mode == "http" { + if class != "http" || record.Cause.Status != 401 { + t.Fatal("HTTP status lost") + } + } else if class != "fetch" { + t.Fatal("existing error class changed", class) + } + if strings.Contains(string(evidence), "private-response") || strings.Contains(string(evidence), "test-token") { + t.Fatal("private value leaked") + } + if mode == "expired" && gql != 1 { + t.Fatal("expected only the authoritative quota refresh") + } + if mode == "missing" || mode == "decode" || mode == "rotation" || mode == "http" { + if gql != 0 { + t.Fatal("guard dispatched GraphQL content") + } + } + }) + } +} + +func TestHistoryCauseRetainsPaginationGuardAndAllowsNormalRetry(t *testing.T) { + expired := true + restCalls, gqlCalls := 0, 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + restCalls++ + reset := time.Now().Add(time.Hour).Unix() + if expired && restCalls == 3 { + reset = time.Now().Add(-time.Minute).Unix() + } + fmt.Fprintf(w, `{"resources":{"graphql":{"limit":20000,"remaining":19000,"reset":%d}}}`, reset) + return + } + gqlCalls++ + var req graphqlEnvelope + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Error(err) + return + } + data := map[string]any{"rateLimit": map[string]any{"cost": 1, "limit": 20000, "remaining": 19000, "resetAt": time.Now().UTC().Add(time.Hour).Format(time.RFC3339)}} + if expired && restCalls == 3 { + historyMap(data["rateLimit"])["resetAt"] = time.Now().UTC().Add(-time.Minute).Format(time.RFC3339) + } + if strings.Contains(req.Query, "issueOrPullRequest") { + node := historyTestNode() + node["labels"] = map[string]any{"totalCount": 2, "nodes": []any{map[string]any{"id": "L1", "name": "one"}}, "pageInfo": map[string]any{"hasNextPage": true, "endCursor": "first"}} + data["repository"] = map[string]any{"databaseId": 42, "nameWithOwner": "fixture/repo", "n0": node} + } + if req.Variables["after"] != nil { + data["node"] = map[string]any{"id": "PR_fixture", "labels": historyTestConnection(map[string]any{"id": "L2", "name": "two"})} + } + json.NewEncoder(w).Encode(map[string]any{"data": data}) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, RateLimitReserve: 1500, TokenProvider: func(context.Context) (string, error) { return "test-token-placeholder", nil }}) + batch, err := c.FetchGraphQLHistory(context.Background(), "fixture", "repo", []int{1}, nil) + if err == nil || len(batch.Items) != 0 || gqlCalls != 3 { + t.Fatal("partial pagination accepted") + } + class, _, evidence := HistoryFailureDetails(err) + var record map[string]json.RawMessage + if err = json.Unmarshal(evidence, &record); err != nil { + t.Fatal(err) + } + if class != "validation" || record["structure"] == nil || !strings.Contains(string(record["cause"]), "pagination_labels") || !strings.Contains(string(record["cause"]), "quota_snapshot_expired") { + t.Fatalf("lost wrapper or cause: %s", evidence) + } + expired = false + batch, err = c.FetchGraphQLHistory(context.Background(), "fixture", "repo", []int{1}, nil) + if err != nil || len(batch.Items) != 1 || len(historyNodes(historyMap(batch.Items[0].Thread["_graphql"]), "labels")) != 2 { + t.Fatalf("normal retry: items=%d err=%v", len(batch.Items), err) + } +} diff --git a/internal/github/history_evidence.go b/internal/github/history_evidence.go new file mode 100644 index 00000000..89e9cd6a --- /dev/null +++ b/internal/github/history_evidence.go @@ -0,0 +1,185 @@ +package github + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "hash" + "io" + "strings" +) + +// HistoryFailure retains a bounded structural receipt, never credentials, HTTP +// headers, prose bodies or actor/profile text. Existing accepted raw evidence +// remains in the archive. A rejected body is represented by its length/hash. +type HistoryFailure struct { + Cause error + Stage string + Number int + Evidence json.RawMessage +} + +type historyResponseReader struct { + reader io.Reader + hash hash.Hash + read, hashed int64 +} + +func (r *historyResponseReader) Read(p []byte) (int, error) { + n, err := r.reader.Read(p) + r.read += int64(n) + keep := min(int64(n), 128*1024-r.hashed) + if keep > 0 { + _, _ = r.hash.Write(p[:keep]) + r.hashed += keep + } + return n, err +} + +func (r *historyResponseReader) failure(err error) *HistoryFailure { + evidence, _ := json.Marshal(map[string]any{"version": 1, "response_bytes_read": r.read, "hashed_prefix_bytes": r.hashed, "prefix_sha256": hex.EncodeToString(r.hash.Sum(nil)), "complete_response": false}) + return &HistoryFailure{Cause: err, Stage: "response_decode", Evidence: evidence} +} + +func (e *HistoryFailure) Error() string { return e.Cause.Error() } +func (e *HistoryFailure) Unwrap() error { return e.Cause } + +// Error messages can contain prose or identities. Retain only bounded provider +// codes and known query path components, separately from the existing receipt. +func graphQLRejection(data any, failures []graphqlResponseError, cause error) error { + var evidence map[string]json.RawMessage + // SafeHistoryEvidence always wraps data in an object, including nil data. + _ = json.Unmarshal(SafeHistoryEvidence(data), &evidence) + items := make([]map[string]any, 0, min(len(failures), 8)) + for _, failure := range failures[:min(len(failures), 8)] { + var code any + switch failure.Type { + case "NOT_FOUND", "FORBIDDEN", "UNAUTHORIZED", "UNPROCESSABLE", "RATE_LIMITED", "INTERNAL", "INTERNAL_SERVER_ERROR", "SERVICE_UNAVAILABLE", "MAX_NODE_LIMIT_EXCEEDED", "EXCESSIVE_PAGINATION", "RESOURCE_LIMITS_EXCEEDED": + code = failure.Type + } + path := make([]any, 0, min(len(failure.Path), 8)) + for _, component := range failure.Path[:min(len(failure.Path), 8)] { + var safe any + switch value := component.(type) { + case string: + switch value { + case "query", "repository", "node", "nodes", "issue", "pullRequest", "issueOrPullRequest", "issues", "pullRequests", "comments", "reviews", "reviewThreads", "labels", "assignees", "edges", "pageInfo", "totalCount", "hasNextPage", "endCursor", "rateLimit", "id", "__typename", "body", "author", "state", "isResolved", "isOutdated": + safe = value + } + if len(value) >= 2 && len(value) <= 3 && value[0] == 'n' && strings.Trim(value[1:], "0123456789") == "" { + safe = value // Native generated aliases, never entity IDs. + } + case json.Number: + if index, err := value.Int64(); err == nil && index >= 0 && index <= 10000 { + safe = index + } + } + path = append(path, safe) + } + items = append(items, map[string]any{"type": code, "path": path, "path_truncated": len(failure.Path) > 8}) + } + evidence["graphql_errors"], _ = json.Marshal(map[string]any{"count": len(failures), "items": items, "truncated": len(failures) > 8}) + encoded, _ := json.Marshal(evidence) + return &HistoryFailure{Cause: cause, Stage: "partial_response", Evidence: encoded} +} + +func historyFailure(stage string, number int, data any, err error) error { + evidence := SafeHistoryEvidence(data) + var upstream *HistoryFailure + if errors.As(err, &upstream) { + stage = upstream.Stage + evidence, _ = json.Marshal(map[string]json.RawMessage{"context": evidence, "upstream_rejection": upstream.Evidence}) + } + return &HistoryFailure{Cause: err, Stage: stage, Number: number, Evidence: evidence} +} + +func SafeHistoryEvidence(data any) json.RawMessage { + raw, _ := json.Marshal(data) + sum := sha256.Sum256(raw) + var project func(any, int) any + project = func(value any, depth int) any { + if depth > 9 { + return map[string]any{"truncated": true} + } + switch v := value.(type) { + case map[string]any: + out := map[string]any{} + for k, child := range v { + if len(k) > 1 && k[0] == 'n' && strings.Trim(k[1:], "0123456789") == "" { + out[k] = project(child, depth+1) + continue + } + switch k { + case "id", "node_id", "fullDatabaseId", "number", "__typename", "totalCount", "hasNextPage", "endCursor", "createdAt", "updatedAt", "publishedAt", "submittedAt", "state", "isResolved", "isOutdated", "type": + switch scalar := child.(type) { + case string: + if len(scalar) <= 512 { + out[k] = scalar + } else { + out[k] = map[string]any{"truncated": true} + } + case bool, float64, int, int64, json.Number, nil: + out[k] = scalar + default: + out[k] = map[string]any{"invalid_type": true} + } + case "repository", "node", "data", "nodes", "pageInfo", "comments", "reviews", "reviewThreads", "labels", "assignees", "errors", "path": + out[k] = project(child, depth+1) + case "body": + if text, ok := child.(string); ok { + h := sha256.Sum256([]byte(text)) + out["body_evidence"] = map[string]any{"bytes": len(text), "sha256": hex.EncodeToString(h[:])} + } + } + } + return out + case []any: + items := make([]any, 0, min(len(v), 12)) + for _, item := range v[:min(len(v), 12)] { + items = append(items, project(item, depth+1)) + } + return map[string]any{"count": len(v), "sample": items, "truncated": len(v) > 12} + default: + // Unknown strings may be provider error messages or private prose. + return nil + } + } + out, _ := json.Marshal(map[string]any{"version": 1, "response_bytes": len(raw), "response_sha256": hex.EncodeToString(sum[:]), "structure": project(data, 0)}) + if len(out) > 128*1024 { + out, _ = json.Marshal(map[string]any{"version": 1, "response_bytes": len(raw), "response_sha256": hex.EncodeToString(sum[:]), "structure_truncated": true}) + } + return out +} + +// HistoryFailureDetails is safe to persist or log even if the original error +// included an HTTP body. It deliberately does not return that body/message. +func HistoryFailureDetails(err error) (string, string, json.RawMessage) { + diagnostic := historyEvidenceWithCause(json.RawMessage(`{}`), err) + if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { + return "cancelled", "GraphQL attempt cancelled or timed out", diagnostic + } + var quota *RateLimitReserveError + if errors.As(err, "a) { + return "rate_limit", quota.Error(), diagnostic + } + var failure *HistoryFailure + if errors.As(err, &failure) { + message := fmt.Sprintf("GraphQL %s rejected for item %d", failure.Stage, failure.Number) + for _, connection := range []string{"comments", "reviews", "reviewThreads", "labels", "assignees"} { + for _, reason := range []string{"incomplete history " + connection + " count", "nonadvancing " + connection + " cursor", "missing history " + connection} { + if strings.HasSuffix(failure.Cause.Error(), reason) { + message += ": " + reason + } + } + } + return failure.Stage, message, historyEvidenceWithCause(failure.Evidence, err) + } + var response *RequestError + if errors.As(err, &response) { + return "http", fmt.Sprintf("GitHub HTTP %d", response.Status), diagnostic + } + return "fetch", "GraphQL collection failed", diagnostic +} diff --git a/internal/github/history_evidence_test.go b/internal/github/history_evidence_test.go new file mode 100644 index 00000000..c4a43226 --- /dev/null +++ b/internal/github/history_evidence_test.go @@ -0,0 +1,142 @@ +package github + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func TestHistoryFailureEvidenceRetainsStructureWithoutSecretsOrBodies(t *testing.T) { + data := map[string]any{"id": "PR_fixture", "body": "private prose", "Authorization": "Bearer secret", "author": map[string]any{"login": "private-login"}, "comments": map[string]any{"totalCount": 2, "nodes": []any{map[string]any{"id": "C1", "body": "retained-by-hash"}}, "pageInfo": map[string]any{"hasNextPage": false}}} + evidence := SafeHistoryEvidence(data) + for _, secret := range []string{"private prose", "Bearer secret", "private-login", "retained-by-hash"} { + if strings.Contains(string(evidence), secret) { + t.Fatal("private data in safe receipt") + } + } + if !json.Valid(evidence) || !strings.Contains(string(evidence), `"totalCount":2`) || !strings.Contains(string(evidence), `"sha256"`) { + t.Fatal("missing structural evidence") + } + err := historyFailure("validation", 7, data, errors.New("GraphQL history #7: incomplete history comments count")) + class, message, stored := HistoryFailureDetails(err) + var preserved map[string]json.RawMessage + if err := json.Unmarshal(stored, &preserved); err != nil { + t.Fatal(err) + } + if preserved["cause"] == nil { + t.Fatal("missing safe cause metadata") + } + delete(preserved, "cause") + originalFields, _ := json.Marshal(preserved) + if class != "validation" || !strings.Contains(message, "incomplete history comments count") || string(originalFields) != string(evidence) { + t.Fatalf("receipt %s %s", class, message) + } +} + +func TestMalformedGraphQLResponseHasPrivateBoundedReceipt(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`{"data":{"body":"private malformed response"`)) + })) + defer server.Close() + var out map[string]any + err := New(Options{BaseURL: server.URL}).doGraphQL(context.Background(), "query { rateLimit { cost } }", nil, nil, &out) + if err == nil { + t.Fatal("malformed JSON accepted") + } + class, _, evidence := HistoryFailureDetails(err) + if class != "response_decode" || !strings.Contains(string(evidence), "prefix_sha256") || strings.Contains(string(evidence), "private malformed response") { + t.Fatalf("bad rejection receipt: %s %s", class, evidence) + } +} + +func TestReviewThreadMissingStateFailsClosed(t *testing.T) { + node := map[string]any{"id": "T1", "__typename": "PullRequestReviewThread", "comments": historyTestConnection()} + h := historySession{} + if err := h.hydrate(context.Background(), node); err == nil { + t.Fatal("unknown resolution became false") + } + for _, field := range []string{"isResolved", "isOutdated", "viewerCanResolve", "viewerCanUnresolve", "viewerCanReply"} { + node[field] = false + } + node["isResolved"] = true + if err := h.hydrate(context.Background(), node); err != nil { + t.Fatal(err) + } +} + +func TestGraphQLRejectedResponseKeepsOnlyBoundedSafeErrorMetadata(t *testing.T) { + failures := []any{} + for i := 0; i < 12; i++ { + typ := "NOT_FOUND" + if i == 1 { + typ = "private-identity" + } + failures = append(failures, map[string]any{"type": typ, "message": "private provider prose", "path": []any{"repository", "n0", "comments", 0, "body", "private-identity", "IC_private_identity", 1000000000, "omitted"}}) + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"repository": map[string]any{"n0": nil, "n1": map[string]any{"body": "private peer prose"}}}, "errors": failures}) + })) + defer server.Close() + out := map[string]any{"unchanged": true} + err := New(Options{BaseURL: server.URL}).doGraphQL(context.Background(), "query { repository { id } }", nil, nil, &out) + if err == nil || len(out) != 1 || out["unchanged"] != true { + t.Fatal("partial data accepted") + } + class, _, evidence := HistoryFailureDetails(err) + if class != "partial_response" { + t.Fatal(class) + } + for _, private := range []string{"private provider prose", "private peer prose", "private-identity", "IC_private_identity", "omitted"} { + if strings.Contains(string(evidence), private) { + t.Fatal("unsafe rejection metadata retained", private) + } + } + var receipt struct { + Errors struct { + Count int `json:"count"` + Items []struct { + Type *string `json:"type"` + Path []any `json:"path"` + Truncated bool `json:"path_truncated"` + } `json:"items"` + Truncated bool `json:"truncated"` + } `json:"graphql_errors"` + } + if err = json.Unmarshal(evidence, &receipt); err != nil { + t.Fatal(err) + } + if receipt.Errors.Count != 12 || len(receipt.Errors.Items) != 8 || !receipt.Errors.Truncated { + t.Fatal("error bound lost") + } + first := receipt.Errors.Items[0] + if first.Type == nil || *first.Type != "NOT_FOUND" || len(first.Path) != 8 || !first.Truncated || first.Path[0] != "repository" || first.Path[1] != "n0" || first.Path[3] != float64(0) || first.Path[5] != nil || first.Path[6] != nil || first.Path[7] != nil || receipt.Errors.Items[1].Type != nil { + t.Fatalf("incorrect safe metadata: %+v", receipt.Errors) + } +} + +func TestGraphQLRejectedNullOrAbsentDataRetainsSafeEnvelope(t *testing.T) { + for _, payload := range []string{ + `{"errors":[{"type":"NOT_FOUND","path":["repository","n0"],"message":"private prose"}]}`, + `{"data":null,"errors":[{"type":"NOT_FOUND","path":["repository","n0"],"message":"private prose"}]}`, + } { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(payload)) })) + var out map[string]any + err := New(Options{BaseURL: server.URL}).doGraphQL(context.Background(), "query { repository { id } }", nil, nil, &out) + server.Close() + if err == nil { + t.Fatal("error envelope accepted") + } + class, _, evidence := HistoryFailureDetails(err) + var record map[string]any + if e := json.Unmarshal(evidence, &record); e != nil { + t.Fatal(e) + } + if class != "partial_response" || record["version"] != float64(1) || record["graphql_errors"] == nil || record["structure"] != nil || strings.Contains(string(evidence), "private prose") { + t.Fatalf("missing/redaction-invalid envelope: %s", evidence) + } + } +} diff --git a/internal/github/history_test.go b/internal/github/history_test.go index 3c3a88c8..60036525 100644 --- a/internal/github/history_test.go +++ b/internal/github/history_test.go @@ -179,6 +179,13 @@ func TestGraphQLHistoryReviewThreadCompleteness(t *testing.T) { node["reviews"] = historyTestConnection(review) thread1 := map[string]any{"id": "T1", "__typename": "PullRequestReviewThread", "comments": historyTestConnection(associated)} thread2 := map[string]any{"id": "T2", "__typename": "PullRequestReviewThread", "comments": historyTestConnection()} + for _, thread := range []map[string]any{thread1, thread2} { + for _, field := range []string{"isResolved", "isOutdated", "viewerCanResolve", "viewerCanUnresolve", "viewerCanReply"} { + thread[field] = false + } + } + thread1["isResolved"] = true + thread2["isOutdated"] = true thread2["comments"].(map[string]any)["totalCount"] = 1 thread2["comments"].(map[string]any)["pageInfo"] = map[string]any{"hasNextPage": true, "endCursor": "comment-first"} node["reviewThreads"] = historyTestConnection(thread1) @@ -247,9 +254,12 @@ func TestGraphQLHistoryReviewThreadCompleteness(t *testing.T) { t.Fatal(err) } item := batch.Items[0] - if pages != 2 || len(item.ReviewComments) != 2 || len(item.Reviews) != 1 || len(item.Comments) != 1 { + if pages != 2 || len(item.ReviewComments) != 2 || len(item.Reviews) != 1 || len(item.Comments) != 1 || len(item.ReviewThreads) != 2 { t.Fatalf("incomplete or duplicate conversation: pages=%d item=%+v", pages, item) } + if item.ReviewThreads[0]["isResolved"] != true || item.ReviewThreads[1]["isOutdated"] != true { + t.Fatal("review state lost") + } if item.Reviews[0]["state"] != "APPROVED" || item.Reviews[0]["body"] != "" || item.Comments[0]["body"] != "discussion" { t.Fatal("review metadata or discussion lost") } diff --git a/internal/github/quota_refresh_test.go b/internal/github/quota_refresh_test.go new file mode 100644 index 00000000..7de4ff06 --- /dev/null +++ b/internal/github/quota_refresh_test.go @@ -0,0 +1,352 @@ +package github + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "reflect" + "strings" + "sync/atomic" + "testing" + "time" +) + +// A fresh REST reply can still contain an expired GraphQL resource. The +// credential-bound GraphQL observation must be refreshed, never invented. +func TestExpiredGraphQLRESTSnapshotRefreshesFromProvider(t *testing.T) { + rest, probes, content := 0, 0, 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + rest++ + writeRateLimits(t, w, time.Now().Add(-time.Second), 19983, 19983) + return + } + var request graphqlEnvelope + if err := json.NewDecoder(r.Body).Decode(&request); err != nil { + t.Error(err) + return + } + if request.Query == "query { viewer { id } }" { + content++ + } else { + probes++ + } + json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"viewer": map[string]any{"id": "fixture"}, "rateLimit": map[string]any{"cost": 1, "limit": 20000, "remaining": 19000, "resetAt": time.Now().UTC().Add(time.Hour).Format(time.RFC3339)}}}) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, RateLimitReserve: 1500, TokenProvider: func(context.Context) (string, error) { return "fixture", nil }}) + var result map[string]any + if err := c.doGraphQL(context.Background(), "query { viewer { id } }", nil, nil, &result); err != nil { + t.Fatal(err) + } + if rest != 1 || probes != 1 || content != 1 || historyString(historyMap(result["viewer"])["id"]) != "fixture" { + t.Fatalf("rest=%d probes=%d content=%d result=%v", rest, probes, content, result) + } +} + +func TestExpiredGraphQLQuotaRefreshFailsClosed(t *testing.T) { + for _, mode := range []string{"low", "stale", "missing_cost", "missing_remaining", "bad_reset", "missing_limit", "http", "partial", "decode", "rotation", "credential_failure", "cancel"} { + t.Run(mode, func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + probes, content, credentials := 0, 0, 0 + canary := "fixture-private-value" + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + writeRateLimits(t, w, time.Now().Add(-time.Second), 19983, 19983) + return + } + var q graphqlEnvelope + json.NewDecoder(r.Body).Decode(&q) + if strings.Contains(q.Query, "viewer") { + content++ + t.Error("content dispatched after failed refresh") + return + } + probes++ + if r.Header.Get("Authorization") != "Bearer fixture" { + t.Error("refresh credential changed") + } + switch mode { + case "http": + http.Error(w, canary, 401) + return + case "decode": + fmt.Fprint(w, "{") + return + case "cancel": + cancel() + // Wait until the client closes this request before the handler + // can return an empty 200 and race cancellation with EOF. + <-r.Context().Done() + return + } + rate := map[string]any{"cost": 1, "limit": 20000, "remaining": 19000, "resetAt": time.Now().UTC().Add(time.Hour).Format(time.RFC3339)} + switch mode { + case "low": + rate["remaining"] = 1500 + case "stale": + rate["resetAt"] = time.Now().UTC().Add(-time.Second).Format(time.RFC3339) + case "missing_cost": + delete(rate, "cost") + case "missing_remaining": + delete(rate, "remaining") + case "bad_reset": + rate["resetAt"] = canary + case "missing_limit": + delete(rate, "limit") + } + envelope := map[string]any{"data": map[string]any{"rateLimit": rate}} + if mode == "partial" { + envelope["errors"] = []any{map[string]any{"type": "FORBIDDEN", "message": canary, "path": []any{"rateLimit"}}} + } + json.NewEncoder(w).Encode(envelope) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, RateLimitReserve: 1500, TokenProvider: func(context.Context) (string, error) { + credentials++ + if credentials == 3 { + if mode == "rotation" { + return "replacement", nil + } + if mode == "credential_failure" { + return "", errors.New(canary) + } + } + return "fixture", nil + }}) + var out map[string]any + err := c.doGraphQL(ctx, "query { viewer { id } }", nil, nil, &out) + wantProbes := 1 + if mode == "decode" { + wantProbes = 3 + } + if err == nil || probes != wantProbes || content != 0 || out != nil { + t.Fatalf("err=%v probes=%d content=%d out=%v", err, probes, content, out) + } + cause := safeHistoryCause(err) + code, _ := cause["code"].(string) + wants := map[string]string{"low": "quota_reserve_reached", "stale": "quota_snapshot_expired", "missing_cost": "quota_cost_missing", "missing_remaining": "quota_remaining_missing", "bad_reset": "quota_reset_invalid", "http": "http_status", "decode": "unexpected_eof", "rotation": "credential_changed", "credential_failure": "credential_provider_failed", "cancel": "cancelled"} + if want := wants[mode]; want != "" && code != want { + t.Fatalf("cause=%v want=%s", cause, want) + } + _, _, evidence := HistoryFailureDetails(err) + if strings.Contains(string(evidence), canary) || !strings.Contains(string(evidence), "graphql_quota_refresh") { + t.Fatalf("unsafe or missing refresh evidence: %s", evidence) + } + }) + } +} + +func TestExpiredGraphQLQuotaRefreshBindsRotatedCredential(t *testing.T) { + var probes, content []string + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + credential := r.Header.Get("Authorization") + if r.URL.Path == "/rate_limit" { + probes = append(probes, credential) + writeRateLimits(t, w, time.Now().Add(-time.Second), 19983, 19983) + return + } + var q graphqlEnvelope + json.NewDecoder(r.Body).Decode(&q) + if strings.Contains(q.Query, "viewer") { + content = append(content, credential) + } else { + probes = append(probes, credential) + } + fmt.Fprint(w, `{"data":{"rateLimit":{"cost":1,"limit":20000,"remaining":19000,"resetAt":"2099-01-01T00:00:00Z"}}}`) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, RateLimitReserve: 1500, TokenProvider: sequenceTokenProvider(t, "first", "second", "second", "second")}) + // Observations from the old credential must not constrain or admit the new one. + c.reserve.bindGraphQLToken("first") + c.reserve.observeGraphQL(RateLimitSnapshot{Resource: "graphql", Remaining: 0, ResetAt: time.Now().Add(time.Hour)}, time.Now()) + var out map[string]any + if err := c.doGraphQL(context.Background(), "query { viewer { id } }", nil, nil, &out); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(probes, []string{"Bearer first", "Bearer second", "Bearer second"}) || !reflect.DeepEqual(content, []string{"Bearer second"}) { + t.Fatalf("wrong binding: probes=%v content=%v", probes, content) + } +} + +func TestExpiredGraphQLQuotaRefreshConcurrentReserve(t *testing.T) { + var active, peak, probes, content atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + n := active.Add(1) + defer active.Add(-1) + for p := peak.Load(); n > p; p = peak.Load() { + if peak.CompareAndSwap(p, n) { + break + } + } + if r.URL.Path == "/rate_limit" { + writeRateLimits(t, w, time.Now().Add(-time.Second), 19983, 19983) + return + } + var q graphqlEnvelope + json.NewDecoder(r.Body).Decode(&q) + balance := 19999 + if strings.Contains(q.Query, "viewer") { + content.Add(1) + } else if probes.Add(1) == 1 { + balance = 1501 + } + fmt.Fprintf(w, `{"data":{"rateLimit":{"cost":1,"limit":20000,"remaining":%d,"resetAt":"2099-01-01T00:00:00Z"}}}`, balance) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, RateLimitReserve: 1500, TokenProvider: func(context.Context) (string, error) { return "fixture", nil }}) + results := make(chan error, 8) + for range 8 { + go func() { + var out map[string]any + results <- c.doGraphQL(context.Background(), "query { viewer { id } }", nil, nil, &out) + }() + } + success, blocked := 0, 0 + for range 8 { + err := <-results + var floor *RateLimitReserveError + if err == nil { + success++ + } else if errors.As(err, &floor) { + blocked++ + } else { + t.Fatal(err) + } + } + if success != 1 || blocked != 7 || content.Load() != 1 || probes.Load() != 8 || peak.Load() != 1 { + t.Fatalf("success=%d blocked=%d content=%d probes=%d simultaneous=%d", success, blocked, content.Load(), probes.Load(), peak.Load()) + } +} + +func TestExpiredGraphQLQuotaRefreshRefusesRedirect(t *testing.T) { + var leaked atomic.Int32 + target := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { leaked.Add(1) })) + defer target.Close() + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + writeRateLimits(t, w, time.Now().Add(-time.Second), 19983, 19983) + return + } + http.Redirect(w, r, target.URL, http.StatusFound) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, RateLimitReserve: 1500, TokenProvider: func(context.Context) (string, error) { return "fixture", nil }}) + var out map[string]any + err := c.doGraphQL(context.Background(), "query { viewer { id } }", nil, nil, &out) + var req *RequestError + if !errors.As(err, &req) || req.Status != 302 || leaked.Load() != 0 { + t.Fatalf("redirect: err=%v target=%d", err, leaked.Load()) + } +} + +func TestExpiredGraphQLQuotaRefreshPreservesPaginationValidation(t *testing.T) { + for _, mode := range []string{"complete", "short_page", "low_page", "stale_refresh"} { + t.Run(mode, func(t *testing.T) { + rest, probes, pages := 0, 0, 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + rest++ + reset := time.Now().Add(time.Hour) + if rest == 3 { + reset = time.Now().Add(-time.Second) + } + writeRateLimits(t, w, reset, 19983, 19983) + return + } + var q graphqlEnvelope + json.NewDecoder(r.Body).Decode(&q) + rate := map[string]any{"cost": 1, "limit": 20000, "remaining": 19000, "resetAt": "2099-01-01T00:00:00Z"} + data := map[string]any{"rateLimit": rate} + if strings.Contains(q.Query, "issueOrPullRequest") { + n := historyTestNode() + n["labels"] = map[string]any{"totalCount": 2, "nodes": []any{map[string]any{"id": "L1", "name": "one"}}, "pageInfo": map[string]any{"hasNextPage": true, "endCursor": "first"}} + data["repository"] = map[string]any{"databaseId": 42, "nameWithOwner": "fixture/repo", "n0": n} + } else if q.Variables["after"] != nil { + pages++ + if q.Variables["after"] != "first" || q.Variables["id"] != "PR_fixture" { + t.Error("continuation identity/cursor changed") + } + page := historyTestConnection(map[string]any{"id": "L2", "name": "two"}) + if mode == "short_page" { + page = historyTestConnection() + } + data["node"] = map[string]any{"id": "PR_fixture", "labels": page} + } else if rest == 3 { + probes++ + if mode == "low_page" { + rate["remaining"] = 1501 + } + if mode == "stale_refresh" { + rate["resetAt"] = time.Now().UTC().Add(-time.Second).Format(time.RFC3339) + } + } + json.NewEncoder(w).Encode(map[string]any{"data": data}) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, RateLimitReserve: 1500, TokenProvider: func(context.Context) (string, error) { return "fixture", nil }}) + batch, err := c.FetchGraphQLHistory(context.Background(), "fixture", "repo", []int{1}, nil) + if probes != 1 || rest != 3 { + t.Fatalf("unbounded refresh: probes=%d rest=%d", probes, rest) + } + if mode == "complete" { + if err != nil || pages != 1 || len(batch.Items) != 1 { + t.Fatalf("items=%d pages=%d err=%v", len(batch.Items), pages, err) + } + item := batch.Items[0] + if item.Thread["id"] != "PR_fixture" || item.Thread["body"] != "body" || item.Pull["id"] != "9007199254740993" || len(historyNodes(historyMap(item.Thread["_graphql"]), "labels")) != 2 { + t.Fatal("content/identity/membership changed") + } + } else { + if err == nil || len(batch.Items) != 0 { + t.Fatal("incomplete membership accepted") + } + if mode == "low_page" { + var floor *RateLimitReserveError + if !errors.As(err, &floor) || pages != 0 { + t.Fatalf("refreshed quota ignored page estimate: pages=%d err=%v", pages, err) + } + } + if mode == "stale_refresh" && pages != 0 { + t.Fatal("old valid observation masked invalid refresh") + } + } + }) + } +} + +func TestFreshGraphQLQuotaDoesNotTriggerRefresh(t *testing.T) { + for _, provider := range []bool{false, true} { + t.Run(fmt.Sprint(provider), func(t *testing.T) { + rest, content := 0, 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/rate_limit" { + rest++ + writeRateLimits(t, w, time.Now().Add(time.Hour), 19000, 19000) + return + } + content++ + var q graphqlEnvelope + json.NewDecoder(r.Body).Decode(&q) + if q.Query != "query { viewer { id } }" { + t.Error("unexpected refresh") + } + fmt.Fprint(w, `{"data":{"viewer":{"id":"fixture"}}}`) + })) + defer server.Close() + opts := Options{BaseURL: server.URL, RateLimitReserve: 1500, Token: "fixture"} + if provider { + opts.TokenProvider = func(context.Context) (string, error) { return "fixture", nil } + } + c := New(opts) + var out map[string]any + if err := c.doGraphQL(context.Background(), "query { viewer { id } }", nil, nil, &out); err != nil || rest != 1 || content != 1 { + t.Fatalf("rest=%d content=%d err=%v", rest, content, err) + } + }) + } +} diff --git a/internal/github/review_state.go b/internal/github/review_state.go new file mode 100644 index 00000000..0de13454 --- /dev/null +++ b/internal/github/review_state.go @@ -0,0 +1,68 @@ +package github + +import ( + "context" + "fmt" + "strings" + "time" +) + +// ReviewStateItem contains only review evidence. It must never enter the +// full-thread projection, since canonical bodies/comments were not requested. +type ReviewStateItem struct { + Number int + NodeID string + RepositoryID string + RepositoryNodeID string + UpdatedAt string + Threads []map[string]any +} + +func (c *Client) FetchGraphQLReviewState(ctx context.Context, owner, repo string, numbers []int, reporter Reporter) ([]ReviewStateItem, error) { + if len(numbers) == 0 || len(numbers) > 8 { + return nil, fmt.Errorf("review-state query requires 1..8 numbers") + } + h := historySession{client: c, reporter: reporter, remaining: 20000} + if _, err := h.quota(ctx); err != nil { + return nil, err + } + var fields strings.Builder + for i, n := range numbers { + if n < 1 { + return nil, fmt.Errorf("invalid review-state number") + } + fmt.Fprintf(&fields, `n%d: issueOrPullRequest(number:%d) {__typename ... on PullRequest{id number updatedAt repository{nameWithOwner} %s}} `, i, n, historyConnection("reviewThreads", historyReviewThread, "")) + } + data, err := h.request(ctx, `query($owner:String!,$repo:String!){repository(owner:$owner,name:$repo){id databaseId nameWithOwner `+fields.String()+`} rateLimit{cost remaining limit used resetAt}}`, map[string]any{"owner": owner, "repo": repo}, 16) + if err != nil { + return nil, err + } + r := historyMap(data["repository"]) + if !strings.EqualFold(historyString(r["nameWithOwner"]), owner+"/"+repo) || historyString(r["id"]) == "" { + return nil, historyFailure("identity", 0, r, fmt.Errorf("review-state repository identity mismatch")) + } + repoID, validRepoID := historyInt(r["databaseId"]) + if !validRepoID || repoID <= 0 { + return nil, historyFailure("identity", 0, r, fmt.Errorf("missing repository database identity")) + } + out := make([]ReviewStateItem, 0, len(numbers)) + for i, n := range numbers { + node := historyMap(r[fmt.Sprint("n", i)]) + got, valid := historyInt(node["number"]) + if !valid || got != n || historyString(node["__typename"]) != "PullRequest" || historyString(node["id"]) == "" || !strings.EqualFold(historyString(historyMap(node["repository"])["nameWithOwner"]), owner+"/"+repo) { + return nil, historyFailure("identity", n, node, fmt.Errorf("review-state PR identity mismatch")) + } + updated := historyString(node["updatedAt"]) + if _, err = time.Parse(time.RFC3339Nano, updated); err != nil { + return nil, historyFailure("validation", n, node, fmt.Errorf("invalid review-state source time")) + } + if historyMap(node["reviewThreads"]) == nil { + return nil, historyFailure("validation", n, node, fmt.Errorf("missing history reviewThreads")) + } + if err = h.hydrateConnections(ctx, map[string]any{"__typename": "PullRequest", "id": node["id"], "reviewThreads": node["reviewThreads"]}); err != nil { + return nil, historyFailure("validation", n, node, err) + } + out = append(out, ReviewStateItem{Number: n, RepositoryID: fmt.Sprint(repoID), RepositoryNodeID: historyString(r["id"]), NodeID: historyString(node["id"]), UpdatedAt: updated, Threads: historyNodes(node, "reviewThreads")}) + } + return out, nil +} diff --git a/internal/github/review_state_test.go b/internal/github/review_state_test.go new file mode 100644 index 00000000..a94101a2 --- /dev/null +++ b/internal/github/review_state_test.go @@ -0,0 +1,166 @@ +package github + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +func TestReviewStateOnlyStrictPaginationAndReplyFields(t *testing.T) { + for _, mode := range []string{"complete", "null", "short", "cursor", "wrong_parent", "missing_state"} { + t.Run(mode, func(t *testing.T) { + conn := func(total int, more bool, cursor string, nodes ...any) map[string]any { + if nodes == nil { + nodes = []any{} + } + return map[string]any{"totalCount": total, "nodes": nodes, "pageInfo": map[string]any{"hasNextPage": more, "endCursor": cursor}} + } + comment := func(id string, reply any) map[string]any { + return map[string]any{"id": id, "__typename": "PullRequestReviewComment", "fullDatabaseId": id, "body": "inline retained", "author": map[string]any{"id": "actor", "login": "fixture", "__typename": "User"}, "url": "https://github.com/fixture/repo/pull/7#comment", "createdAt": "2026-01-01T00:00:00Z", "updatedAt": "2026-01-02T00:00:00Z", "replyTo": reply} + } + thread := func(id string, comments any) map[string]any { + return map[string]any{"id": id, "__typename": "PullRequestReviewThread", "isResolved": true, "isOutdated": false, "viewerCanResolve": false, "viewerCanUnresolve": true, "viewerCanReply": true, "path": "file.go", "line": 9, "comments": comments} + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req struct { + Query string + Variables map[string]any + } + json.NewDecoder(r.Body).Decode(&req) + data := map[string]any{"rateLimit": map[string]any{"cost": 1, "limit": 20000, "remaining": 19000, "resetAt": time.Now().UTC().Add(time.Hour).Format(time.RFC3339)}} + if strings.Contains(req.Query, "issueOrPullRequest") { + if strings.Contains(req.Query, "title") || strings.Contains(req.Query, "labels") || strings.Contains(req.Query, "reviews(") { + t.Error("unrelated history requested") + } + first := thread("RT1", conn(2, true, "c1", comment("C1", nil))) + if mode == "missing_state" { + delete(first, "isResolved") + } + node := map[string]any{"__typename": "PullRequest", "id": "PR7", "number": 7, "updatedAt": "2026-01-02T00:00:00Z", "repository": map[string]any{"nameWithOwner": "fixture/repo"}, "reviewThreads": conn(2, true, "rt1", first)} + if mode == "short" { + node["reviewThreads"] = conn(2, false, "end", first) + } + var selected any = node + if mode == "null" { + selected = nil + } + data["repository"] = map[string]any{"id": "R1", "databaseId": 1, "nameWithOwner": "fixture/repo", "n0": selected} + } else if req.Variables["id"] == "PR7" { + parent := "PR7" + if mode == "wrong_parent" { + parent = "OTHER" + } + data["node"] = map[string]any{"id": parent, "reviewThreads": conn(2, mode == "cursor", "rt1", thread("RT2", conn(0, false, "")))} + } else if req.Variables["id"] == "RT1" { + data["node"] = map[string]any{"id": "RT1", "comments": conn(2, false, "c2", comment("C2", map[string]any{"id": "C1", "fullDatabaseId": "C1"}))} + } + json.NewEncoder(w).Encode(map[string]any{"data": data}) + })) + defer server.Close() + items, err := New(Options{BaseURL: server.URL}).FetchGraphQLReviewState(context.Background(), "fixture", "repo", []int{7}, nil) + if mode != "complete" { + if err == nil { + t.Fatal("incomplete evidence accepted", mode) + } + return + } + if err != nil || len(items) != 1 || len(items[0].Threads) != 2 { + t.Fatalf("%+v %v", items, err) + } + comments := historyNodes(items[0].Threads[0], "comments") + if len(comments) != 2 || historyMap(comments[1]["replyTo"])["id"] != "C1" || comments[0]["body"] != "inline retained" || historyMap(comments[0]["author"])["login"] != "fixture" { + t.Fatal("reply/body/author evidence lost") + } + }) + } +} +func TestReviewStateResponseByteLimitRejectsWithoutPartialResult(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Write([]byte(`{"data":{"padding":"` + strings.Repeat("x", 8192) + `"}}`)) + })) + defer server.Close() + var out map[string]any + err := New(Options{BaseURL: server.URL, GraphQLResponseLimit: 1024}).doGraphQL(context.Background(), "query { rateLimit {cost} }", nil, nil, &out) + class, _, _ := HistoryFailureDetails(err) + if err == nil || class != "response_size" || out != nil { + t.Fatalf("oversize response accepted: %v %s", err, class) + } +} + +func TestReviewStateGraphQLGuardRequiresObservedCredentialQuota(t *testing.T) { + for _, mode := range []string{"normal", "low_probe", "low_page", "rotation", "expired", "unprobed"} { + t.Run(mode, func(t *testing.T) { + calls, rest, tokenCalls := 0, 0, 0 + reset := time.Now().UTC().Add(time.Hour) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/graphql" { + rest++ + t.Error("redundant REST quota request") + http.Error(w, "unexpected", 500) + return + } + calls++ + remaining := 19000 + if mode == "low_probe" || mode == "low_page" && calls == 2 { + remaining = 3000 + } + if mode == "expired" { + reset = time.Now().Add(-time.Second) + } + json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"rateLimit": map[string]any{"cost": 1, "limit": 20000, "remaining": remaining, "resetAt": reset.Format(time.RFC3339)}}}) + })) + defer server.Close() + c := New(Options{BaseURL: server.URL, GraphQLQuotaGuard: true, RateLimitReserve: 3000, TokenProvider: func(context.Context) (string, error) { + tokenCalls++ + if mode == "rotation" && tokenCalls > 1 { + return "changed-fixture", nil + } + return "fixture", nil + }}) + h := historySession{client: c, remaining: 20000} + if mode != "unprobed" { + if _, err := h.quota(context.Background()); err != nil { + t.Fatal(err) + } + } + _, err := h.request(context.Background(), `query { node(id:"fixture"){id} rateLimit{cost remaining limit resetAt}}`, nil, 16) + if mode == "normal" || mode == "low_page" { + if err != nil { + t.Fatal(err) + } + _, err = h.request(context.Background(), `query { node(id:"page"){id} rateLimit{cost remaining limit resetAt}}`, nil, 16) + } + if mode == "normal" { + if err != nil || calls != 3 { + t.Fatalf("calls=%d err=%v", calls, err) + } + } else if err == nil { + t.Fatal("unguarded content accepted") + } + expected := 1 + if mode == "normal" { + expected = 3 + } + if mode == "low_page" { + expected = 2 + } + if mode == "unprobed" { + expected = 0 + } + if calls != expected || rest != 0 { + t.Fatalf("calls=%d rest=%d", calls, rest) + } + if mode == "low_probe" || mode == "low_page" { + var reserve *RateLimitReserveError + if !errors.As(err, &reserve) { + t.Fatalf("not reserve error: %v", err) + } + } + }) + } +} diff --git a/internal/github/review_threads.go b/internal/github/review_threads.go index 1d8cf1f2..390f3ab5 100644 --- a/internal/github/review_threads.go +++ b/internal/github/review_threads.go @@ -3,8 +3,10 @@ package github import ( "bytes" "context" + "crypto/sha256" "encoding/json" "fmt" + "io" "net/http" "strings" ) @@ -112,10 +114,14 @@ type graphqlEnvelope struct { } type graphqlResponseEnvelope struct { - Data json.RawMessage `json:"data"` - Errors []struct { - Message string `json:"message"` - } `json:"errors"` + Data json.RawMessage `json:"data"` + Errors []graphqlResponseError `json:"errors"` +} + +type graphqlResponseError struct { + Message string `json:"message"` + Type string `json:"type"` + Path []any `json:"path"` } // ListPullReviewThreads fetches GitHub's review-thread graph for a pull request. @@ -235,21 +241,40 @@ func (c *Client) doGraphQL(ctx context.Context, query string, variables map[stri return fmt.Errorf("encode graphql request: %w", err) } var envelope graphqlResponseEnvelope - if err := c.doJSON(ctx, http.MethodPost, c.graphQLURL, bytes.NewReader(payload), reporter, &envelope); err != nil { - return err + response, err := c.do(ctx, http.MethodPost, c.graphQLURL, bytes.NewReader(payload), reporter) + if err != nil { + return requestFailureAt("graphql_request", "", err) + } + defer response.Body.Close() + reader := &historyResponseReader{reader: response.Body, hash: sha256.New()} + var input io.Reader = reader + if c.graphQLResponseLimit > 0 { + input = io.LimitReader(reader, c.graphQLResponseLimit+1) + } + decodeErr := decodeJSON(input, &envelope) + reporter.Printf("[github] graphql bytes %d", reader.read) + if c.graphQLResponseLimit > 0 && reader.read > c.graphQLResponseLimit { + failure := reader.failure(fmt.Errorf("review-state response exceeded byte limit")) + failure.Stage = "response_size" + return failure + } + if err := decodeErr; err != nil { + return reader.failure(fmt.Errorf("decode github response: %w", err)) } if len(envelope.Errors) > 0 { messages := make([]string, 0, len(envelope.Errors)) for _, graphqlErr := range envelope.Errors { messages = append(messages, graphqlErr.Message) } - return fmt.Errorf("github graphql: %s", strings.Join(messages, "; ")) + var rejected any + _ = decodeJSON(bytes.NewReader(envelope.Data), &rejected) + return graphQLRejection(rejected, envelope.Errors, fmt.Errorf("github graphql: %s", strings.Join(messages, "; "))) } if len(envelope.Data) == 0 || string(envelope.Data) == "null" { - return fmt.Errorf("github graphql response missing data") + return historyFailure("missing_data", 0, nil, fmt.Errorf("github graphql response missing data")) } if err := decodeJSON(bytes.NewReader(envelope.Data), out); err != nil { - return fmt.Errorf("decode github graphql data: %w", err) + return historyFailure("response_decode", 0, nil, fmt.Errorf("decode github graphql data: %w", err)) } return nil } diff --git a/internal/store/analytics_actor_recovery_test.go b/internal/store/analytics_actor_recovery_test.go new file mode 100644 index 00000000..ffb024ef --- /dev/null +++ b/internal/store/analytics_actor_recovery_test.go @@ -0,0 +1,129 @@ +package store + +import ( + "context" + "encoding/json" + "path/filepath" + "reflect" + "slices" + "testing" + "time" +) + +func TestAnalyticsActorRecoverySeedsAndRefreshesNativeIdentitiesAtomically(t *testing.T) { + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + // Historical rows predate the enqueue-on-capture path. + _, err = s.DB().ExecContext(ctx, `INSERT INTO repositories(id,owner,name,full_name,github_repo_id,raw_json,updated_at) VALUES(1,'fixture','repo','fixture/repo','1','{}','2026-01-01'); + INSERT INTO threads(id,repo_id,github_id,number,kind,state,title,html_url,labels_json,assignees_json,raw_json,content_hash,updated_at) VALUES + (1,1,'1',1,'issue','open','fixture','','[]','[]','{"node_id":"content-unknown","user":null}','h','2026-01-01'), + (2,1,'2',2,'issue','open','fixture','','[]','[]','{"node_id":"content-known","user":{"node_id":"actor-known"}}','h','2026-01-01')`) + if err != nil { + t.Fatal(err) + } + for _, profiles := range []bool{false, true} { + if err = s.SeedAnalyticsNodes(ctx, profiles); err != nil { + t.Fatal(err) + } + } + assertNodes := func(profiles bool, want []string) { + t.Helper() + var got []string + var err error + if profiles { + got, err = s.AnalyticsProfileNodes(ctx, 100) + } else { + got, err = s.AnalyticsIdentityNodes(ctx, 100) + } + if err != nil { + t.Fatal(err) + } + slices.Sort(got) + slices.Sort(want) + if !slices.Equal(got, want) { + t.Fatalf("profiles=%v nodes=%v want%v", profiles, got, want) + } + } + assertNodes(false, []string{"content-unknown"}) + assertNodes(true, []string{"actor-known"}) + now := time.Now().UTC() + at := now.Format(time.RFC3339Nano) + evidence := map[string]any{"id": "content-unknown", "author": map[string]any{"id": "actor-recovered", "login": "shared-login", "__typename": "User"}} + if err = s.SaveActorEvidence(ctx, []map[string]any{evidence}, at); err != nil { + t.Fatal(err) + } + assertNodes(false, nil) + assertNodes(true, []string{"actor-known", "actor-recovered"}) + profile := func(id, name string) map[string]any { + return map[string]any{"id": id, "login": "shared-login", "__typename": "User", "name": name, "bio": "fixture profile", "url": "https://github.com/shared-login", "createdAt": "2026-01-01T02:00:00+02:00"} + } + first, second := profile("actor-known", "Known"), profile("actor-recovered", "Recovered") + if err = s.SaveActorProfiles(ctx, []map[string]any{first, second}, at); err != nil { + t.Fatal(err) + } + assertNodes(true, nil) + var count int + if err = s.DB().QueryRowContext(ctx, "SELECT count(*) FROM actor_profiles WHERE login='shared-login'").Scan(&count); err != nil || count != 2 { + t.Fatalf("login reuse merged native actors: %d %v", count, err) + } + read := func(id string) (string, string, string) { + t.Helper() + var raw, created, observed string + if err := s.DB().QueryRowContext(ctx, "SELECT raw_json,created_at,observed_at FROM actor_profiles WHERE node_id=?", id).Scan(&raw, &created, &observed); err != nil { + t.Fatal(err) + } + return raw, created, observed + } + raw, created, observed := read("actor-known") + var retained map[string]any + if err = json.Unmarshal([]byte(raw), &retained); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(retained, first) || created != "2026-01-01T00:00:00Z" || observed != at { + t.Fatal("native profile evidence or provider date changed") + } + changed := profile("actor-known", "Changed") + invalid := profile("actor-recovered", "Must not apply") + invalid["unsupported"] = make(chan int) + if err = s.SaveActorProfiles(ctx, []map[string]any{changed, invalid}, at); err == nil { + t.Fatal("invalid batch accepted") + } + after, _, _ := read("actor-known") + if after != raw { + t.Fatal("failed later profile leaked an earlier update") + } + // Failed identity capture must roll back its newly enqueued profile as well. + invalidEvidence := map[string]any{"id": "other-content", "author": map[string]any{"id": "actor-leaked"}, "unsupported": make(chan int)} + if err = s.SaveActorEvidence(ctx, []map[string]any{invalidEvidence}, at); err == nil { + t.Fatal("invalid actor evidence accepted") + } + assertNodes(true, nil) + if err = s.DB().QueryRowContext(ctx, "SELECT count(*) FROM actor_identity_evidence").Scan(&count); err != nil || count != 1 { + t.Fatalf("failed identity batch changed evidence: %d %v", count, err) + } + // Fresh profiles are not polled repeatedly, but become eligible after 24 hours. + if err = s.SaveActorProfiles(ctx, []map[string]any{first}, now.Add(-25*time.Hour).Format(time.RFC3339Nano)); err != nil { + t.Fatal(err) + } + assertNodes(true, []string{"actor-known"}) + if err = s.SaveActorProfiles(ctx, []map[string]any{changed}, at); err != nil { + t.Fatal(err) + } + assertNodes(true, nil) + refreshed, _, _ := read("actor-known") + if err = json.Unmarshal([]byte(refreshed), &retained); err != nil || retained["name"] != "Changed" || retained["id"] != "actor-known" { + t.Fatalf("refresh lost native identity: %+v %v", retained, err) + } + if err = s.SeedAnalyticsNodes(ctx, true); err != nil { + t.Fatal(err) + } + if err = s.SeedAnalyticsNodes(ctx, false); err != nil { + t.Fatal(err) + } + assertNodes(true, nil) + assertNodes(false, nil) +} diff --git a/internal/store/analytics_integrity.go b/internal/store/analytics_integrity.go new file mode 100644 index 00000000..9418d23c --- /dev/null +++ b/internal/store/analytics_integrity.go @@ -0,0 +1,446 @@ +package store + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "strings" + "time" +) + +// Diagnostic receipts and retry state are local operational evidence, not +// public conversation payloads. Resolved rows are retained, never reset/deleted. +func (s *Store) ensureAnalyticsIntegritySchema(ctx context.Context) error { + if err := s.ensureColumn(ctx, "pull_request_review_thread_syncs", "review_thread_ids_json", "text"); err != nil { + return err + } + _, err := s.q().ExecContext(ctx, ` +CREATE TABLE IF NOT EXISTS analytics_fetch_attempts( + id INTEGER PRIMARY KEY,repository TEXT NOT NULL,number INTEGER NOT NULL,operation TEXT NOT NULL, + started_at TEXT NOT NULL,finished_at TEXT NOT NULL,status TEXT NOT NULL, + error_class TEXT,error_text TEXT,evidence_json TEXT NOT NULL); +CREATE INDEX IF NOT EXISTS analytics_attempt_item ON analytics_fetch_attempts(repository,number,id); +CREATE INDEX IF NOT EXISTS analytics_attempt_repository ON analytics_fetch_attempts(repository,id); +CREATE TABLE IF NOT EXISTS analytics_retries( + repository TEXT NOT NULL,number INTEGER NOT NULL,operation TEXT NOT NULL, + first_seen_at TEXT NOT NULL,last_seen_at TEXT NOT NULL,next_attempt_at TEXT NOT NULL, + attempts INTEGER NOT NULL DEFAULT 0,last_attempt_id INTEGER,resolved_at TEXT, + PRIMARY KEY(repository,number,operation)); +CREATE INDEX IF NOT EXISTS analytics_retry_due ON analytics_retries(repository,resolved_at,next_attempt_at,number); +CREATE TABLE IF NOT EXISTS analytics_review_state_coverage( + repository TEXT PRIMARY KEY,cursor INTEGER NOT NULL,ceiling INTEGER NOT NULL, + scanned INTEGER NOT NULL,queued INTEGER NOT NULL,pending_items INTEGER NOT NULL, + scan_complete INTEGER NOT NULL,complete INTEGER NOT NULL,observed_at TEXT NOT NULL); +`) + return err +} + +type AnalyticsAttempt struct { + Repository string `json:"repository"` + Number int `json:"number"` + Operation string `json:"operation"` + StartedAt string `json:"started_at"` + FinishedAt string `json:"finished_at"` + Status string `json:"status"` + ErrorClass string `json:"error_class,omitempty"` + ErrorText string `json:"error_text,omitempty"` + Evidence json.RawMessage `json:"evidence"` +} + +func (s *Store) RecordAnalyticsAttempt(ctx context.Context, a AnalyticsAttempt) error { + if a.Repository == "" || a.Operation == "" || (a.Status != "success" && a.Status != "failed") || !json.Valid(a.Evidence) { + return fmt.Errorf("invalid analytics attempt") + } + finished, err := time.Parse(time.RFC3339Nano, a.FinishedAt) + if err != nil { + return err + } + return s.WithTx(ctx, func(tx *Store) error { + excluded, err := tx.ThreadExcluded(ctx, a.Repository, a.Number) + if err != nil { + return err + } + if excluded { + return nil + } + status, class, message := a.Status, a.ErrorClass, a.ErrorText + knownReview := true + if status == "success" && (a.Operation == "graphql_history" || a.Operation == "review_state") { + if err := tx.q().QueryRowContext(ctx, `SELECT EXISTS(SELECT 1 FROM threads t JOIN repositories r ON r.id=t.repo_id WHERE r.full_name=? COLLATE NOCASE AND t.number=? AND (t.kind<>'pull_request' OR EXISTS(SELECT 1 FROM pull_request_review_thread_syncs x WHERE x.thread_id=t.id AND x.review_thread_ids_json IS NOT NULL)))`, a.Repository, a.Number).Scan(&knownReview); err != nil { + return err + } + if a.Operation == "review_state" && !knownReview { + status = "failed" + class = "unapplied_review_state" + message = "Fetch did not establish a review-state membership observation" + } + } + r, err := tx.q().ExecContext(ctx, `INSERT INTO analytics_fetch_attempts(repository,number,operation,started_at,finished_at,status,error_class,error_text,evidence_json) VALUES(?,?,?,?,?,?,?,?,?)`, a.Repository, a.Number, a.Operation, a.StartedAt, a.FinishedAt, status, nullString(class), nullString(message), string(a.Evidence)) + if err != nil { + return err + } + id, err := r.LastInsertId() + if err != nil { + return err + } + if status == "success" { + _, err = tx.q().ExecContext(ctx, `UPDATE analytics_retries SET resolved_at=?,last_attempt_id=? WHERE repository=? AND number=? AND resolved_at IS NULL AND (operation=? OR (?='graphql_history' AND operation IN ('graphql_history','review_state'))) AND (operation<>'review_state' OR ?)`, a.FinishedAt, id, a.Repository, a.Number, a.Operation, a.Operation, knownReview) + return err + } + if a.Operation != "review_state" { + if _, err = tx.q().ExecContext(ctx, "UPDATE analytics_coverage SET complete=0 WHERE repository=?", a.Repository); err != nil { + return err + } + } + var attempts int + if err = tx.q().QueryRowContext(ctx, `SELECT coalesce((SELECT attempts FROM analytics_retries WHERE repository=? AND number=? AND operation=?),0)`, a.Repository, a.Number, a.Operation).Scan(&attempts); err != nil { + return err + } + delay := time.Duration(1<0 AND next_attempt_at<=? AND NOT EXISTS(SELECT 1 FROM analytics_retries core WHERE core.repository=analytics_retries.repository AND core.number=analytics_retries.number AND core.operation='graphql_history' AND core.resolved_at IS NULL)` + read := func(attempted bool, n int) ([]int, error) { + predicate := "attempts=0" + if attempted { + predicate = "attempts>0" + } + rows, err := s.q().QueryContext(ctx, "SELECT number FROM analytics_retries WHERE "+eligible+" AND "+predicate+" ORDER BY next_attempt_at,number LIMIT ?", repository, at, n) + if err != nil { + return nil, err + } + defer rows.Close() + var out []int + for rows.Next() { + var number int + if err = rows.Scan(&number); err != nil { + return nil, err + } + out = append(out, number) + } + return out, rows.Err() + } + retries, err := read(true, max(1, limit/4)) + if err != nil { + return nil, err + } + fresh, err := read(false, limit-len(retries)) + if err != nil { + return nil, err + } + if len(fresh)+len(retries) < limit { + retries, err = read(true, limit-len(fresh)) + if err != nil { + return nil, err + } + } + return append(retries, fresh...), nil +} + +func (s *Store) ReviewStateParent(ctx context.Context, repository string, number int, providerRepositoryID, providerNodeID string) (Thread, error) { + var t Thread + var repoID, rawRepo string + err := s.q().QueryRowContext(ctx, `SELECT t.id,t.repo_id,t.github_id,t.kind,t.raw_json,r.github_repo_id,r.raw_json FROM threads t JOIN repositories r ON r.id=t.repo_id WHERE r.full_name=? COLLATE NOCASE AND t.number=?`, repository, number).Scan(&t.ID, &t.RepoID, &t.GitHubID, &t.Kind, &t.RawJSON, &repoID, &rawRepo) + if err != nil { + return t, err + } + if t.Kind != "pull_request" || repoID != providerRepositoryID { + return t, fmt.Errorf("review-state archived repository/PR identity mismatch") + } + var repo map[string]any + if err = json.Unmarshal([]byte(rawRepo), &repo); err != nil { + return t, err + } + if node, ok := repo["node_id"].(string); ok && node != "" && node != providerNodeID { + return t, fmt.Errorf("review-state repository node mismatch") + } + return t, nil +} + +func (s *Store) DueAnalyticsRetries(ctx context.Context, repository, at string, limit int, operations ...string) ([]int, error) { + operation := "" + if len(operations) > 0 { + operation = operations[0] + } + rows, err := s.q().QueryContext(ctx, `SELECT r.number FROM analytics_retries r WHERE r.repository=? AND r.resolved_at IS NULL AND r.number>0 AND r.next_attempt_at<=? AND (?='' OR r.operation=?) + AND (r.operation<>'review_state' OR NOT EXISTS(SELECT 1 FROM analytics_retries core WHERE core.repository=r.repository AND core.number=r.number AND core.operation='graphql_history' AND core.resolved_at IS NULL)) + GROUP BY r.number ORDER BY min(r.next_attempt_at),r.number LIMIT ?`, repository, at, operation, operation, limit) + if err != nil { + return nil, err + } + defer rows.Close() + var out []int + for rows.Next() { + var n int + if err = rows.Scan(&n); err != nil { + return nil, err + } + out = append(out, n) + } + return out, rows.Err() +} + +func (s *Store) AnalyticsOutstanding(ctx context.Context, repository string) (int, error) { + var n int + err := s.q().QueryRowContext(ctx, `SELECT count(*) FROM analytics_retries WHERE repository=? AND resolved_at IS NULL`, repository).Scan(&n) + return n, err +} + +func (s *Store) AnalyticsCoreOutstanding(ctx context.Context, repository string) (int, error) { + var n int + err := s.q().QueryRowContext(ctx, `SELECT count(*) FROM analytics_retries WHERE repository=? AND resolved_at IS NULL AND operation<>'review_state'`, repository).Scan(&n) + return n, err +} + +func (s *Store) AnalyticsItemQueued(ctx context.Context, repository string, number int, operations ...string) (bool, error) { + operation := "graphql_history" + if len(operations) > 0 { + operation = operations[0] + } + var n int + err := s.q().QueryRowContext(ctx, `SELECT count(*) FROM analytics_retries WHERE repository=? AND number=? AND operation=? AND resolved_at IS NULL`, repository, number, operation).Scan(&n) + return n > 0, err +} + +type ReviewStateRecovery struct { + Cursor int64 `json:"cursor"` + Ceiling int64 `json:"ceiling"` + Scanned int64 `json:"scanned"` + Queued int64 `json:"queued"` + Done bool `json:"done"` +} + +// SeedReviewStateRecovery walks at most limit primary-key rows per watch cycle. +// It queues only PRs with retained review threads (or unknown connection data), +// and never restarts historical discovery or invents missing resolution state. +func (s *Store) SeedReviewStateRecovery(ctx context.Context, repository string, limit int) (ReviewStateRecovery, error) { + key := "review_state_recovery:" + repository + value, err := s.AnalyticsState(ctx, key) + if err != nil { + return ReviewStateRecovery{}, err + } + var progress ReviewStateRecovery + if value != "" { + if err = json.Unmarshal([]byte(value), &progress); err != nil { + return progress, err + } + } else { + if err = s.q().QueryRowContext(ctx, "SELECT coalesce(max(id),0) FROM threads").Scan(&progress.Ceiling); err != nil { + return progress, err + } + } + if progress.Done { + return progress, nil + } + rows, err := s.q().QueryContext(ctx, `SELECT t.id,t.number,t.kind,r.full_name, + CASE WHEN json_valid(t.raw_json) THEN coalesce( + json_extract(t.raw_json,'$._gitcrawl_source')='graphql' AND + json_type(t.raw_json,'$._graphql.reviewThreads.totalCount')='integer' AND + json_extract(t.raw_json,'$._graphql.reviewThreads.totalCount')=0 AND + json_type(t.raw_json,'$._graphql.reviewThreads.nodes')='array' AND + json_array_length(t.raw_json,'$._graphql.reviewThreads.nodes')=0 AND + json_type(t.raw_json,'$._graphql.reviewThreads.pageInfo.hasNextPage')='false',0) ELSE 0 END, + t.observation_sequence,coalesce(t.last_pulled_at,''), + EXISTS(SELECT 1 FROM pull_request_review_thread_syncs x WHERE x.thread_id=t.id AND x.review_thread_ids_json IS NOT NULL) + FROM threads t JOIN repositories r ON r.id=t.repo_id WHERE t.id>? AND t.id<=? ORDER BY t.id LIMIT ?`, progress.Cursor, progress.Ceiling, limit) + if err != nil { + return progress, err + } + type item struct { + id int64 + number int + kind, repo string + empty bool + sequence int64 + pulled string + known bool + } + var items []item + for rows.Next() { + var v item + if err = rows.Scan(&v.id, &v.number, &v.kind, &v.repo, &v.empty, &v.sequence, &v.pulled, &v.known); err != nil { + rows.Close() + return progress, err + } + items = append(items, v) + } + err = rows.Err() + rows.Close() + if err != nil { + return progress, err + } + var committed ReviewStateRecovery + err = s.WithTx(ctx, func(tx *Store) error { + next := progress // A busy-transaction retry must not double-count progress. + at := time.Now().UTC().Format(time.RFC3339Nano) + for _, v := range items { + next.Cursor = v.id + next.Scanned++ + if !strings.EqualFold(v.repo, repository) || v.kind != "pull_request" || v.known { + continue + } + if _, e := time.Parse(time.RFC3339Nano, v.pulled); v.empty && e == nil { + // Materialize only an actual retained complete-empty observation. + // CAS checks prevent an older scan from overwriting a live refresh. + _, e = tx.q().ExecContext(ctx, `INSERT INTO pull_request_review_thread_syncs(thread_id,fetched_at,review_thread_ids_json) + SELECT id,last_pulled_at,'[]' FROM threads WHERE id=? AND observation_sequence=? AND last_pulled_at=? + ON CONFLICT(thread_id) DO UPDATE SET fetched_at=excluded.fetched_at,review_thread_ids_json=excluded.review_thread_ids_json + WHERE pull_request_review_thread_syncs.review_thread_ids_json IS NULL AND pull_request_review_thread_syncs.fetched_at<=excluded.fetched_at`, v.id, v.sequence, v.pulled) + if e != nil { + return e + } + var known bool + if e = tx.q().QueryRowContext(ctx, `SELECT EXISTS(SELECT 1 FROM pull_request_review_thread_syncs WHERE thread_id=? AND review_thread_ids_json IS NOT NULL)`, v.id).Scan(&known); e != nil { + return e + } + if known { + continue + } + } + r, e := tx.q().ExecContext(ctx, `INSERT OR IGNORE INTO analytics_retries(repository,number,operation,first_seen_at,last_seen_at,next_attempt_at) VALUES(?,?,'review_state',?,?,?)`, repository, v.number, at, at, at) + if e != nil { + return e + } + n, e := r.RowsAffected() + if e != nil { + return e + } + next.Queued += n + } + next.Done = len(items) < limit || next.Cursor >= next.Ceiling + encoded, _ := json.Marshal(next) + if e := tx.SetAnalyticsState(ctx, key, string(encoded)); e != nil { + return e + } + committed = next + return nil + }) + if err == nil { + progress = committed + } + return progress, err +} + +// Core completeness concerns issue/PR/comment traversal. Review-state enrichment +// is deliberately independent so existing response/contributor KPIs remain usable. +func (s *Store) SetAnalyticsCoverageComplete(ctx context.Context, repository string, complete bool) error { + _, err := s.q().ExecContext(ctx, "UPDATE analytics_coverage SET complete=? WHERE repository=?", boolInt(complete), repository) + return err +} + +func (s *Store) SaveReviewStateCoverage(ctx context.Context, repository string, progress ReviewStateRecovery) error { + var pending int + if err := s.q().QueryRowContext(ctx, `SELECT count(*) FROM analytics_retries WHERE repository=? AND operation='review_state' AND resolved_at IS NULL`, repository).Scan(&pending); err != nil { + return err + } + excluded, err := s.OwnerExcludedCount(ctx, repository, true) + if err != nil { + return err + } + _, err = s.q().ExecContext(ctx, `INSERT INTO analytics_review_state_coverage(repository,cursor,ceiling,scanned,queued,pending_items,scan_complete,complete,observed_at,owner_excluded_items) VALUES(?,?,?,?,?,?,?,?,?,?) ON CONFLICT(repository) DO UPDATE SET cursor=excluded.cursor,ceiling=excluded.ceiling,scanned=excluded.scanned,queued=excluded.queued,pending_items=excluded.pending_items,scan_complete=excluded.scan_complete,complete=excluded.complete,observed_at=excluded.observed_at,owner_excluded_items=excluded.owner_excluded_items`, repository, progress.Cursor, progress.Ceiling, progress.Scanned, progress.Queued, pending, boolInt(progress.Done), boolInt(progress.Done && pending == 0 && excluded == 0), time.Now().UTC().Format(time.RFC3339Nano), excluded) + return err +} + +func (s *Store) AnalyticsIntegrityStatus(ctx context.Context, repository string) (map[string]any, error) { + out := map[string]any{"repository": repository, "checked_at": time.Now().UTC().Format(time.RFC3339Nano)} + if s.hasTable(ctx, "thread_exclusions") { + excluded, err := s.OwnerExcludedCount(ctx, repository, false) + if err != nil { + return nil, err + } + out["owner_excluded_items"] = excluded + } + var through, observed string + var issues, prs, complete int + coverageErr := s.q().QueryRowContext(ctx, "SELECT through,issues,pull_requests,complete,observed_at FROM analytics_coverage WHERE repository=?", repository).Scan(&through, &issues, &prs, &complete, &observed) + if coverageErr == sql.ErrNoRows { + out["coverage"] = map[string]any{"state": "not_started", "through": nil, "issues": nil, "pull_requests": nil, "complete": false, "observed_at": nil} + } else if coverageErr != nil { + return nil, coverageErr + } else { + out["coverage"] = map[string]any{"state": "observed", "through": through, "issues": issues, "pull_requests": prs, "complete": complete == 1, "observed_at": observed} + } + n, err := s.AnalyticsOutstanding(ctx, repository) + if err != nil { + return nil, err + } + out["unresolved_retries"] = n + progress, err := s.AnalyticsState(ctx, "review_state_recovery:"+repository) + if err != nil { + return nil, err + } + if progress != "" { + out["review_state_recovery"] = json.RawMessage(progress) + var recovery ReviewStateRecovery + if err = json.Unmarshal([]byte(progress), &recovery); err != nil { + return nil, err + } + out["review_state_scan_complete"] = recovery.Done + } else { + out["review_state_recovery"] = nil + out["review_state_scan_complete"] = false + } + corePending, err := s.AnalyticsCoreOutstanding(ctx, repository) + if err != nil { + return nil, err + } + out["core_unresolved_retries"] = corePending + out["coverage"].(map[string]any)["complete"] = complete == 1 && corePending == 0 + var reviewPending, reviewComplete int + if err = s.q().QueryRowContext(ctx, `SELECT pending_items,complete FROM analytics_review_state_coverage WHERE repository=?`, repository).Scan(&reviewPending, &reviewComplete); err == nil { + out["review_state_coverage"] = map[string]any{"pending_items": reviewPending, "complete": reviewComplete == 1} + } else if err == sql.ErrNoRows { + out["review_state_coverage"] = nil + } else { + return nil, err + } + rows, err := s.q().QueryContext(ctx, `SELECT id,number,operation,started_at,finished_at,status,coalesce(error_class,'') FROM analytics_fetch_attempts WHERE repository=? ORDER BY id DESC LIMIT 10`, repository) + if err != nil { + return nil, err + } + defer rows.Close() + items := []map[string]any{} + for rows.Next() { + var id, number int64 + var operation, start, finish, status, class string + if err = rows.Scan(&id, &number, &operation, &start, &finish, &status, &class); err != nil { + return nil, err + } + items = append(items, map[string]any{"id": id, "number": number, "operation": operation, "started_at": start, "finished_at": finish, "status": status, "error_class": class}) + } + out["latest_attempts"] = items + return out, rows.Err() +} diff --git a/internal/store/analytics_integrity_test.go b/internal/store/analytics_integrity_test.go new file mode 100644 index 00000000..19d440ab --- /dev/null +++ b/internal/store/analytics_integrity_test.go @@ -0,0 +1,404 @@ +package store + +import ( + "context" + "encoding/json" + "path/filepath" + "testing" +) + +func TestAnalyticsFailureRecoveryRetainsReceiptsAndFairRetryTimes(t *testing.T) { + ctx := context.Background() + path := filepath.Join(t.TempDir(), "archive.db") + s, err := Open(ctx, path) + if err != nil { + t.Fatal(err) + } + attempt := AnalyticsAttempt{Repository: "fixture/repo", Number: 7, Operation: "graphql_history", StartedAt: "2026-01-01T00:00:00Z", FinishedAt: "2026-01-01T00:00:01Z", Status: "failed", ErrorClass: "validation", ErrorText: "incomplete connection", Evidence: json.RawMessage(`{"totalCount":2,"received":1}`)} + if err = s.RecordAnalyticsAttempt(ctx, attempt); err != nil { + t.Fatal(err) + } + if due, err := s.DueAnalyticsRetries(ctx, "fixture/repo", "2026-01-01T00:00:10Z", 5); err != nil || len(due) != 0 { + t.Fatalf("early retry %v %v", due, err) + } + s.Close() + s, err = Open(ctx, path) + if err != nil { + t.Fatal(err) + } + defer s.Close() + if due, err := s.DueAnalyticsRetries(ctx, "fixture/repo", "2026-01-01T00:00:40Z", 5); err != nil || len(due) != 1 || due[0] != 7 { + t.Fatalf("restart lost failure %v %v", due, err) + } + attempt.Number = 8 + attempt.FinishedAt = "2026-01-01T00:00:02Z" + if err = s.RecordAnalyticsAttempt(ctx, attempt); err != nil { + t.Fatal(err) + } + attempt.Number = 7 + attempt.FinishedAt = "2026-01-01T00:00:40Z" + if err = s.RecordAnalyticsAttempt(ctx, attempt); err != nil { + t.Fatal(err) + } + if due, err := s.DueAnalyticsRetries(ctx, "fixture/repo", "2026-01-01T00:00:41Z", 1); err != nil || len(due) != 1 || due[0] != 8 { + t.Fatalf("poison item starved peer %v %v", due, err) + } + attempt.Status = "success" + attempt.FinishedAt = "2026-01-01T00:02:00Z" + attempt.ErrorClass = "" + attempt.ErrorText = "" + if err = s.RecordAnalyticsAttempt(ctx, attempt); err != nil { + t.Fatal(err) + } + var receipts, resolved int + s.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts").Scan(&receipts) + s.DB().QueryRow("SELECT count(*) FROM analytics_retries WHERE number=7 AND resolved_at IS NOT NULL").Scan(&resolved) + if receipts != 4 || resolved != 1 { + t.Fatalf("history lost receipts=%d resolved=%d", receipts, resolved) + } +} + +func TestReviewStateRecoveryIsBoundedResumableAndPreservesHistory(t *testing.T) { + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + _, err = s.DB().Exec(`INSERT INTO repositories(id,owner,name,full_name,github_repo_id,raw_json,updated_at) VALUES(1,'fixture','repo','Fixture/Repo',1,'{}','2026-01-01'); +INSERT INTO threads(id,repo_id,github_id,number,kind,state,title,html_url,labels_json,assignees_json,raw_json,content_hash,updated_at,last_pulled_at) VALUES +(1,1,'P1',1,'pull_request','open','','','[]','[]','{"_graphql":{"reviewThreads":{"totalCount":1}}}','h1','2026-01-01','2026-01-01T00:00:00Z'), +(2,1,'P2',2,'pull_request','open','','','[]','[]','{"_gitcrawl_source":"graphql","_graphql":{"reviewThreads":{"totalCount":0,"nodes":[],"pageInfo":{"hasNextPage":false}}}}','h2','2026-01-01','2026-01-01T00:00:00Z');`) + if err != nil { + t.Fatal(err) + } + p, err := s.SeedReviewStateRecovery(ctx, "fixture/repo", 1) + if err != nil || p.Done || p.Queued != 1 || p.Cursor != 1 { + t.Fatalf("first step %+v %v", p, err) + } + p, err = s.SeedReviewStateRecovery(ctx, "fixture/repo", 1) + if err != nil || !p.Done || p.Queued != 1 || p.Scanned != 2 { + t.Fatalf("resume %+v %v", p, err) + } + var emptyIDs, observed string + if err = s.DB().QueryRow("SELECT review_thread_ids_json,fetched_at FROM pull_request_review_thread_syncs WHERE thread_id=2").Scan(&emptyIDs, &observed); err != nil || emptyIDs != "[]" || observed != "2026-01-01T00:00:00Z" { + t.Fatalf("known empty evidence not materialized: %s %s %v", emptyIDs, observed, err) + } + threads := []PullRequestReviewThread{{ReviewThreadID: "T1", ThreadID: 1, IsResolved: true, IsOutdated: false, RawJSON: `{"id":"T1","isResolved":true}`, CommentsJSON: `[]`, FetchedAt: "2026-01-02T00:00:00Z"}} + if err = s.UpsertPullRequestReviewThreads(ctx, 1, "2026-01-02T00:00:00Z", threads); err != nil { + t.Fatal(err) + } + threads[0].IsResolved = false + threads[0].RawJSON = `{"id":"T1","isResolved":false}` + if err = s.UpsertPullRequestReviewThreads(ctx, 1, "2026-01-03T00:00:00Z", threads); err != nil { + t.Fatal(err) + } + if err = s.UpsertPullRequestReviewThreads(ctx, 1, "2026-01-04T00:00:00Z", nil); err != nil { + t.Fatal(err) + } + var revisions, retained int + var membership string + s.DB().QueryRow("SELECT count(*) FROM pull_request_review_thread_revisions WHERE thread_id=1").Scan(&revisions) + s.DB().QueryRow("SELECT count(*) FROM pull_request_review_threads WHERE thread_id=1 AND deleted_at IS NULL").Scan(&retained) + s.DB().QueryRow("SELECT review_thread_ids_json FROM pull_request_review_thread_syncs WHERE thread_id=1").Scan(&membership) + if revisions != 2 || retained != 1 || membership != "[]" { + t.Fatalf("absence became deletion or history lost: %d %d %s", revisions, retained, membership) + } + for _, invalid := range [][]PullRequestReviewThread{{{ReviewThreadID: ""}}, {{ReviewThreadID: "duplicate"}, {ReviewThreadID: "duplicate"}}} { + if err = s.UpsertPullRequestReviewThreads(ctx, 1, "2026-01-05T00:00:00Z", invalid); err == nil { + t.Fatal("invalid membership was certified") + } + } + s.DB().QueryRow("SELECT review_thread_ids_json FROM pull_request_review_thread_syncs WHERE thread_id=1").Scan(&membership) + if membership != "[]" { + t.Fatal("rejected input changed prior membership") + } + if err = s.UpsertPullRequestReviewThreads(ctx, 1, "2026-01-01T00:00:00Z", threads); err != nil { + t.Fatal(err) + } + s.DB().QueryRow("SELECT review_thread_ids_json,fetched_at FROM pull_request_review_thread_syncs WHERE thread_id=1").Scan(&membership, &observed) + if membership != "[]" || observed != "2026-01-04T00:00:00Z" { + t.Fatalf("older observation overwrote newer membership: %s %s", membership, observed) + } +} + +func TestAnalyticsV14MigrationPreservesReceiptsAndCoverageWatermark(t *testing.T) { + ctx := context.Background() + path := filepath.Join(t.TempDir(), "archive.db") + s, err := Open(ctx, path) + if err != nil { + t.Fatal(err) + } + if err = s.SaveAnalyticsCoverage(ctx, "fixture/repo", "2026-01-01T00:00:00Z", 1, 2); err != nil { + t.Fatal(err) + } + if err = s.SetAnalyticsState(ctx, "updates:fixture/repo", `{"kind":1,"cursor":"provider-cursor"}`); err != nil { + t.Fatal(err) + } + if _, err = s.DB().Exec(`DROP TABLE analytics_fetch_attempts; DROP TABLE analytics_retries; ALTER TABLE pull_request_review_thread_syncs DROP COLUMN review_thread_ids_json; PRAGMA user_version=14`); err != nil { + t.Fatal(err) + } + if err = s.markObservationSchemaConverged(ctx); err != nil { + t.Fatal(err) + } + s.Close() + s, err = Open(ctx, path) + if err != nil { + t.Fatal(err) + } + defer s.Close() + var version, complete int + var through string + s.DB().QueryRow("PRAGMA user_version").Scan(&version) + s.DB().QueryRow("SELECT through,complete FROM analytics_coverage WHERE repository='fixture/repo'").Scan(&through, &complete) + cp, err := s.AnalyticsState(ctx, "updates:fixture/repo") + if err != nil || version != schemaVersion || through != "2026-01-01T00:00:00Z" || complete != 1 || cp != `{"kind":1,"cursor":"provider-cursor"}` { + t.Fatalf("migration changed evidence: %d %s %d %s %v", version, through, complete, cp, err) + } +} + +func TestReviewStateFailuresDoNotInvalidateVerifiedCoreCoverage(t *testing.T) { + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + if err = s.SaveAnalyticsCoverage(ctx, "fixture/repo", "2026-01-01T00:00:00Z", 1, 2); err != nil { + t.Fatal(err) + } + a := AnalyticsAttempt{Repository: "fixture/repo", Number: 2, Operation: "review_state", StartedAt: "2026-01-02T00:00:00Z", FinishedAt: "2026-01-02T00:00:01Z", Status: "failed", ErrorClass: "validation", Evidence: json.RawMessage(`{}`)} + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + if err = s.SaveReviewStateCoverage(ctx, "fixture/repo", ReviewStateRecovery{Done: true, Scanned: 2, Ceiling: 2, Cursor: 2, Queued: 1}); err != nil { + t.Fatal(err) + } + var core, review int + s.DB().QueryRow("SELECT complete FROM analytics_coverage WHERE repository='fixture/repo'").Scan(&core) + s.DB().QueryRow("SELECT complete FROM analytics_review_state_coverage WHERE repository='fixture/repo'").Scan(&review) + if core != 1 || review != 0 { + t.Fatalf("enrichment invalidated core: core=%d review=%d", core, review) + } + if n, err := s.AnalyticsCoreOutstanding(ctx, "fixture/repo"); err != nil || n != 0 { + t.Fatalf("wrong core queue %d %v", n, err) + } + a.Operation = "graphql_history" + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + s.DB().QueryRow("SELECT complete FROM analytics_coverage WHERE repository='fixture/repo'").Scan(&core) + if core != 0 { + t.Fatal("core failure left core coverage complete") + } +} + +func TestAnalyticsStatusSelectsRepositoryBeforeLimit(t *testing.T) { + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + s.SaveAnalyticsCoverage(ctx, "fixture/repo", "2026-01-01T00:00:00Z", 1, 2) + a := AnalyticsAttempt{Repository: "fixture/repo", Number: 1, Operation: "graphql_history", StartedAt: "2026-01-01T00:00:00Z", FinishedAt: "2026-01-01T00:00:01Z", Status: "success", Evidence: json.RawMessage(`{}`)} + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + a.Repository = "fixture/other" + for i := 0; i < 60; i++ { + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + } + status, err := s.AnalyticsIntegrityStatus(ctx, "fixture/repo") + if err != nil { + t.Fatal(err) + } + if len(status["latest_attempts"].([]map[string]any)) != 1 { + t.Fatal("another repository hid scoped history") + } +} + +func TestAnalyticsStatusBeforeCollectionIsUnknown(t *testing.T) { + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + status, err := s.AnalyticsIntegrityStatus(ctx, "fixture/repo") + if err != nil { + t.Fatal(err) + } + coverage := status["coverage"].(map[string]any) + if coverage["state"] != "not_started" || coverage["complete"] != false || coverage["issues"] != nil { + t.Fatalf("invented coverage: %+v", coverage) + } + var n int + s.DB().QueryRow("SELECT count(*) FROM analytics_coverage").Scan(&n) + if n != 0 { + t.Fatal("read-only status wrote a baseline") + } +} + +func TestAnalyticsV14WithoutExtensionTablesCanUpgrade(t *testing.T) { + ctx := context.Background() + path := filepath.Join(t.TempDir(), "archive.db") + s, err := Open(ctx, path) + if err != nil { + t.Fatal(err) + } + if _, err = s.DB().Exec(`DROP TABLE actor_profiles; DROP TABLE analytics_collection_state; PRAGMA user_version=14`); err != nil { + t.Fatal(err) + } + if err = s.markObservationSchemaConverged(ctx); err != nil { + t.Fatal(err) + } + s.Close() + s, err = Open(ctx, path) + if err != nil { + t.Fatal(err) + } + defer s.Close() + if _, err = s.AnalyticsState(ctx, "missing"); err != nil { + t.Fatal(err) + } + if _, err = s.AnalyticsProfileNodes(ctx, 1); err != nil { + t.Fatal(err) + } +} + +func TestAnalyticsCoreRetryOwnsBackoffWhenReviewRetryAlsoExists(t *testing.T) { + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + a := AnalyticsAttempt{Repository: "fixture/repo", Number: 1, Operation: "graphql_history", StartedAt: "2026-01-01T00:00:00Z", FinishedAt: "2026-01-01T00:00:01Z", Status: "failed", Evidence: json.RawMessage(`{}`)} + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + a.Operation = "review_state" + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + if due, err := s.DueAnalyticsRetries(ctx, a.Repository, "2026-01-01T00:00:40Z", 8, "review_state"); err != nil || len(due) != 0 { + t.Fatalf("review bypassed due core scheduler: %v %v", due, err) + } + a.Operation = "graphql_history" + a.FinishedAt = "2026-01-01T00:00:50Z" + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + if due, err := s.DueAnalyticsRetries(ctx, a.Repository, "2026-01-01T00:01:00Z", 8, "review_state"); err != nil || len(due) != 0 { + t.Fatalf("review bypassed core backoff: %v %v", due, err) + } +} + +func TestAnalyticsRecoveryRequiresAnAcceptedMembershipObservation(t *testing.T) { + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + a := AnalyticsAttempt{Repository: "fixture/repo", Number: 1, Operation: "review_state", StartedAt: "2026-01-01T00:00:00Z", FinishedAt: "2026-01-01T00:00:01Z", Status: "success", Evidence: json.RawMessage(`{"threads_skipped_stale":1}`)} + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + var status, class string + s.DB().QueryRow("SELECT status,error_class FROM analytics_fetch_attempts ORDER BY id DESC LIMIT 1").Scan(&status, &class) + if status != "failed" || class != "unapplied_review_state" { + t.Fatal("fetch success certified missing membership") + } + repo, err := s.UpsertRepository(ctx, Repository{Owner: "fixture", Name: "repo", FullName: "fixture/repo", RawJSON: "{}", UpdatedAt: "2026-01-02T00:00:00Z"}) + if err != nil { + t.Fatal(err) + } + tid, err := s.UpsertThread(ctx, Thread{RepoID: repo, GitHubID: "P1", Number: 1, Kind: "pull_request", State: "open", Title: "fixture", HTMLURL: "https://github.com/fixture/repo/pull/1", LabelsJSON: "[]", AssigneesJSON: "[]", RawJSON: "{}", ContentHash: "h", UpdatedAt: "2026-01-02T00:00:00Z"}) + if err != nil { + t.Fatal(err) + } + if err = s.UpsertPullRequestReviewThreads(ctx, tid, "2026-01-02T00:00:00Z", nil); err != nil { + t.Fatal(err) + } + a.FinishedAt = "2026-01-02T00:00:01Z" + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + if n, err := s.AnalyticsOutstanding(ctx, a.Repository); err != nil || n != 0 { + t.Fatalf("proven empty membership did not reconcile: %d %v", n, err) + } +} + +func TestReviewRetryFairShareAndExplicitUnavailableBackoff(t *testing.T) { + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + for n := 1; n <= 100; n++ { + if _, err = s.DB().Exec("insert into analytics_retries(repository,number,operation,first_seen_at,last_seen_at,next_attempt_at) values('fixture/repo',?,'review_state','2000-01-01T00:00:00Z','2000-01-01T00:00:00Z','2000-01-01T00:00:00Z')", n); err != nil { + t.Fatal(err) + } + } + a := AnalyticsAttempt{Repository: "fixture/repo", Number: 999, Operation: "review_state", StartedAt: "2026-01-01T00:00:00Z", FinishedAt: "2026-01-01T00:00:01Z", Status: "failed", ErrorClass: "partial_response", Evidence: json.RawMessage(`{"graphql_errors":{"items":[{"type":"NOT_FOUND"}]}}`)} + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + due, err := s.DueReviewStateWork(ctx, "fixture/repo", "2026-01-01T00:02:00Z", 16) + if err != nil { + t.Fatal(err) + } + for _, n := range due { + if n == 999 { + t.Fatal("unavailable backoff ignored") + } + } + due, err = s.DueReviewStateWork(ctx, "fixture/repo", "2026-01-01T00:16:00Z", 16) + if err != nil || len(due) != 16 || due[0] != 999 { + t.Fatalf("failed retry starved by bulk: %v %v", due, err) + } + a.Operation = "graphql_history" + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + due, err = s.DueReviewStateWork(ctx, "fixture/repo", "2026-01-01T00:16:00Z", 16) + if err != nil { + t.Fatal(err) + } + for _, n := range due { + if n == 999 { + t.Fatal("review bypassed core obligation") + } + } + repoID, err := s.UpsertRepository(ctx, Repository{Owner: "fixture", Name: "repo", FullName: "fixture/repo", RawJSON: "{}", UpdatedAt: a.FinishedAt}) + if err != nil { + t.Fatal(err) + } + tid, err := s.UpsertThread(ctx, Thread{RepoID: repoID, GitHubID: "P999", Number: 999, Kind: "pull_request", State: "open", Title: "fixture", HTMLURL: "https://github.com/fixture/repo/pull/999", LabelsJSON: "[]", AssigneesJSON: "[]", RawJSON: "{}", ContentHash: "h", UpdatedAt: a.FinishedAt}) + if err != nil { + t.Fatal(err) + } + if err = s.UpsertPullRequestReviewThreads(ctx, tid, a.FinishedAt, nil); err != nil { + t.Fatal(err) + } + a.Operation = "review_state" + a.Status = "success" + if err = s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + var status string + s.DB().QueryRow("select status from analytics_fetch_attempts order by id desc limit 1").Scan(&status) + if status != "success" { + t.Fatal("review success not proven", status) + } + var unresolved int + s.DB().QueryRow("select count(*) from analytics_retries where operation='graphql_history' and resolved_at is null").Scan(&unresolved) + if unresolved != 1 { + t.Fatal("review-only success cleared core failure") + } +} diff --git a/internal/store/analytics_recovery_contract_test.go b/internal/store/analytics_recovery_contract_test.go new file mode 100644 index 00000000..a564e9e6 --- /dev/null +++ b/internal/store/analytics_recovery_contract_test.go @@ -0,0 +1,145 @@ +package store + +import ( + "context" + "database/sql" + "encoding/json" + "errors" + "path/filepath" + "testing" +) + +func recoveryContractStore(t *testing.T) (*Store, int64) { + t.Helper() + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { s.Close() }) + repoID, err := s.UpsertRepository(ctx, Repository{Owner: "fixture", Name: "repo", FullName: "fixture/repo", GitHubRepoID: "101", RawJSON: `{"node_id":"R1"}`, UpdatedAt: "2026-01-01T00:00:00Z"}) + if err != nil { + t.Fatal(err) + } + id, err := s.UpsertThread(ctx, Thread{RepoID: repoID, GitHubID: "201", Number: 7, Kind: "pull_request", State: "open", Title: "retained", HTMLURL: "https://github.com/fixture/repo/pull/7", LabelsJSON: "[]", AssigneesJSON: "[]", RawJSON: `{"node_id":"P7"}`, ContentHash: "retained-hash", UpdatedAt: "2026-01-01T00:00:00Z"}) + if err != nil { + t.Fatal(err) + } + return s, id +} + +func TestReviewStateParentBindsIdentityInOwningTransaction(t *testing.T) { + ctx := context.Background() + s, id := recoveryContractStore(t) + parent, err := s.ReviewStateParent(ctx, "FIXTURE/REPO", 7, "101", "R1") + if err != nil || parent.ID != id || parent.GitHubID != "201" || parent.RawJSON != `{"node_id":"P7"}` { + t.Fatalf("parent=%+v err=%v", parent, err) + } + for _, tc := range []struct { + repo string + number int + database, node string + }{ + {"fixture/other", 7, "101", "R1"}, {"fixture/repo", 8, "101", "R1"}, {"fixture/repo", 7, "102", "R1"}, {"fixture/repo", 7, "101", "R2"}, + } { + if _, err := s.ReviewStateParent(ctx, tc.repo, tc.number, tc.database, tc.node); err == nil { + t.Fatalf("mismatched identity accepted: %+v", tc) + } + } + rolledBack := errors.New("fixture rollback") + err = s.WithTx(ctx, func(tx *Store) error { + if _, err := tx.q().ExecContext(ctx, `UPDATE repositories SET raw_json='{"node_id":"R-new"}' WHERE id=?`, parent.RepoID); err != nil { + return err + } + if _, err := tx.ReviewStateParent(ctx, "fixture/repo", 7, "101", "R1"); err == nil { + t.Fatal("read stale identity outside owning transaction") + } + got, err := tx.ReviewStateParent(ctx, "fixture/repo", 7, "101", "R-new") + if err != nil || got.ID != id { + t.Fatalf("transaction-local binding failed: %+v %v", got, err) + } + return rolledBack + }) + if !errors.Is(err, rolledBack) { + t.Fatal(err) + } + if _, err = s.ReviewStateParent(ctx, "fixture/repo", 7, "101", "R1"); err != nil { + t.Fatal("identity mutation escaped rollback", err) + } + if _, err = s.DB().ExecContext(ctx, "UPDATE threads SET kind='issue' WHERE id=?", id); err != nil { + t.Fatal(err) + } + if _, err = s.ReviewStateParent(ctx, "fixture/repo", 7, "101", "R1"); err == nil { + t.Fatal("issue accepted as review-state parent") + } +} + +func TestReviewRecoveryCannotDischargeCoreRetryOrCertifyCoreCoverage(t *testing.T) { + ctx := context.Background() + s, id := recoveryContractStore(t) + through := "2026-01-01T00:00:00Z" + if err := s.SaveAnalyticsCoverage(ctx, "fixture/repo", through, 2, 1); err != nil { + t.Fatal(err) + } + a := AnalyticsAttempt{Repository: "fixture/repo", Number: 7, Operation: "review_state", StartedAt: through, FinishedAt: "2026-01-01T00:00:01Z", Status: "failed", ErrorClass: "validation", Evidence: json.RawMessage(`{}`)} + record := func() { + t.Helper() + if err := s.RecordAnalyticsAttempt(ctx, a); err != nil { + t.Fatal(err) + } + } + queued := func(want bool, operations ...string) { + t.Helper() + got, err := s.AnalyticsItemQueued(ctx, a.Repository, a.Number, operations...) + if err != nil || got != want { + t.Fatalf("queued(%v)=%v want%v err%v", operations, got, want, err) + } + } + coverage := func(want int) { + t.Helper() + var complete int + var watermark string + if err := s.DB().QueryRowContext(ctx, "SELECT complete,through FROM analytics_coverage WHERE repository=?", a.Repository).Scan(&complete, &watermark); err != nil || complete != want || watermark != through { + t.Fatalf("coverage=%d through=%s err=%v", complete, watermark, err) + } + } + record() + queued(false) + queued(true, "review_state") + coverage(1) + a.Operation = "graphql_history" + record() + queued(true) + coverage(0) + if err := s.UpsertPullRequestReviewThreads(ctx, id, a.FinishedAt, nil); err != nil { + t.Fatal(err) + } + a.Operation = "review_state" + a.Status = "success" + record() + queued(false, "review_state") + queued(true) + coverage(0) + a.Operation = "graphql_history" + record() + queued(false) + coverage(0) + // A successful item is not itself a completed traversal watermark. + if err := s.SetAnalyticsCoverageComplete(ctx, a.Repository, true); err != nil { + t.Fatal(err) + } + coverage(1) + a.Status = "failed" + a.FinishedAt = "2026-01-01T00:01:00Z" + record() + queued(true) + coverage(0) + var receipts int + if err := s.DB().QueryRowContext(ctx, "SELECT count(*) FROM analytics_fetch_attempts WHERE repository=?", a.Repository).Scan(&receipts); err != nil || receipts != 5 { + t.Fatalf("lost recovery history: %d %v", receipts, err) + } + var resolved sql.NullString + if err := s.DB().QueryRowContext(ctx, "SELECT resolved_at FROM analytics_retries WHERE repository=? AND operation='review_state'", a.Repository).Scan(&resolved); err != nil || !resolved.Valid { + t.Fatalf("independent review resolution lost: %v %v", resolved, err) + } +} diff --git a/internal/store/analytics_source.go b/internal/store/analytics_source.go new file mode 100644 index 00000000..0a75667b --- /dev/null +++ b/internal/store/analytics_source.go @@ -0,0 +1,341 @@ +package store + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "strings" + "time" +) + +// Publication describes provider-authored timestamps, not capture/repair time. +type Publication struct { + Submitted string `json:"submitted_at_gh,omitempty"` + Published string `json:"publication_at_gh,omitempty"` +} + +func ProviderTime(value any) string { + s, ok := value.(string) + if !ok || strings.TrimSpace(s) != s { + return "" + } + t, err := time.Parse(time.RFC3339Nano, s) + if err != nil { + return "" + } + return t.UTC().Format(time.RFC3339Nano) +} +func CommentPublication(kind, raw string) Publication { + var p map[string]any + if json.Unmarshal([]byte(raw), &p) != nil { + return Publication{} + } + g, _ := p["_graphql"].(map[string]any) + pick := func(values ...any) string { + for _, v := range values { + if s := ProviderTime(v); s != "" { + return s + } + } + return "" + } + if kind == "pull_review" { + if strings.EqualFold(fmt.Sprint(p["state"]), "pending") { + return Publication{} + } + at := pick(p["submitted_at"], g["submittedAt"]) + return Publication{Submitted: at, Published: at} + } + return Publication{Published: pick(g["publishedAt"], p["published_at"], p["created_at"], g["createdAt"])} +} +func (s *Store) ensureAnalyticsSourceSchema(ctx context.Context) error { + for _, table := range []string{"comments", "comment_revisions"} { + for _, col := range []string{"submitted_at_gh", "publication_at_gh"} { + if err := s.ensureColumn(ctx, table, col, "text"); err != nil { + return err + } + } + } + _, err := s.q().ExecContext(ctx, `CREATE TABLE IF NOT EXISTS actor_identity_evidence( + node_id TEXT PRIMARY KEY,actor_node_id TEXT,login TEXT,actor_type TEXT,observed_at TEXT NOT NULL,raw_json TEXT NOT NULL); + CREATE TABLE IF NOT EXISTS actor_profiles(node_id TEXT PRIMARY KEY,login TEXT NOT NULL,actor_type TEXT NOT NULL,name TEXT,bio TEXT,url TEXT,created_at TEXT,observed_at TEXT NOT NULL,raw_json TEXT NOT NULL); + CREATE TABLE IF NOT EXISTS analytics_coverage(repository TEXT PRIMARY KEY,through TEXT NOT NULL,issues INTEGER,pull_requests INTEGER,complete INTEGER NOT NULL,observed_at TEXT NOT NULL); + CREATE TABLE IF NOT EXISTS analytics_repair_receipts(name TEXT PRIMARY KEY,cursor INTEGER NOT NULL DEFAULT 0,updated_at TEXT NOT NULL); + CREATE TABLE IF NOT EXISTS analytics_pending_nodes(node_id TEXT NOT NULL,kind TEXT NOT NULL,PRIMARY KEY(kind,node_id)); + CREATE TABLE IF NOT EXISTS analytics_collection_state(name TEXT PRIMARY KEY,value TEXT NOT NULL,updated_at TEXT NOT NULL);`) + return err +} +func (s *Store) queueAnalyticsActor(ctx context.Context, raw string) error { + var payload struct { + NodeID string `json:"node_id"` + User struct { + NodeID string `json:"node_id"` + } `json:"user"` + } + if json.Unmarshal([]byte(raw), &payload) != nil { + return nil + } + id, kind := payload.User.NodeID, "profile" + if id == "" { + id, kind = payload.NodeID, "identity" + } + if id == "" { + return nil + } + if excluded, err := s.NodeExcluded(ctx, id); err != nil { + return err + } else if excluded { + return nil + } + _, err := s.q().ExecContext(ctx, "INSERT OR IGNORE INTO analytics_pending_nodes(node_id,kind) VALUES(?,?)", id, kind) + return err +} +func (s *Store) upsertCommentPublication(ctx context.Context, id int64, kind, raw string) error { + p := CommentPublication(kind, raw) + if err := s.queueAnalyticsActor(ctx, raw); err != nil { + return err + } + _, err := s.q().ExecContext(ctx, "UPDATE comments SET submitted_at_gh=?,publication_at_gh=? WHERE id=?", nullString(p.Submitted), nullString(p.Published), id) + return err +} + +type PublicationRepair struct { + Scanned int `json:"scanned"` + Changed int `json:"changed"` + WouldChange int `json:"would_change"` + Unknown int `json:"unknown"` +} + +// RepairPublication uses retained native payloads. Raw JSON, IDs and original +// recorded_at values are never rewritten, and no synthetic revision is appended. +func (s *Store) RepairPublication(ctx context.Context, apply bool) (PublicationRepair, error) { + result := PublicationRepair{} + pending := false + if apply { + value, e := s.AnalyticsState(ctx, "publication_repair_in_progress") + if e != nil { + return result, e + } + pending = value == "1" + if e = s.SetAnalyticsState(ctx, "publication_repair_in_progress", "1"); e != nil { + return result, e + } + } + for _, table := range []string{"comments", "comment_revisions"} { + var cursor int64 + var hasFields int + if err := s.q().QueryRowContext(ctx, "SELECT count(*) FROM pragma_table_info(?) WHERE name IN('submitted_at_gh','publication_at_gh')", table).Scan(&hasFields); err != nil { + return result, err + } + for { + fields := ",NULL,NULL" + if hasFields == 2 { + fields = ",c.submitted_at_gh,c.publication_at_gh" + } + query := "SELECT c.id,c.comment_type,c.raw_json" + fields + " FROM comments c WHERE c.id>? ORDER BY c.id LIMIT 1000" + if table == "comment_revisions" { + fields = ",NULL,NULL" + if hasFields == 2 { + fields = ",r.submitted_at_gh,r.publication_at_gh" + } + query = "SELECT r.id,c.comment_type,r.raw_json" + fields + " FROM comment_revisions r JOIN comments c ON c.id=r.comment_id WHERE r.id>? ORDER BY r.id LIMIT 1000" + } + rows, err := s.q().QueryContext(ctx, query, cursor) + if err != nil { + return result, err + } + type item struct { + id int64 + kind, raw string + submitted, published sql.NullString + } + var batch []item + for rows.Next() { + var r item + if err := rows.Scan(&r.id, &r.kind, &r.raw, &r.submitted, &r.published); err != nil { + rows.Close() + return result, err + } + batch = append(batch, r) + } + err = rows.Err() + rows.Close() + if err != nil { + return result, err + } + if len(batch) == 0 { + break + } + if apply { + err = s.WithTx(ctx, func(tx *Store) error { + for _, r := range batch { + p := CommentPublication(r.kind, r.raw) + if p.Submitted != r.submitted.String || p.Published != r.published.String { + result.WouldChange++ + } + result.Scanned++ + if r.kind == "pull_review" && p.Submitted == "" { + result.Unknown++ + } + res, e := tx.q().ExecContext(ctx, "UPDATE "+table+" SET submitted_at_gh=?,publication_at_gh=? WHERE id=? AND raw_json=? AND (submitted_at_gh IS NOT ? OR publication_at_gh IS NOT ?)", nullString(p.Submitted), nullString(p.Published), r.id, r.raw, nullString(p.Submitted), nullString(p.Published)) + if e != nil { + return e + } + n, e := res.RowsAffected() + if e != nil { + return e + } + result.Changed += int(n) + } + _, e := tx.q().ExecContext(ctx, "INSERT INTO analytics_repair_receipts(name,cursor,updated_at) VALUES(?,?,?) ON CONFLICT(name) DO UPDATE SET cursor=excluded.cursor,updated_at=excluded.updated_at", table, batch[len(batch)-1].id, time.Now().UTC().Format(time.RFC3339Nano)) + return e + }) + if err != nil { + return result, err + } + } else { + for _, r := range batch { + result.Scanned++ + p := CommentPublication(r.kind, r.raw) + if p.Submitted != r.submitted.String || p.Published != r.published.String { + result.WouldChange++ + } + if r.kind == "pull_review" && p.Submitted == "" { + result.Unknown++ + } + } + } + cursor = batch[len(batch)-1].id + } + } + if apply { + if result.Changed > 0 || pending { + if e := s.SetAnalyticsState(ctx, "publication_repair_generation", time.Now().UTC().Format(time.RFC3339Nano)); e != nil { + return result, e + } + } + if e := s.SetAnalyticsState(ctx, "publication_repair_in_progress", "0"); e != nil { + return result, e + } + } + return result, nil +} + +// Use the native node itself as the identity evidence key: login reuse is not an +// identity join. Deleted/unavailable actors are retained as explicit unknowns. +func (s *Store) SaveActorEvidence(ctx context.Context, nodes []map[string]any, at string) error { + return s.WithTx(ctx, func(tx *Store) error { + for _, n := range nodes { + id, _ := n["id"].(string) + if id == "" { + continue + } + if excluded, err := tx.NodeExcluded(ctx, id); err != nil { + return err + } else if excluded { + continue + } + a, _ := n["author"].(map[string]any) + actor, _ := a["id"].(string) + if actor != "" { + if _, e := tx.q().ExecContext(ctx, "INSERT OR IGNORE INTO analytics_pending_nodes(node_id,kind) VALUES(?,'profile')", actor); e != nil { + return e + } + } + login, _ := a["login"].(string) + typ, _ := a["__typename"].(string) + raw, e := json.Marshal(n) + if e != nil { + return e + } + if _, e = tx.q().ExecContext(ctx, `INSERT INTO actor_identity_evidence VALUES(?,?,?,?,?,?) ON CONFLICT(node_id) DO UPDATE SET actor_node_id=excluded.actor_node_id,login=excluded.login,actor_type=excluded.actor_type,observed_at=excluded.observed_at,raw_json=excluded.raw_json`, id, nullString(actor), nullString(login), nullString(typ), at, string(raw)); e != nil { + return e + } + } + return nil + }) +} +func (s *Store) SaveActorProfiles(ctx context.Context, nodes []map[string]any, at string) error { + return s.WithTx(ctx, func(tx *Store) error { + for _, n := range nodes { + id, _ := n["id"].(string) + login, _ := n["login"].(string) + typ, _ := n["__typename"].(string) + if id == "" { + continue + } + raw, e := json.Marshal(n) + if e != nil { + return e + } + name, _ := n["name"].(string) + bio, _ := n["bio"].(string) + url, _ := n["url"].(string) + _, e = tx.q().ExecContext(ctx, `INSERT INTO actor_profiles VALUES(?,?,?,?,?,?,?,?,?) ON CONFLICT(node_id) DO UPDATE SET login=excluded.login,actor_type=excluded.actor_type,name=excluded.name,bio=excluded.bio,url=excluded.url,created_at=excluded.created_at,observed_at=excluded.observed_at,raw_json=excluded.raw_json`, id, login, typ, nullString(name), nullString(bio), nullString(url), nullString(ProviderTime(n["createdAt"])), at, string(raw)) + if e != nil { + return e + } + } + return nil + }) +} +func (s *Store) SeedAnalyticsNodes(ctx context.Context, profiles bool) error { + query := `INSERT OR IGNORE INTO analytics_pending_nodes(node_id,kind) SELECT json_extract(raw_json,'$.node_id'),'identity' FROM threads WHERE json_valid(raw_json) AND json_extract(raw_json,'$.user.node_id') IS NULL AND json_extract(raw_json,'$.node_id') IS NOT NULL + UNION SELECT json_extract(raw_json,'$.node_id'),'identity' FROM comments WHERE json_valid(raw_json) AND json_extract(raw_json,'$.user.node_id') IS NULL AND json_extract(raw_json,'$.node_id') IS NOT NULL` + if profiles { + query = `INSERT OR IGNORE INTO analytics_pending_nodes(node_id,kind) SELECT json_extract(raw_json,'$.user.node_id'),'profile' FROM threads WHERE json_valid(raw_json) AND json_extract(raw_json,'$.user.node_id') IS NOT NULL + UNION SELECT json_extract(raw_json,'$.user.node_id'),'profile' FROM comments WHERE json_valid(raw_json) AND json_extract(raw_json,'$.user.node_id') IS NOT NULL + UNION SELECT actor_node_id,'profile' FROM actor_identity_evidence WHERE actor_node_id IS NOT NULL` + } + _, err := s.q().ExecContext(ctx, query) + return err +} +func (s *Store) AnalyticsIdentityNodes(ctx context.Context, limit int) ([]string, error) { + return s.analyticsNodes(ctx, limit, false) +} +func (s *Store) AnalyticsProfileNodes(ctx context.Context, limit int) ([]string, error) { + return s.analyticsNodes(ctx, limit, true) +} +func (s *Store) analyticsNodes(ctx context.Context, limit int, profiles bool) ([]string, error) { + query := `SELECT q.node_id FROM analytics_pending_nodes q WHERE kind='identity' AND NOT EXISTS(SELECT 1 FROM actor_identity_evidence e WHERE e.node_id=q.node_id) LIMIT ?` + if profiles { + query = `SELECT q.node_id FROM analytics_pending_nodes q WHERE kind='profile' AND NOT EXISTS(SELECT 1 FROM actor_profiles p WHERE p.node_id=q.node_id AND p.observed_at>=?) LIMIT ?` + } + var rows *sql.Rows + var e error + if profiles { + rows, e = s.q().QueryContext(ctx, query, time.Now().UTC().Add(-24*time.Hour).Format(time.RFC3339Nano), limit) + } else { + rows, e = s.q().QueryContext(ctx, query, limit) + } + if e != nil { + return nil, e + } + defer rows.Close() + var out []string + for rows.Next() { + var id string + if e = rows.Scan(&id); e != nil { + return nil, e + } + out = append(out, id) + } + return out, rows.Err() +} +func (s *Store) AnalyticsState(ctx context.Context, key string) (string, error) { + var value string + err := s.q().QueryRowContext(ctx, "SELECT value FROM analytics_collection_state WHERE name=?", key).Scan(&value) + if err == sql.ErrNoRows { + return "", nil + } + return value, err +} +func (s *Store) SetAnalyticsState(ctx context.Context, key, value string) error { + _, err := s.q().ExecContext(ctx, "INSERT INTO analytics_collection_state VALUES(?,?,?) ON CONFLICT(name) DO UPDATE SET value=excluded.value,updated_at=excluded.updated_at", key, value, time.Now().UTC().Format(time.RFC3339Nano)) + return err +} +func (s *Store) SaveAnalyticsCoverage(ctx context.Context, repo, through string, issues, prs int) error { + _, e := s.q().ExecContext(ctx, `INSERT INTO analytics_coverage VALUES(?,?,?,?,1,?) ON CONFLICT(repository) DO UPDATE SET through=excluded.through,issues=excluded.issues,pull_requests=excluded.pull_requests,complete=1,observed_at=excluded.observed_at`, repo, through, issues, prs, time.Now().UTC().Format(time.RFC3339Nano)) + return e +} diff --git a/internal/store/analytics_source_test.go b/internal/store/analytics_source_test.go new file mode 100644 index 00000000..bd2390a0 --- /dev/null +++ b/internal/store/analytics_source_test.go @@ -0,0 +1,93 @@ +package store + +import ( + "context" + "encoding/json" + "path/filepath" + "testing" +) + +func TestPublicationRepairPreservesSourceEvidence(t *testing.T) { + ctx := context.Background() + s, e := Open(ctx, filepath.Join(t.TempDir(), "source.db")) + if e != nil { + t.Fatal(e) + } + defer s.Close() + _, e = s.db.Exec(`INSERT INTO repositories(id,owner,name,full_name,github_repo_id,raw_json,updated_at) VALUES(1,'fixture','repo','fixture/repo',1,'{}','2026-01-01'); + INSERT INTO threads(id,repo_id,github_id,number,kind,state,title,body,html_url,labels_json,assignees_json,raw_json,content_hash,updated_at) VALUES(1,1,'pr',1,'pull_request','open','','','','[]','[]','{}','h','2026-01-01')`) + if e != nil { + t.Fatal(e) + } + raw := `{"state":"APPROVED","created_at":"2026-01-01T00:00:00Z","submitted_at":"2026-01-03T12:34:56Z","user":{"login":"a","type":"User"}}` + id, e := s.UpsertComment(ctx, Comment{ThreadID: 1, GitHubID: "review", CommentType: "pull_review", RawJSON: raw, CreatedAtGitHub: "2026-01-01T00:00:00Z"}) + if e != nil { + t.Fatal(e) + } + _, e = s.db.Exec("UPDATE comments SET submitted_at_gh=NULL,publication_at_gh=NULL; UPDATE comment_revisions SET submitted_at_gh=NULL,publication_at_gh=NULL") + if e != nil { + t.Fatal(e) + } + var originalRecorded string + s.db.QueryRow("SELECT recorded_at FROM comment_revisions").Scan(&originalRecorded) + preview, e := s.RepairPublication(ctx, false) + if e != nil || preview.WouldChange != 2 || preview.Changed != 0 { + t.Fatalf("dry-run %+v %v", preview, e) + } + r, e := s.RepairPublication(ctx, true) + if e != nil || r.Changed != 2 { + t.Fatalf("%+v %v", r, e) + } + r, e = s.RepairPublication(ctx, true) + if e != nil || r.Changed != 0 { + t.Fatalf("replay %+v %v", r, e) + } + var published, submitted, created, stored string + s.db.QueryRow("SELECT publication_at_gh,submitted_at_gh,created_at_gh,raw_json FROM comments WHERE id=?", id).Scan(&published, &submitted, &created, &stored) + if published != "2026-01-03T12:34:56Z" || submitted != published || created != "2026-01-01T00:00:00Z" || stored != raw { + t.Fatalf("source timestamps/evidence changed incorrectly") + } + var n int + var recorded string + s.db.QueryRow("SELECT count(*),min(recorded_at) FROM comment_revisions").Scan(&n, &recorded) + if n != 1 || recorded != originalRecorded { + t.Fatal("repair manufactured a source revision") + } + for _, c := range []struct{ kind, raw, want string }{ + {"pull_review", `{"state":"PENDING","submitted_at":"2026-01-01T00:00:00Z"}`, ""}, + {"pull_review", `{"submitted_at":"not a date"}`, ""}, + {"pull_review", `{"_graphql":{"submittedAt":"2026-01-01T00:00:00+02:00"}}`, "2025-12-31T22:00:00Z"}, + {"pull_review_comment", `{"created_at":"2026-01-01T00:00:00Z","_graphql":{"publishedAt":"2026-01-02T00:00:00Z"}}`, "2026-01-02T00:00:00Z"}, + } { + if got := CommentPublication(c.kind, c.raw).Published; got != c.want { + t.Fatalf("publication=%q want %q", got, c.want) + } + } + nodes := []map[string]any{{"id": "comment-node", "author": map[string]any{"id": "user-one", "login": "same", "__typename": "User"}}, {"id": "other-node", "author": map[string]any{"id": "user-two", "login": "same", "__typename": "User"}}} + if e = s.SaveActorEvidence(ctx, nodes, "2026-01-01T00:00:00Z"); e != nil { + t.Fatal(e) + } + s.db.QueryRow("SELECT count(distinct actor_node_id) FROM actor_identity_evidence").Scan(&n) + if n != 2 { + t.Fatal("login reuse merged identities") + } + _, _ = json.Marshal(nodes) +} + +func TestAnalyticsQueuesNewUnresolvedSourceIdentities(t *testing.T) { + ctx := context.Background() + s, err := Open(ctx, filepath.Join(t.TempDir(), "source.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + for _, raw := range []string{`{"node_id":"content","user":{"node_id":"actor"}}`, `{"node_id":"unresolved","user":null}`} { + if err := s.queueAnalyticsActor(ctx, raw); err != nil { + t.Fatal(err) + } + } + var count int + if err := s.DB().QueryRowContext(ctx, `SELECT count(*) FROM analytics_pending_nodes WHERE (node_id='actor' AND kind='profile') OR (node_id='unresolved' AND kind='identity')`).Scan(&count); err != nil || count != 2 { + t.Fatalf("count=%d err=%v", count, err) + } +} diff --git a/internal/store/comments.go b/internal/store/comments.go index f2c599ed..d523faf6 100644 --- a/internal/store/comments.go +++ b/internal/store/comments.go @@ -12,15 +12,16 @@ import ( ) type Comment struct { - ID int64 `json:"id"` - ThreadID int64 `json:"thread_id"` - GitHubID string `json:"github_id"` - CommentType string `json:"comment_type"` - AuthorLogin string `json:"author_login,omitempty"` - AuthorType string `json:"author_type,omitempty"` - Body string `json:"body"` - IsBot bool `json:"is_bot"` - ReviewState string `json:"review_state,omitempty"` + ID int64 `json:"id"` + ThreadID int64 `json:"thread_id"` + GitHubID string `json:"github_id"` + CommentType string `json:"comment_type"` + AuthorLogin string `json:"author_login,omitempty"` + AuthorType string `json:"author_type,omitempty"` + Body string `json:"body"` + IsBot bool `json:"is_bot"` + ReviewState string `json:"review_state,omitempty"` + Publication RawJSON string `json:"-"` CreatedAtGitHub string `json:"created_at_gh,omitempty"` UpdatedAtGitHub string `json:"updated_at_gh,omitempty"` @@ -71,6 +72,9 @@ func (s *Store) upsertComment(ctx context.Context, comment Comment) (int64, erro if err != nil { return 0, fmt.Errorf("upsert comment: %w", err) } + if err := s.upsertCommentPublication(ctx, id, comment.CommentType, comment.RawJSON); err != nil { + return 0, err + } if s.portableCommentBodyMetadata { if _, err := s.q().ExecContext(ctx, ` update comments @@ -131,10 +135,10 @@ func (s *Store) recordCommentRevision(ctx context.Context, commentID int64, reco if _, err := s.q().ExecContext(ctx, ` insert into comment_revisions( comment_id, author_login, author_type, body, is_bot, raw_json, - created_at_gh, updated_at_gh, deleted_at, deletion_reason, recorded_at + created_at_gh, updated_at_gh, deleted_at, deletion_reason, recorded_at,submitted_at_gh,publication_at_gh ) select c.id, c.author_login, c.author_type, c.body, c.is_bot, c.raw_json, - c.created_at_gh, c.updated_at_gh, c.deleted_at, c.deletion_reason, ? + c.created_at_gh, c.updated_at_gh, c.deleted_at, c.deletion_reason, ?,c.submitted_at_gh,c.publication_at_gh from comments c where c.id = ? and not exists ( @@ -180,6 +184,7 @@ func (s *Store) ListComments(ctx context.Context, threadID int64) ([]Comment, er Body: row.Body, IsBot: int64Bool(row.IsBot), ReviewState: reviewStateFromRawJSON(row.RawJson), + Publication: CommentPublication(row.CommentType, row.RawJson), RawJSON: row.RawJson, CreatedAtGitHub: stringValue(row.CreatedAtGh), UpdatedAtGitHub: stringValue(row.UpdatedAtGh), diff --git a/internal/store/portable.go b/internal/store/portable.go index 062e9801..1b23b7ec 100644 --- a/internal/store/portable.go +++ b/internal/store/portable.go @@ -94,6 +94,17 @@ type PortablePruneStats struct { } func (s *Store) PrunePortablePayloads(ctx context.Context, options PortablePruneOptions) (PortablePruneStats, error) { + // Owner exclusions belong to this local archive. Do not silently drop their + // policy or publish owner-request metadata through a portable export. + if s.hasTable(ctx, "thread_exclusions") { + var n int + if err := s.q().QueryRowContext(ctx, "SELECT count(*) FROM thread_exclusions").Scan(&n); err != nil { + return PortablePruneStats{}, err + } + if n > 0 { + return PortablePruneStats{}, fmt.Errorf("portable publication is unavailable for an archive with local owner exclusions") + } + } if options.BodyChars <= 0 { options.BodyChars = 256 } diff --git a/internal/store/review_threads.go b/internal/store/review_threads.go index baf3d1c6..8bc50795 100644 --- a/internal/store/review_threads.go +++ b/internal/store/review_threads.go @@ -3,7 +3,9 @@ package store import ( "context" "database/sql" + "encoding/json" "fmt" + "strings" "time" "github.com/openclaw/gitcrawl/internal/store/storedb" @@ -34,7 +36,12 @@ type PullRequestReviewThread struct { } func (s *Store) UpsertPullRequestReviewThreads(ctx context.Context, threadID int64, fetchedAt string, threads []PullRequestReviewThread) error { + seen := map[string]bool{} for _, thread := range threads { + if thread.ReviewThreadID == "" || strings.TrimSpace(thread.ReviewThreadID) != thread.ReviewThreadID || seen[thread.ReviewThreadID] { + return fmt.Errorf("review-thread membership requires nonempty unique native IDs") + } + seen[thread.ReviewThreadID] = true if err := validateTombstone(thread.DeletedAt, thread.DeletionReason); err != nil { return fmt.Errorf("upsert pull request review thread %q: %w", thread.ReviewThreadID, err) } @@ -48,12 +55,43 @@ func (s *Store) UpsertPullRequestReviewThreads(ctx context.Context, threadID int } func (s *Store) upsertPullRequestReviewThreads(ctx context.Context, threadID int64, fetchedAt string, threads []PullRequestReviewThread) error { + observed, err := time.Parse(time.RFC3339Nano, fetchedAt) + if err != nil { + return fmt.Errorf("invalid review-thread observation time: %w", err) + } + var previous string + err = s.q().QueryRowContext(ctx, "SELECT fetched_at FROM pull_request_review_thread_syncs WHERE thread_id=?", threadID).Scan(&previous) + if err != nil && err != sql.ErrNoRows { + return err + } + if err == nil { + prior, parseErr := time.Parse(time.RFC3339Nano, previous) + if parseErr != nil { + return fmt.Errorf("invalid prior review-thread observation time: %w", parseErr) + } + // This runs inside the same native transaction as state/history writes. + // An older completion must not replace either membership or row state. + if observed.Before(prior) { + return nil + } + } if err := s.qsql().UpsertPullRequestReviewThreadSync(ctx, storedb.UpsertPullRequestReviewThreadSyncParams{ ThreadID: threadID, FetchedAt: fetchedAt, }); err != nil { return fmt.Errorf("mark pull request review threads fetched: %w", err) } + ids := make([]string, 0, len(threads)) + for _, thread := range threads { + ids = append(ids, thread.ReviewThreadID) + } + encoded, err := json.Marshal(ids) + if err != nil { + return err + } + if _, err = s.q().ExecContext(ctx, "UPDATE pull_request_review_thread_syncs SET review_thread_ids_json=? WHERE thread_id=? AND fetched_at=?", string(encoded), threadID, fetchedAt); err != nil { + return err + } for _, thread := range threads { if thread.ReviewThreadID == "" { continue diff --git a/internal/store/schema_convergence.go b/internal/store/schema_convergence.go index 59698397..7a76520e 100644 --- a/internal/store/schema_convergence.go +++ b/internal/store/schema_convergence.go @@ -150,6 +150,9 @@ func inspectCompatibilityMigrationsMode( if current > 0 && !st.observationSchemaConvergenceHasCurrentShape(ctx) { add(migrationObservationSchemaConvergence) } + if current >= 16 && (!st.hasTable(ctx, "thread_exclusions") || !st.hasTable(ctx, "thread_excluded_nodes") || !st.hasColumn(ctx, "analytics_review_state_coverage", "owner_excluded_items")) { + add("thread_exclusions_schema") + } if !includeSemantic { return pending, nil } diff --git a/internal/store/store.go b/internal/store/store.go index 501a1e32..75d2d602 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -17,7 +17,7 @@ import ( ) const ( - schemaVersion = 13 + schemaVersion = 16 timeLayout = time.RFC3339Nano ) @@ -285,6 +285,33 @@ func (s *Store) migrate(ctx context.Context) error { if _, err := s.db.ExecContext(ctx, schemaSQL); err != nil { return fmt.Errorf("apply schema: %w", err) } + // Additive analytics/policy upgrades need no row/history rebuild on a + // converged archive; exclusion changes never rewrite source observations. + if current == 14 || current == 15 { + structural, e := inspectStructuralCompatibilityMigrations(ctx, s, current, inspectPRDetailSchema(ctx, s)) + if e != nil { + return e + } + converged, e := s.observationSchemaConvergenceIsCurrent(ctx) + if e != nil { + return e + } + if len(structural) == 1 && structural[0] == fmt.Sprintf("schema_version_%d_to_%d", current, schemaVersion) && converged { + // Some v14 archives predate the optional analytics extension. These + // additive tables/columns are cheap to ensure and require no row scan. + if e = s.ensureAnalyticsSourceSchema(ctx); e != nil { + return e + } + if e = s.ensureAnalyticsIntegritySchema(ctx); e != nil { + return e + } + if e = s.ensureThreadExclusionsSchema(ctx); e != nil { + return e + } + _, e = s.db.ExecContext(ctx, fmt.Sprintf("PRAGMA user_version=%d", schemaVersion)) + return e + } + } if current == schemaVersion { prDetails := inspectPRDetailSchema(ctx, s) structural, err := inspectStructuralCompatibilityMigrations( @@ -319,6 +346,15 @@ func (s *Store) migrate(ctx context.Context) error { if err := s.ensureFamilyTombstoneSchema(ctx); err != nil { return err } + if err := s.ensureAnalyticsSourceSchema(ctx); err != nil { + return err + } + if err := s.ensureAnalyticsIntegritySchema(ctx); err != nil { + return err + } + if err := s.ensureThreadExclusionsSchema(ctx); err != nil { + return err + } if err := s.ensureCanonicalObservationTables(ctx); err != nil { return err } diff --git a/internal/store/thread_exclusions.go b/internal/store/thread_exclusions.go new file mode 100644 index 00000000..4dcc12d2 --- /dev/null +++ b/internal/store/thread_exclusions.go @@ -0,0 +1,503 @@ +package store + +import ( + "context" + "database/sql" + "errors" + "fmt" + "net/url" + "sort" + "strings" + "time" + + "github.com/openclaw/gitcrawl/internal/store/storedb" +) + +var ErrThreadExcluded = errors.New("thread is excluded by owner policy") + +func (s *Store) ensureThreadExclusionsSchema(ctx context.Context) error { + _, err := s.q().ExecContext(ctx, ` +CREATE TABLE IF NOT EXISTS thread_exclusions( + repository TEXT NOT NULL,number INTEGER NOT NULL CHECK(number>0),kind TEXT NOT NULL, + original_thread_id INTEGER NOT NULL,github_id TEXT NOT NULL,excluded_at TEXT NOT NULL, + reason TEXT NOT NULL CHECK(reason='owner_requested'),request_id TEXT NOT NULL, + PRIMARY KEY(repository,number)); +CREATE TABLE IF NOT EXISTS thread_excluded_nodes( + node_id TEXT PRIMARY KEY,repository TEXT NOT NULL,number INTEGER NOT NULL, + FOREIGN KEY(repository,number) REFERENCES thread_exclusions(repository,number)); +`) + if err != nil { + return err + } + + if s.hasTable(ctx, "analytics_review_state_coverage") && !s.hasColumn(ctx, "analytics_review_state_coverage", "owner_excluded_items") { + _, err = s.q().ExecContext(ctx, "ALTER TABLE analytics_review_state_coverage ADD COLUMN owner_excluded_items INTEGER NOT NULL DEFAULT 0") + } + return err +} + +func (s *Store) ensureThreadExclusionGuards(ctx context.Context) error { + _, err := s.q().ExecContext(ctx, ` +CREATE TRIGGER IF NOT EXISTS owner_excluded_thread_insert BEFORE INSERT ON threads + WHEN EXISTS(SELECT 1 FROM thread_exclusions e JOIN repositories r ON lower(r.full_name)=e.repository WHERE r.id=NEW.repo_id AND e.number=NEW.number) + BEGIN SELECT RAISE(ABORT,'owner_excluded_thread'); END; +CREATE TRIGGER IF NOT EXISTS owner_excluded_thread_update BEFORE UPDATE ON threads + WHEN EXISTS(SELECT 1 FROM thread_exclusions e JOIN repositories r ON lower(r.full_name)=e.repository WHERE r.id=NEW.repo_id AND e.number=NEW.number) + BEGIN SELECT RAISE(ABORT,'owner_excluded_thread'); END; +`) + if err != nil { + return err + } + for _, guard := range []struct{ table, name, condition, message string }{ + {"analytics_retries", "retry", "EXISTS(SELECT 1 FROM thread_exclusions WHERE repository=lower(NEW.repository) AND number=NEW.number)", "owner_excluded_thread"}, + {"actor_identity_evidence", "identity", "EXISTS(SELECT 1 FROM thread_excluded_nodes WHERE node_id=NEW.node_id)", "owner_excluded_node"}, + {"analytics_pending_nodes", "pending_node", "EXISTS(SELECT 1 FROM thread_excluded_nodes WHERE node_id=NEW.node_id)", "owner_excluded_node"}, + } { + if !s.hasTable(ctx, guard.table) { + continue + } + if _, err = s.q().ExecContext(ctx, "CREATE TRIGGER IF NOT EXISTS owner_excluded_"+guard.name+" BEFORE INSERT ON "+guard.table+" WHEN "+guard.condition+" BEGIN SELECT RAISE(ABORT,'"+guard.message+"'); END"); err != nil { + return err + } + } + return nil +} + +// OpenThreadPurgeMirror opens an existing managed portable mirror without a full +// archive migration, body reconstruction, or publication. The CLI holds the +// portable owner lease and records writable-mirror ownership first. +func OpenThreadPurgeMirror(ctx context.Context, path string) (*Store, error) { + probe, err := OpenReadOnly(ctx, path) + if err != nil { + return nil, err + } + portable := probe.hasTable(ctx, "portable_metadata") + // Older sparse formats can contain dangling blob FKs. Do not disable FK + // validation or rebuild them as part of a targeted owner purge. + portable = portable && (!probe.hasColumn(ctx, "comments", "raw_json_blob_id") || probe.hasTable(ctx, "blobs")) + version, versionErr := probe.schemaVersion(ctx) + probe.Close() + if !portable || versionErr != nil || version != portableSchemaVersion { + return nil, fmt.Errorf("owner purge requires a current portable mirror") + } + // Preserve the mirror's journal mode. A failed transaction must not turn a + // pristine replica into local data merely by switching its SQLite header. + encoded, err := immutableSQLiteURI(path) + if err != nil { + return nil, err + } + uri, err := url.Parse(encoded) + if err != nil { + return nil, err + } + query := uri.Query() + query.Del("immutable") + query.Set("mode", "rw") + query.Add("_pragma", "foreign_keys(1)") + query.Add("_pragma", "busy_timeout(5000)") + uri.RawQuery = query.Encode() + db, err := sql.Open("sqlite", uri.String()) + if err != nil { + return nil, err + } + db.SetMaxOpenConns(1) + db.SetMaxIdleConns(1) + if err = db.PingContext(ctx); err != nil { + db.Close() + return nil, err + } + return &Store{db: db, sqlc: storedb.New(db), path: path}, nil +} + +func (s *Store) ThreadExcluded(ctx context.Context, repository string, number int) (bool, error) { + var excluded bool + err := s.q().QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM thread_exclusions WHERE repository=lower(?) AND number=?)", repository, number).Scan(&excluded) + return excluded, err +} +func (s *Store) NodeExcluded(ctx context.Context, id string) (bool, error) { + var excluded bool + err := s.q().QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM thread_excluded_nodes WHERE node_id=?)", id).Scan(&excluded) + return excluded, err +} +func (s *Store) FilterExcludedNumbers(ctx context.Context, repository string, numbers []int) ([]int, error) { + kept := make([]int, 0, len(numbers)) + for _, n := range numbers { + excluded, err := s.ThreadExcluded(ctx, repository, n) + if err != nil { + return nil, err + } + if !excluded { + kept = append(kept, n) + } + } + return kept, nil +} +func (s *Store) OwnerExcludedCount(ctx context.Context, repository string, reviewOnly bool) (int, error) { + var n int + err := s.q().QueryRowContext(ctx, "SELECT count(*) FROM thread_exclusions WHERE repository=lower(?) AND (?=0 OR kind='pull_request')", repository, boolInt(reviewOnly)).Scan(&n) + return n, err +} + +type PurgeTarget struct { + Number int `json:"number"` + ThreadID int64 `json:"thread_id"` + Kind string `json:"kind"` + GitHubID string `json:"github_id"` + Nodes []string `json:"content_node_ids"` + AlreadyExcluded bool `json:"already_excluded"` +} +type ThreadPurgePlan struct { + Repository string `json:"repository"` + RepoID int64 `json:"repo_id"` + Targets []PurgeTarget `json:"targets"` + Counts map[string]int `json:"counts"` + BlobIDs []int64 `json:"candidate_blob_ids"` + RequestID string `json:"request_id,omitempty"` + Applied bool `json:"applied"` + Reason string `json:"reason"` +} + +// Only source-owned relationships are traversed. Shared cluster descriptions +// need their own owner repair and are refused instead of discarding peer data. +func purgeRelations(ids string) map[string]string { + rev := "select id from thread_revisions where thread_id in (" + ids + ")" + snapshots := "select id from thread_code_snapshots where thread_revision_id in (" + rev + ")" + out := map[string]string{} + for _, table := range []string{"threads", "comments", "documents", "document_embeddings", "document_summaries", "pull_request_details", "pull_request_files", "pull_request_commits", "pull_request_checks", "pull_request_review_threads", "pull_request_review_thread_revisions", "pull_request_review_thread_syncs", "thread_revisions", "thread_vectors", "thread_child_observation_memberships", "thread_child_observation_reservations", "cluster_members", "cluster_memberships", "cluster_overrides"} { + col := "thread_id" + if table == "threads" { + col = "id" + } + out[table] = col + " in (" + ids + ")" + } + out["comment_revisions"] = "comment_id in (select id from comments where thread_id in (" + ids + "))" + for _, table := range []string{"thread_code_snapshots", "thread_fingerprints", "thread_key_summaries"} { + out[table] = "thread_revision_id in (" + rev + ")" + } + for _, table := range []string{"thread_changed_files", "thread_hunk_signatures"} { + out[table] = "snapshot_id in (" + snapshots + ")" + } + out["similarity_edges"] = "left_thread_id in (" + ids + ") or right_thread_id in (" + ids + ")" + return out +} + +var purgeBlobColumns = map[string][]string{"comments": {"raw_json_blob_id"}, "thread_revisions": {"raw_json_blob_id"}, "thread_changed_files": {"patch_blob_id"}, "thread_code_snapshots": {"raw_diff_blob_id"}} + +func (s *Store) PlanThreadPurge(ctx context.Context, repository string, numbers []int) (ThreadPurgePlan, error) { + p := ThreadPurgePlan{Repository: strings.ToLower(strings.TrimSpace(repository)), Reason: "owner_requested", Counts: map[string]int{}, Targets: []PurgeTarget{}, BlobIDs: []int64{}} + if len(numbers) == 0 || len(numbers) > 100 { + return p, fmt.Errorf("purge requires 1..100 explicit numbers") + } + var canonical string + var matches int + if err := s.q().QueryRowContext(ctx, "SELECT count(*),coalesce(min(full_name),'') FROM repositories WHERE lower(full_name)=?", p.Repository).Scan(&matches, &canonical); err != nil { + return p, err + } + if matches != 1 { + return p, fmt.Errorf("purge repository must match exactly one local identity") + } + repo, err := s.RepositoryByFullName(ctx, canonical) + if err != nil { + return p, err + } + p.RepoID = repo.ID + seen := map[int]bool{} + var ids []string + for _, n := range numbers { + if n <= 0 || seen[n] { + return p, fmt.Errorf("purge numbers must be positive and unique") + } + seen[n] = true + t := PurgeTarget{Number: n, Nodes: []string{}} + var matches int + if err := s.q().QueryRowContext(ctx, "SELECT count(*) FROM threads WHERE repo_id=? AND number=?", repo.ID, n).Scan(&matches); err != nil { + return p, err + } + if matches > 1 { + return p, fmt.Errorf("ambiguous native thread number #%d", n) + } + err := s.q().QueryRowContext(ctx, "SELECT id,kind,github_id FROM threads WHERE repo_id=? AND number=?", repo.ID, n).Scan(&t.ThreadID, &t.Kind, &t.GitHubID) + if err == sql.ErrNoRows && s.hasTable(ctx, "thread_exclusions") { + err = s.q().QueryRowContext(ctx, "SELECT original_thread_id,kind,github_id FROM thread_exclusions WHERE repository=? AND number=?", p.Repository, n).Scan(&t.ThreadID, &t.Kind, &t.GitHubID) + t.AlreadyExcluded = err == nil + } + if err != nil { + return p, fmt.Errorf("purge target #%d: %w", n, err) + } + if t.AlreadyExcluded { + p.Targets = append(p.Targets, t) + continue + } + ids = append(ids, fmt.Sprint(t.ThreadID)) + // historyProjection normalizes GraphQL id to node_id; top-level id + // is the REST/fullDatabaseId and must not become a node exclusion. + // Portable schemas may intentionally omit raw payload columns. Retain + // only available content-node keys; the repository/number guard is primary. + nodeQueries := []string{} + for _, rel := range []struct{ table, predicate string }{ + {"threads", fmt.Sprintf("id=%d", t.ThreadID)}, + {"comments", fmt.Sprintf("thread_id=%d", t.ThreadID)}, + {"comment_revisions", fmt.Sprintf("comment_id IN(SELECT id FROM comments WHERE thread_id=%d)", t.ThreadID)}, + {"thread_revisions", fmt.Sprintf("thread_id=%d", t.ThreadID)}, + } { + if s.hasColumn(ctx, rel.table, "raw_json") { + nodeQueries = append(nodeQueries, "SELECT json_extract(CASE WHEN json_valid(raw_json) THEN raw_json ELSE '{}' END,'$.node_id') node_id FROM "+rel.table+" WHERE "+rel.predicate) + } + } + if len(nodeQueries) > 0 { + rows, e := s.q().QueryContext(ctx, "SELECT DISTINCT node_id FROM ("+strings.Join(nodeQueries, " UNION ALL ")+") WHERE node_id IS NOT NULL AND node_id<>'' ORDER BY node_id") + if e != nil { + return p, e + } + for rows.Next() { + var node string + if e = rows.Scan(&node); e != nil { + rows.Close() + return p, e + } + t.Nodes = append(t.Nodes, node) + } + e = rows.Err() + rows.Close() + if e != nil { + return p, e + } + } + p.Targets = append(p.Targets, t) + } + sort.Slice(p.Targets, func(i, j int) bool { return p.Targets[i].Number < p.Targets[j].Number }) + if len(ids) == 0 { + return p, nil + } + idSQL := strings.Join(ids, ",") + relations := purgeRelations(idSQL) + for table, predicate := range relations { + if !s.hasTable(ctx, table) { + continue + } + var n int + if err = s.q().QueryRowContext(ctx, "SELECT count(*) FROM "+table+" WHERE "+predicate).Scan(&n); err != nil { + return p, err + } + p.Counts[table] = n + } + for _, table := range []string{"clusters", "cluster_groups"} { + if !s.hasTable(ctx, table) { + continue + } + var n int + if err = s.q().QueryRowContext(ctx, "SELECT count(*) FROM "+table+" WHERE representative_thread_id in ("+idSQL+")").Scan(&n); err != nil { + return p, err + } + if n > 0 { + return p, fmt.Errorf("purge requires a separate cluster representative repair") + } + } + for _, table := range []string{"cluster_members", "cluster_memberships", "cluster_overrides"} { + if p.Counts[table] > 0 { + return p, fmt.Errorf("purge requires a separate shared cluster repair") + } + } + // This deliberately bounded command does not resolve ownership inside + // retained workflow payloads. Refuse the repository surface rather than + // guessing from today's PR head or deleting a shared run. + if s.hasTable(ctx, "github_workflow_runs") { + var exists bool + if err = s.q().QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM github_workflow_runs WHERE repo_id=?)", p.RepoID).Scan(&exists); err != nil { + return p, err + } + if exists { + return p, fmt.Errorf("purge requires a separate repository workflow-snapshot repair") + } + } + heads := []string{} + if s.hasTable(ctx, "pull_request_details") { + heads = append(heads, "SELECT head_sha FROM pull_request_details WHERE thread_id IN("+idSQL+")") + } + if s.hasTable(ctx, "thread_code_snapshots") { + heads = append(heads, "SELECT head_sha FROM thread_code_snapshots WHERE "+relations["thread_code_snapshots"]) + } + for _, table := range []string{"threads", "thread_revisions"} { + if !s.hasColumn(ctx, table, "raw_json") { + continue + } + for _, path := range []string{"$._graphql.headRefOid", "$.head.sha"} { + heads = append(heads, "SELECT json_extract(CASE WHEN json_valid(raw_json) THEN raw_json ELSE '{}' END,'"+path+"') FROM "+table+" WHERE "+relations[table]) + } + } + if s.hasTable(ctx, "workflow_run_observation_reservations") && len(heads) > 0 { + var n int + if err = s.q().QueryRowContext(ctx, "SELECT count(*) FROM workflow_run_observation_reservations WHERE repo_id=? AND head_sha IN("+strings.Join(heads, " UNION ")+")", p.RepoID).Scan(&n); err != nil { + return p, err + } + if n > 0 { + return p, fmt.Errorf("purge requires a separate linked workflow-reservation repair") + } + p.Counts["workflow_run_observation_reservations"] = 0 + } + // Blob-backed payloads need an identity/ownership-aware repair of their + // own. Do not silently leave their nodes behind or read unknown blob prose. + for table, cols := range purgeBlobColumns { + for _, col := range cols { + if !s.hasColumn(ctx, table, col) { + continue + } + var exists bool + if err = s.q().QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM "+table+" WHERE ("+relations[table]+") AND "+col+" IS NOT NULL)").Scan(&exists); err != nil { + return p, err + } + if exists { + return p, fmt.Errorf("purge requires a separate blob-backed payload repair") + } + } + } + + for _, t := range p.Targets { + if t.AlreadyExcluded { + continue + } + for _, table := range []string{"analytics_retries", "analytics_fetch_attempts", "sync_attempt_failures"} { + if !s.hasTable(ctx, table) { + continue + } + clause, args := "lower(repository)=? AND number=?", []any{p.Repository, t.Number} + if table == "sync_attempt_failures" { + clause, args = "repo_id=? AND number=?", []any{p.RepoID, t.Number} + } + var n int + if err = s.q().QueryRowContext(ctx, "SELECT count(*) FROM "+table+" WHERE "+clause, args...).Scan(&n); err != nil { + return p, err + } + p.Counts[table] += n + } + for _, node := range t.Nodes { + for _, table := range []string{"actor_identity_evidence", "analytics_pending_nodes"} { + if !s.hasTable(ctx, table) { + continue + } + var n int + if err = s.q().QueryRowContext(ctx, "SELECT count(*) FROM "+table+" WHERE node_id=?", node).Scan(&n); err != nil { + return p, err + } + p.Counts[table] += n + } + } + } + // A durable downstream exclusion is keyed by native row ID. Refuse a tail + // deletion that SQLite could later reuse; never create dummy rows or IDs. + guardRelations := purgeRelations(idSQL) + var numeric []string + for _, t := range p.Targets { + numeric = append(numeric, fmt.Sprint(t.Number)) + } + guardRelations["analytics_fetch_attempts"] = fmt.Sprintf("lower(repository)='%s' AND number IN(%s)", strings.ReplaceAll(p.Repository, "'", "''"), strings.Join(numeric, ",")) + guardRelations["sync_attempt_failures"] = fmt.Sprintf("repo_id=%d AND number IN(%s)", p.RepoID, strings.Join(numeric, ",")) + for table, predicate := range guardRelations { + var singleID bool + if err := s.q().QueryRowContext(ctx, "SELECT count(*)=1 AND min(name)='id' AND min(upper(type))='INTEGER' FROM pragma_table_info(?) WHERE pk>0", table).Scan(&singleID); err != nil { + return p, err + } + if singleID { + if err := s.guardPurgeIDTail(ctx, table, predicate); err != nil { + return p, err + } + } + } + return p, nil +} + +func (s *Store) guardPurgeIDTail(ctx context.Context, table, predicate string) error { + var selected, maximum sql.NullInt64 + if err := s.q().QueryRowContext(ctx, "SELECT max(id) FROM "+table+" WHERE "+predicate).Scan(&selected); err != nil { + return err + } + if !selected.Valid { + return nil + } + if err := s.q().QueryRowContext(ctx, "SELECT max(id) FROM "+table).Scan(&maximum); err != nil { + return err + } + if selected.Int64 == maximum.Int64 { + return fmt.Errorf("purge would permit native ID reuse in %s; a retained higher ID is required", table) + } + return nil +} + +// PurgeThreads is owner-directed removal, not a provider deletion or successful +// fetch. Caller holds runner.lock; a single transaction fences all native writes. +func (s *Store) PurgeThreads(ctx context.Context, repository string, numbers []int, requestID string) (ThreadPurgePlan, error) { + var result ThreadPurgePlan + if strings.TrimSpace(requestID) == "" || len(requestID) > 128 { + return result, fmt.Errorf("purge requires a bounded owner request id") + } + err := s.WithTx(ctx, func(tx *Store) error { + if err := tx.ensureThreadExclusionsSchema(ctx); err != nil { + return err + } + if err := tx.ensureThreadExclusionGuards(ctx); err != nil { + return err + } + p, err := tx.PlanThreadPurge(ctx, repository, numbers) + if err != nil { + return err + } + var enabled int + if err = tx.q().QueryRowContext(ctx, "PRAGMA foreign_keys").Scan(&enabled); err != nil { + return err + } + if enabled != 1 { + return fmt.Errorf("purge requires foreign keys") + } + at := time.Now().UTC().Format(time.RFC3339Nano) + for _, t := range p.Targets { + if t.AlreadyExcluded { + continue + } + if _, err = tx.q().ExecContext(ctx, `INSERT INTO thread_exclusions(repository,number,kind,original_thread_id,github_id,excluded_at,reason,request_id) VALUES(?,?,?,?,?,?,'owner_requested',?)`, p.Repository, t.Number, t.Kind, t.ThreadID, t.GitHubID, at, requestID); err != nil { + return err + } + for _, node := range t.Nodes { + if _, err = tx.q().ExecContext(ctx, "INSERT INTO thread_excluded_nodes(node_id,repository,number) VALUES(?,?,?)", node, p.Repository, t.Number); err != nil { + return err + } + for _, table := range []string{"actor_identity_evidence", "analytics_pending_nodes"} { + if !tx.hasTable(ctx, table) { + continue + } + if _, err = tx.q().ExecContext(ctx, "DELETE FROM "+table+" WHERE node_id=?", node); err != nil { + return err + } + } + } + for _, table := range []string{"analytics_retries", "analytics_fetch_attempts"} { + if !tx.hasTable(ctx, table) { + continue + } + if _, err = tx.q().ExecContext(ctx, "DELETE FROM "+table+" WHERE lower(repository)=? AND number=?", p.Repository, t.Number); err != nil { + return err + } + } + if tx.hasTable(ctx, "sync_attempt_failures") { + if _, err = tx.q().ExecContext(ctx, "DELETE FROM sync_attempt_failures WHERE repo_id=? AND number=?", p.RepoID, t.Number); err != nil { + return err + } + } + if _, err = tx.q().ExecContext(ctx, "DELETE FROM threads WHERE id=?", t.ThreadID); err != nil { + return err + } + } + // Never turn removal of unavailable work into claimed provider completeness. + if tx.hasTable(ctx, "analytics_review_state_coverage") { + if _, err = tx.q().ExecContext(ctx, `UPDATE analytics_review_state_coverage SET pending_items=(SELECT count(*) FROM analytics_retries WHERE lower(repository)=? AND operation='review_state' AND resolved_at IS NULL),owner_excluded_items=(SELECT count(*) FROM thread_exclusions WHERE repository=? AND kind='pull_request'),complete=0 WHERE lower(repository)=?`, p.Repository, p.Repository, p.Repository); err != nil { + return err + } + } + p.RequestID = requestID + result = p + return nil + }) + if err == nil { + result.Applied = true + } + return result, err +} diff --git a/internal/store/thread_exclusions_test.go b/internal/store/thread_exclusions_test.go new file mode 100644 index 00000000..9107a044 --- /dev/null +++ b/internal/store/thread_exclusions_test.go @@ -0,0 +1,384 @@ +package store + +import ( + "context" + "errors" + "fmt" + "path/filepath" + "strings" + "testing" +) + +func exclusionFixture(t *testing.T) *Store { + t.Helper() + s, err := Open(context.Background(), filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { s.Close() }) + _, err = s.DB().Exec(` +INSERT INTO repositories(id,owner,name,full_name,github_repo_id,raw_json,updated_at) VALUES(1,'fixture','repo','fixture/repo','100','{}','2026-01-01'); +INSERT INTO threads(id,repo_id,github_id,number,kind,state,title,body,html_url,labels_json,assignees_json,raw_json,content_hash,updated_at) VALUES + (1,1,'gh10',10,'pull_request','closed','remove','OWNER_REMOVAL_SENTINEL','https://github.com/fixture/repo/pull/10','[]','[]','{"id":1010,"node_id":"node10","_graphql":{"id":"node10"}}','hash10','2026-01-01'), + (2,1,'gh20',20,'pull_request','open','keep','UNCHANGED_BODY','https://github.com/fixture/repo/pull/20','[]','[]','{"node_id":"node20"}','hash20','2026-01-01'); +INSERT INTO comments(id,thread_id,github_id,comment_type,body,raw_json) VALUES + (1,1,'comment10','issue_comment','removed comment','{"node_id":"comment-node10"}'), + (2,2,'comment20','issue_comment','kept comment','{"node_id":"comment-node20"}'); +INSERT INTO comment_revisions(id,comment_id,body,raw_json,recorded_at) VALUES + (1,1,'removed old comment','{"node_id":"comment-node10"}','2026-01-01'), + (2,2,'kept old comment','{"node_id":"comment-node20"}','2026-01-01'); +INSERT INTO documents(id,thread_id,title,body,raw_text,dedupe_text,updated_at) VALUES + (1,1,'remove','OWNER_REMOVAL_SENTINEL','OWNER_REMOVAL_SENTINEL','remove','2026-01-01'), + (2,2,'keep','UNCHANGED_BODY','UNCHANGED_BODY','keep','2026-01-01'); +INSERT INTO pull_request_details(thread_id,repo_id,number,raw_json,fetched_at,updated_at) VALUES(1,1,10,'{"fixture":"removed"}','2026-01-01','2026-01-01'),(2,1,20,'{"fixture":"kept"}','2026-01-01','2026-01-01'); +INSERT INTO actor_identity_evidence VALUES('node10','shared-actor','fixture','User','2026-01-01','{}'),('comment-node10','shared-actor','fixture','User','2026-01-01','{}'); +INSERT INTO actor_profiles(node_id,login,actor_type,observed_at,raw_json) VALUES('shared-actor','fixture','User','2026-01-01','{}'); +INSERT INTO analytics_pending_nodes VALUES('node10','identity'),('comment-node10','identity'),('shared-actor','profile'); +`) + if err != nil { + t.Fatal(err) + } + ctx := context.Background() + at := "2026-01-01T00:00:00Z" + if err = s.SaveAnalyticsCoverage(ctx, "fixture/repo", at, 0, 2); err != nil { + t.Fatal(err) + } + for _, n := range []int{10, 20} { + if err = s.RecordAnalyticsAttempt(ctx, AnalyticsAttempt{Repository: "fixture/repo", Number: n, Operation: "review_state", StartedAt: at, FinishedAt: at, Status: "failed", ErrorClass: "partial_response", Evidence: []byte(`{"numbers":[10,20],"body_evidence":{"sha256":"fixture-hash","bytes":10}}`)}); err != nil { + t.Fatal(err) + } + } + if err = s.SaveReviewStateCoverage(ctx, "fixture/repo", ReviewStateRecovery{Done: true, Cursor: 2, Ceiling: 2, Scanned: 2, Queued: 2}); err != nil { + t.Fatal(err) + } + return s +} + +func TestOwnerPurgeRemovesExactClosureAndPreventsReintroduction(t *testing.T) { + s := exclusionFixture(t) + ctx := context.Background() + plan, err := s.PlanThreadPurge(ctx, "fixture/repo", []int{10}) + if err != nil { + t.Fatal(err) + } + if plan.Applied || len(plan.Targets) != 1 || len(plan.Targets[0].Nodes) != 2 || plan.Counts["comments"] != 1 || plan.Counts["comment_revisions"] != 1 { + t.Fatalf("plan=%+v", plan) + } + var retained string + s.DB().QueryRow("SELECT raw_json FROM pull_request_details WHERE thread_id=2").Scan(&retained) + result, err := s.PurgeThreads(ctx, "fixture/repo", []int{10}, "owner-request-fixture") + if err != nil || !result.Applied { + t.Fatalf("%+v %v", result, err) + } + for _, table := range []string{"threads", "comments", "comment_revisions", "documents", "pull_request_details"} { + var n int + if err = s.DB().QueryRow("SELECT count(*) FROM " + table).Scan(&n); err != nil || n != 1 { + t.Fatalf("%s: %d %v", table, n, err) + } + } + var after string + s.DB().QueryRow("SELECT raw_json FROM pull_request_details WHERE thread_id=2").Scan(&after) + if after != retained { + t.Fatal("peer evidence changed") + } + var fts int + if err = s.DB().QueryRow("SELECT count(*) FROM documents_fts WHERE documents_fts MATCH 'OWNER_REMOVAL_SENTINEL'").Scan(&fts); err != nil || fts != 0 { + t.Fatalf("FTS leaked removed document: %d %v", fts, err) + } + var profiles int + s.DB().QueryRow("SELECT count(*) FROM actor_profiles").Scan(&profiles) + if profiles != 1 { + t.Fatal("shared actor removed") + } + var remaining string + s.DB().QueryRow("SELECT evidence_json FROM analytics_fetch_attempts WHERE number=20").Scan(&remaining) + if !strings.Contains(remaining, `"numbers":[10,20]`) { + t.Fatal("unrelated mixed receipt changed") + } + if excluded, err := s.ThreadExcluded(ctx, "FIXTURE/REPO", 10); err != nil || !excluded { + t.Fatal("missing permanent policy") + } + if excluded, err := s.NodeExcluded(ctx, "1010"); err != nil || excluded { + t.Fatal("REST database ID mistaken for GraphQL node", err) + } + if _, err = s.UpsertThread(ctx, Thread{RepoID: 1, Number: 10, Kind: "pull_request", GitHubID: "gh10"}); !errors.Is(err, ErrThreadExcluded) { + t.Fatalf("reinsertion allowed: %v", err) + } + if err = s.SaveActorEvidence(ctx, []map[string]any{{"id": "node10", "author": map[string]any{"id": "shared-actor"}}}, "2026-01-02T00:00:00Z"); err != nil { + t.Fatal(err) + } + if err = s.RecordAnalyticsAttempt(ctx, AnalyticsAttempt{Repository: "fixture/repo", Number: 10, Operation: "review_state", StartedAt: "2026-01-02T00:00:00Z", FinishedAt: "2026-01-02T00:00:00Z", Status: "success", Evidence: []byte(`{}`)}); err != nil { + t.Fatal(err) + } + for _, q := range []string{"SELECT count(*) FROM analytics_fetch_attempts WHERE number=10", "SELECT count(*) FROM analytics_retries WHERE number=10", "SELECT count(*) FROM actor_identity_evidence WHERE node_id='node10'", "SELECT count(*) FROM analytics_pending_nodes WHERE node_id='node10'"} { + var n int + s.DB().QueryRow(q).Scan(&n) + if n != 0 { + t.Fatal("excluded evidence recreated", q, n) + } + } + if _, err = s.DB().Exec("INSERT INTO analytics_pending_nodes VALUES('node10','identity')"); err == nil { + t.Fatal("SQL guard bypass") + } + // The remaining ordinary retry can resolve, but owner removal is not provider completeness. + s.DB().Exec("UPDATE analytics_retries SET resolved_at='2026-01-02T00:00:00Z' WHERE number=20") + if err = s.SaveReviewStateCoverage(ctx, "fixture/repo", ReviewStateRecovery{Done: true}); err != nil { + t.Fatal(err) + } + var complete, pending, excluded, core int + s.DB().QueryRow("SELECT complete,pending_items,owner_excluded_items FROM analytics_review_state_coverage").Scan(&complete, &pending, &excluded) + s.DB().QueryRow("SELECT complete FROM analytics_coverage").Scan(&core) + if complete != 0 || pending != 0 || excluded != 1 || core != 1 { + t.Fatalf("false recovery: review=%d pending=%d excluded=%d core=%d", complete, pending, excluded, core) + } + again, err := s.PurgeThreads(ctx, "fixture/repo", []int{10}, "same-request") + if err != nil || !again.Targets[0].AlreadyExcluded { + t.Fatalf("idempotency: %+v %v", again, err) + } + var original string + s.DB().QueryRow("SELECT request_id FROM thread_exclusions").Scan(&original) + if original != "owner-request-fixture" { + t.Fatal("original owner receipt rewritten") + } +} + +func TestOwnerPurgeRejectsTailAndRollsBack(t *testing.T) { + s := exclusionFixture(t) + ctx := context.Background() + if _, err := s.PurgeThreads(ctx, "fixture/repo", []int{20}, "tail"); err == nil || !strings.Contains(err.Error(), "ID reuse") { + t.Fatalf("tail admitted: %v", err) + } + var n int + s.DB().QueryRow("SELECT count(*) FROM thread_exclusions").Scan(&n) + if n != 0 { + t.Fatal("tail failure wrote policy") + } + id, err := s.UpsertThread(ctx, Thread{RepoID: 1, Number: 30, Kind: "issue", GitHubID: "gh30", Title: "new", RawJSON: "{}", LabelsJSON: "[]", AssigneesJSON: "[]", UpdatedAt: "2026-01-02T00:00:00Z"}) + if err != nil || id <= 2 { + t.Fatalf("native ID reused: %d %v", id, err) + } + if _, err = s.PlanThreadPurge(ctx, "fixture/repo", []int{20}); err == nil || !strings.Contains(err.Error(), "ID reuse") { + t.Fatal("child tail admitted", err) + } + _, err = s.DB().Exec(`CREATE TRIGGER deny_fixture_delete BEFORE DELETE ON comments WHEN OLD.id=1 BEGIN SELECT RAISE(ABORT,'fixture interruption'); END`) + if err != nil { + t.Fatal(err) + } + if _, err = s.PurgeThreads(ctx, "fixture/repo", []int{10}, "rollback"); err == nil { + t.Fatal("failure accepted") + } + for _, q := range []string{"SELECT count(*) FROM threads WHERE id=1", "SELECT count(*) FROM actor_identity_evidence WHERE node_id='node10'", "SELECT count(*) FROM analytics_retries WHERE number=10"} { + s.DB().QueryRow(q).Scan(&n) + if n != 1 { + t.Fatal("partial purge", q, n) + } + } + s.DB().QueryRow("SELECT count(*) FROM thread_exclusions").Scan(&n) + if n != 0 { + t.Fatal("rollback left exclusion") + } + for _, numbers := range [][]int{nil, {0}, {10, 10}, {999}} { + if _, err = s.PlanThreadPurge(ctx, "fixture/repo", numbers); err == nil { + t.Fatal("invalid selection accepted", numbers) + } + } + if _, err = s.PurgeThreads(ctx, "fixture/repo", []int{10}, ""); err == nil { + t.Fatal("missing owner request accepted") + } +} + +func TestOwnerExclusionV15MigrationIsAdditive(t *testing.T) { + s := exclusionFixture(t) + path := s.Path() + _, err := s.DB().Exec(`DROP TABLE thread_excluded_nodes;DROP TABLE thread_exclusions;ALTER TABLE analytics_review_state_coverage DROP COLUMN owner_excluded_items;PRAGMA user_version=15;`) + if err != nil { + t.Fatal(err) + } + if err = s.markObservationSchemaConverged(context.Background()); err != nil { + t.Fatal(err) + } + s.Close() + s, err = Open(context.Background(), path) + if err != nil { + t.Fatal(err) + } + defer s.Close() + var version, count int + s.DB().QueryRow("pragma user_version").Scan(&version) + s.DB().QueryRow("select count(*) from comments").Scan(&count) + if version != 16 || count != 2 { + t.Fatalf("migration altered data: %d %d", version, count) + } +} + +func TestOwnerPurgeRefusesBlobBackedTargetsWithoutMutation(t *testing.T) { + for _, mode := range []string{"shared", "unshared", "external", "tail"} { + t.Run(mode, func(t *testing.T) { + s := exclusionFixture(t) + ctx := context.Background() + _, err := s.DB().Exec(`INSERT INTO blobs(id,sha256,media_type,size_bytes,storage_kind,inline_text,created_at) VALUES + (1,'one','application/json',20,'inline','removed-blob','2026-01-01'), + (2,'two','application/json',20,'inline','peer-blob','2026-01-01'); + UPDATE comments SET raw_json_blob_id=1 WHERE id=1;`) + if err != nil { + t.Fatal(err) + } + switch mode { + case "shared": + _, err = s.DB().Exec("UPDATE comments SET raw_json_blob_id=1 WHERE id=2") + case "external": + _, err = s.DB().Exec("UPDATE blobs SET storage_kind='file',storage_path='/fixture/private' WHERE id=1") + case "tail": + _, err = s.DB().Exec("DELETE FROM blobs WHERE id=2") + } + if err != nil { + t.Fatal(err) + } + result, err := s.PurgeThreads(ctx, "fixture/repo", []int{10}, "blob-fixture") + var count, threads int + s.DB().QueryRow("SELECT count(*) FROM blobs WHERE id=1").Scan(&count) + s.DB().QueryRow("SELECT count(*) FROM threads WHERE id=1").Scan(&threads) + if err == nil || result.Applied || !strings.Contains(err.Error(), "blob-backed") || count != 1 || threads != 1 { + t.Fatalf("unsupported blob target was altered: %+v %v count=%d threads=%d", result, err, count, threads) + } + var policy int + s.DB().QueryRow("SELECT count(*) FROM thread_exclusions").Scan(&policy) + if policy != 0 { + t.Fatal("refusal installed an exclusion") + } + + }) + } +} + +func TestOwnerPurgePortableMirrorKeepsSparseSchemaAndPeers(t *testing.T) { + for _, sanitized := range []bool{true} { + t.Run(fmt.Sprint(sanitized), func(t *testing.T) { + s := exclusionFixture(t) + ctx := context.Background() + path := s.Path() + _, err := s.PrunePortablePayloads(ctx, PortablePruneOptions{BodyChars: 32, RetainSanitizedPayloadColumns: sanitized}) + if err != nil { + t.Fatal(err) + } + var before string + if err = s.DB().QueryRow("SELECT title||body_excerpt||content_hash FROM threads WHERE id=2").Scan(&before); err != nil { + t.Fatal(err) + } + s.Close() + s, err = OpenThreadPurgeMirror(ctx, path) + if err != nil { + t.Fatal(err) + } + defer s.Close() + result, err := s.PurgeThreads(ctx, "fixture/repo", []int{10}, "mirror-fixture") + if err != nil || !result.Applied { + t.Fatalf("mirror purge: %+v %v", result, err) + } + var after string + var version, n int + s.DB().QueryRow("SELECT title||body_excerpt||content_hash FROM threads WHERE id=2").Scan(&after) + s.DB().QueryRow("PRAGMA user_version").Scan(&version) + s.DB().QueryRow("SELECT count(*) FROM comments WHERE thread_id=1").Scan(&n) + if version != portableSchemaVersion || after != before || n != 0 || s.hasTable(ctx, "documents") { + t.Fatalf("mirror migration or leak: %d %q %q %d", version, before, after, n) + } + if _, err = s.DB().Exec("UPDATE threads SET number=10 WHERE id=2"); err == nil { + t.Fatal("mirror resurrection allowed") + } + }) + } +} + +func TestOwnerPurgePublicationCannotDiscardPolicy(t *testing.T) { + s := exclusionFixture(t) + ctx := context.Background() + if _, err := s.PurgeThreads(ctx, "fixture/repo", []int{10}, "private-owner-reference"); err != nil { + t.Fatal(err) + } + if _, err := s.PrunePortablePayloads(ctx, PortablePruneOptions{BodyChars: 1}); err == nil || !strings.Contains(err.Error(), "local owner exclusions") { + t.Fatal("private policy publication admitted", err) + } + var body string + if err := s.DB().QueryRow("SELECT body FROM threads WHERE id=2").Scan(&body); err != nil || body != "UNCHANGED_BODY" { + t.Fatal("refusal modified retained data", err) + } + if _, err := OpenThreadPurgeMirror(ctx, s.Path()); err == nil { + t.Fatal("native archive accepted as portable mirror") + } +} + +func TestOwnerPurgeCanonicalRepositoryMatchesCaseVariantReceipts(t *testing.T) { + s := exclusionFixture(t) + ctx := context.Background() + _, err := s.DB().Exec("UPDATE repositories SET full_name='Fixture/Repo';UPDATE analytics_fetch_attempts SET repository='Fixture/Repo' WHERE number=10;UPDATE analytics_retries SET repository='Fixture/Repo' WHERE number=10;UPDATE analytics_review_state_coverage SET repository='Fixture/Repo'") + if err != nil { + t.Fatal(err) + } + result, err := s.PurgeThreads(ctx, "FIXTURE/REPO", []int{10}, "case-fixture") + if err != nil || result.Counts["analytics_fetch_attempts"] != 1 || result.Counts["analytics_retries"] != 1 { + t.Fatalf("variant not planned: %+v %v", result, err) + } + for _, table := range []string{"analytics_fetch_attempts", "analytics_retries"} { + var n int + if err = s.DB().QueryRow("SELECT count(*) FROM " + table + " WHERE number=10").Scan(&n); err != nil || n != 0 { + t.Fatal("variant retained", table, n, err) + } + } + var complete, excluded int + s.DB().QueryRow("SELECT complete,owner_excluded_items FROM analytics_review_state_coverage WHERE repository='Fixture/Repo'").Scan(&complete, &excluded) + if complete != 0 || excluded != 1 { + t.Fatal("variant coverage falsely complete", complete, excluded) + } + if err = s.RecordAnalyticsAttempt(ctx, AnalyticsAttempt{Repository: "Fixture/Repo", Number: 10, Operation: "review_state", StartedAt: "2026-01-01T00:00:00Z", FinishedAt: "2026-01-01T00:00:01Z", Status: "failed", ErrorClass: "partial_response", Evidence: []byte(`{}`)}); err != nil { + t.Fatal(err) + } + var n int + s.DB().QueryRow("SELECT count(*) FROM analytics_retries WHERE number=10").Scan(&n) + if n != 0 { + t.Fatal("variant requeued") + } +} + +func TestOwnerPurgeRefusesWorkflowEvidenceAndKeepsUnrelatedReservations(t *testing.T) { + for _, mode := range []string{"run", "current-head", "historical-head", "unrelated"} { + t.Run(mode, func(t *testing.T) { + s := exclusionFixture(t) + ctx := context.Background() + var err error + switch mode { + case "run": + _, err = s.DB().Exec("INSERT INTO github_workflow_runs(repo_id,run_id,head_sha,raw_json,fetched_at) VALUES(1,1,'older-sha','{}','2026-01-01')") + case "current-head": + _, err = s.DB().Exec("UPDATE pull_request_details SET head_sha='linked-sha' WHERE thread_id=1;INSERT INTO workflow_run_observation_reservations VALUES(1,'linked-sha','2026-01-01',1)") + case "historical-head": + _, err = s.DB().Exec(`UPDATE threads SET raw_json='{"node_id":"node10","_graphql":{"headRefOid":"older-sha"}}' WHERE id=1;INSERT INTO workflow_run_observation_reservations VALUES(1,'older-sha','2026-01-01',1)`) + case "unrelated": + _, err = s.DB().Exec("UPDATE pull_request_details SET head_sha='peer-sha' WHERE thread_id=2;INSERT INTO workflow_run_observation_reservations VALUES(1,'peer-sha','2026-01-01',1)") + } + if err != nil { + t.Fatal(err) + } + result, err := s.PurgeThreads(ctx, "fixture/repo", []int{10}, "workflow-fixture") + var n int + s.DB().QueryRow("SELECT count(*) FROM threads WHERE id=1").Scan(&n) + if mode == "unrelated" { + if err != nil || !result.Applied || n != 0 { + t.Fatalf("unrelated reservation blocked target: %+v %v", result, err) + } + s.DB().QueryRow("SELECT count(*) FROM workflow_run_observation_reservations WHERE head_sha='peer-sha'").Scan(&n) + if n != 1 { + t.Fatal("peer reservation lost") + } + } else { + if err == nil || !strings.Contains(err.Error(), "workflow") || result.Applied || n != 1 { + t.Fatalf("workflow evidence ignored: %+v %v count=%d", result, err, n) + } + s.DB().QueryRow("SELECT count(*) FROM thread_exclusions").Scan(&n) + if n != 0 { + t.Fatal("refusal wrote policy") + } + } + }) + } +} diff --git a/internal/store/threads.go b/internal/store/threads.go index 0a3dd744..45375ff5 100644 --- a/internal/store/threads.go +++ b/internal/store/threads.go @@ -81,6 +81,13 @@ func (s *Store) UpsertThreadObservation(ctx context.Context, thread Thread, opti } func (s *Store) upsertThreadObservation(ctx context.Context, thread Thread, options UpsertThreadOptions) (UpsertThreadResult, error) { + var excluded bool + if err := s.q().QueryRowContext(ctx, `SELECT EXISTS(SELECT 1 FROM thread_exclusions e JOIN repositories r ON lower(r.full_name)=e.repository WHERE r.id=? AND e.number=?)`, thread.RepoID, thread.Number).Scan(&excluded); err != nil { + return UpsertThreadResult{}, err + } + if excluded { + return UpsertThreadResult{}, ErrThreadExcluded + } if options.ObservationSequence <= 0 { sequence, err := s.NextThreadObservationSequence(ctx, thread.UpdatedAt) if err != nil { @@ -254,6 +261,9 @@ func (s *Store) upsertThreadObservation(ctx context.Context, thread Thread, opti if err != nil { return UpsertThreadResult{}, fmt.Errorf("upsert thread: %w", err) } + if err := s.queueAnalyticsActor(ctx, thread.RawJSON); err != nil { + return UpsertThreadResult{}, err + } if s.portableThreadBodyMetadata { if _, err := s.q().ExecContext(ctx, ` update threads diff --git a/internal/syncer/graphql_history_test.go b/internal/syncer/graphql_history_test.go index d1ce8928..4ac01adf 100644 --- a/internal/syncer/graphql_history_test.go +++ b/internal/syncer/graphql_history_test.go @@ -14,11 +14,15 @@ import ( type historyFixtureClient struct { *gh.Client - batch gh.HistoryBatch - err error + batch gh.HistoryBatch + err error + beforeFetch func() } func (f historyFixtureClient) FetchGraphQLHistory(context.Context, string, string, []int, gh.Reporter) (gh.HistoryBatch, error) { + if f.beforeFetch != nil { + f.beforeFetch() + } return f.batch, f.err } func TestGraphQLHistoryUsesNativeTransactionsAndPreservesLegacyIdentity(t *testing.T) { @@ -59,6 +63,7 @@ func TestGraphQLHistoryUsesNativeTransactionsAndPreservesLegacyIdentity(t *testi defer server.Close() client := historyFixtureClient{Client: gh.New(gh.Options{BaseURL: server.URL}), batch: gh.HistoryBatch{Repository: rawRepo, Items: []gh.HistoryItem{{Thread: row, Pull: pull, Comments: comments, Reviews: reviews, ReviewComments: inline}}}} options.GraphQLHistory = true + client.batch.Items[0].ReviewThreads = []map[string]any{{"id": "RT_fixture", "isResolved": true, "isOutdated": false, "comments": map[string]any{"nodes": []any{map[string]any{"id": "inline-node", "body": "reply", "replyTo": map[string]any{"id": "parent-node"}}}}}} stats, err := New(client, st).Sync(ctx, options) if err != nil { t.Fatal(err) @@ -73,6 +78,16 @@ func TestGraphQLHistoryUsesNativeTransactionsAndPreservesLegacyIdentity(t *testi if stats.ThreadsSynced != 1 || stats.PRDetailsSynced != 1 || stats.CommentsSynced == 0 { t.Fatalf("stats %+v", stats) } + if stats.ReviewThreadsSynced != 1 { + t.Fatalf("review states missing: %+v", stats) + } + var resolved int + var members string + st.DB().QueryRow("SELECT is_resolved FROM pull_request_review_threads WHERE review_thread_id='RT_fixture'").Scan(&resolved) + st.DB().QueryRow("SELECT review_thread_ids_json FROM pull_request_review_thread_syncs WHERE thread_id=?", after[0].ID).Scan(&members) + if resolved != 1 || members != `["RT_fixture"]` { + t.Fatalf("review projection %d %s", resolved, members) + } if _, err := New(client, st).Sync(ctx, options); err != nil { t.Fatal(err) } @@ -82,8 +97,27 @@ func TestGraphQLHistoryUsesNativeTransactionsAndPreservesLegacyIdentity(t *testi t.Fatal("failed batch persisted") } assertTableRowCount(t, st, "threads", 1) + var failed int + if err := st.DB().QueryRow("SELECT count(*) FROM analytics_fetch_attempts WHERE status='failed'").Scan(&failed); err != nil || failed != 1 { + t.Fatalf("missing durable fetch failure: %d %v", failed, err) + } options.IncludePRDetails = true if _, err := New(client, st).Sync(ctx, options); err == nil { t.Fatal("unsupported hydration accepted") } } + +func TestGraphQLCancellationDoesNotHideReceiptPersistenceFailure(t *testing.T) { + ctx := context.Background() + st, err := store.Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + // Close during the fetch, after the owner-exclusion preflight. This tests + // a cancelled provider request whose durable receipt cannot be persisted. + client := historyFixtureClient{err: context.DeadlineExceeded, beforeFetch: func() { st.Close() }} + _, err = New(client, st).Sync(ctx, Options{Owner: "fixture", Repo: "repo", Numbers: []int{1}, State: "all", GraphQLHistory: true, IncludeComments: true, IncludePRMetadata: true, ReceiptOperation: "review_state"}) + if !errors.Is(err, context.DeadlineExceeded) || !errors.Is(err, errAnalyticsReceipt) { + t.Fatalf("missing distinguishable receipt failure: %v", err) + } +} diff --git a/internal/syncer/review_state.go b/internal/syncer/review_state.go new file mode 100644 index 00000000..4a118c96 --- /dev/null +++ b/internal/syncer/review_state.go @@ -0,0 +1,75 @@ +package syncer + +import ( + "context" + "encoding/json" + "fmt" + "time" + + gh "github.com/openclaw/gitcrawl/internal/github" + "github.com/openclaw/gitcrawl/internal/store" +) + +func (s *Syncer) syncReviewState(ctx context.Context, options Options, started string) (Stats, error) { + stats := Stats{Repository: options.Owner + "/" + options.Repo, Numbers: uniquePositiveNumbers(options.Numbers), StartedAt: started, ReviewStateOnly: true} + client, ok := s.client.(interface { + FetchGraphQLReviewState(context.Context, string, string, []int, gh.Reporter) ([]gh.ReviewStateItem, error) + }) + if !ok { + return stats, fmt.Errorf("client does not support targeted review state") + } + // Reserve an observation sequence before fetching, as in normal capture. + sequence, err := s.store.NextThreadObservationSequence(ctx, started) + if err != nil { + return stats, err + } + fetched := time.Now() + items, err := client.FetchGraphQLReviewState(ctx, options.Owner, options.Repo, stats.Numbers, options.Reporter) + stats.FetchMillis = time.Since(fetched).Milliseconds() + if err != nil { + return stats, err + } + if len(items) != len(stats.Numbers) { + return stats, fmt.Errorf("incomplete review-state batch") + } + persist := time.Now() + err = s.store.WithTx(ctx, func(tx *store.Store) error { + applied, threads := 0, 0 + for i, item := range items { + if item.Number != stats.Numbers[i] { + return fmt.Errorf("review-state selection mismatch") + } + t, err := tx.ReviewStateParent(ctx, stats.Repository, item.Number, item.RepositoryID, item.RepositoryNodeID) + if err != nil { + return err + } + var raw map[string]any + if err := json.Unmarshal([]byte(t.RawJSON), &raw); err != nil { + return err + } + if known := stringValue(raw["node_id"]); known != "" && known != item.NodeID { + return fmt.Errorf("review-state archived node identity mismatch") + } + reserved, err := tx.ReserveThreadChildObservation(ctx, t.ID, store.ThreadChildReviewThreads, item.UpdatedAt, sequence) + if err != nil { + return err + } + if !reserved { + continue + } + count, err := s.persistPullReviewThreads(ctx, tx, t, item.Threads, started) + if err != nil { + return err + } + applied++ + threads += count + } + stats.ThreadsSynced = applied + stats.PullRequestsSynced = applied + stats.ReviewThreadsSynced = threads + return nil + }) + stats.PersistMillis = time.Since(persist).Milliseconds() + stats.FinishedAt = s.now().Format(time.RFC3339Nano) + return stats, err +} diff --git a/internal/syncer/review_state_test.go b/internal/syncer/review_state_test.go new file mode 100644 index 00000000..8c207fce --- /dev/null +++ b/internal/syncer/review_state_test.go @@ -0,0 +1,154 @@ +package syncer + +import ( + "context" + "encoding/json" + "errors" + gh "github.com/openclaw/gitcrawl/internal/github" + "github.com/openclaw/gitcrawl/internal/store" + "path/filepath" + "testing" + "time" +) + +type reviewOnlyFixture struct { + *gh.Client + items []gh.ReviewStateItem + err error +} + +func (f reviewOnlyFixture) FetchGraphQLReviewState(context.Context, string, string, []int, gh.Reporter) ([]gh.ReviewStateItem, error) { + return f.items, f.err +} +func TestReviewOnlyNeverOverwritesCanonicalContentHistoryOrVectors(t *testing.T) { + ctx := context.Background() + s, err := store.Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + legacy := &metadataGitHub{} + opts := Options{Owner: "openclaw", Repo: "gitcrawl", Numbers: []int{8}, State: "all", IncludeComments: true, IncludePRMetadata: true} + if _, err = New(legacy, s).Sync(ctx, opts); err != nil { + t.Fatal(err) + } + repo, err := s.RepositoryByFullName(ctx, "openclaw/gitcrawl") + if err != nil { + t.Fatal(err) + } + heads, err := s.ListThreads(ctx, repo.ID, true) + if err != nil { + t.Fatal(err) + } + head := heads[0] + if _, err = s.DB().Exec("update threads set raw_json=json_set(raw_json,'$.node_id','PR8') where id=?", head.ID); err != nil { + t.Fatal(err) + } + if _, err = s.DB().Exec("update repositories set raw_json=json_set(raw_json,'$.node_id','R1') where id=?", repo.ID); err != nil { + t.Fatal(err) + } + s.DB().QueryRow("select raw_json from threads where id=?", head.ID).Scan(&head.RawJSON) + s.DB().QueryRow("select raw_json from repositories where id=?", repo.ID).Scan(&repo.RawJSON) + if err = s.UpsertThreadVector(ctx, store.ThreadVector{ThreadID: head.ID, Basis: "title_original", Model: "fixture", Dimensions: 2, ContentHash: head.ContentHash, Vector: []float64{1, 2}, CreatedAt: "2026-01-01T00:00:00Z", UpdatedAt: "2026-01-01T00:00:00Z"}); err != nil { + t.Fatal(err) + } + snapshot := func() string { + tables := []string{"threads", "comments", "comment_revisions", "thread_revisions", "thread_fingerprints", "thread_vectors"} + out := map[string][][]any{} + for _, table := range tables { + rows, e := s.DB().Query("select * from " + table) + if e != nil { + t.Fatal(e) + } + cols, _ := rows.Columns() + for rows.Next() { + v := make([]any, len(cols)) + p := make([]any, len(v)) + for i := range v { + p[i] = &v[i] + } + if e = rows.Scan(p...); e != nil { + t.Fatal(e) + } + out[table] = append(out[table], v) + } + rows.Close() + } + b, _ := json.Marshal(out) + return string(b) + } + before := snapshot() + var raw map[string]any + json.Unmarshal([]byte(head.RawJSON), &raw) + nodeID := stringValue(raw["node_id"]) + if nodeID == "" { + nodeID = "PR8" + } + // Fixture repo supplies its retained database identity to the targeted path. + var repoRaw map[string]any + json.Unmarshal([]byte(repo.RawJSON), &repoRaw) + item := gh.ReviewStateItem{Number: 8, NodeID: nodeID, RepositoryID: repo.GitHubRepoID, RepositoryNodeID: stringValue(repoRaw["node_id"]), UpdatedAt: time.Now().UTC().Format(time.RFC3339Nano), Threads: []map[string]any{{"id": "RT1", "isResolved": true, "isOutdated": true, "path": "file.go", "line": 3, "comments": map[string]any{"nodes": []any{map[string]any{"id": "RC1", "body": "inline body", "url": "https://github.com/openclaw/gitcrawl/pull/8#r1", "createdAt": "2026-01-01T00:00:00Z", "updatedAt": "2026-01-02T00:00:00Z", "author": map[string]any{"login": "fixture", "__typename": "User"}, "replyTo": map[string]any{"id": "RC0"}}}}}}} + f := reviewOnlyFixture{items: []gh.ReviewStateItem{item}} + opts.GraphQLHistory = true + opts.ReviewStateOnly = true + opts.ReceiptOperation = "review_state" + stats, err := New(f, s).Sync(ctx, opts) + if err != nil { + t.Fatal(err) + } + if !stats.ReviewStateOnly || stats.CommentsSynced != 0 || stats.ReviewThreadsSynced != 1 { + t.Fatalf("unexpected stats %+v", stats) + } + if before != snapshot() { + t.Fatal("canonical content/history/vector changed") + } + var membership, body, login, comments, url, created, updated, authorType string + var resolved, outdated int + s.DB().QueryRow("select review_thread_ids_json from pull_request_review_thread_syncs where thread_id=?", head.ID).Scan(&membership) + s.DB().QueryRow("select first_comment_body,first_author_login,comments_json,is_resolved,is_outdated,first_comment_url,first_comment_created_at,first_comment_updated_at,first_author_type from pull_request_review_threads where thread_id=?", head.ID).Scan(&body, &login, &comments, &resolved, &outdated, &url, &created, &updated, &authorType) + if membership != `["RT1"]` || body != "inline body" || login != "fixture" || resolved != 1 || outdated != 1 { + t.Fatal("review contract lost", membership, body, login) + } + if url != "https://github.com/openclaw/gitcrawl/pull/8#r1" || created != "2026-01-01T00:00:00Z" || updated != "2026-01-02T00:00:00Z" || authorType != "User" { + t.Fatal("first-comment metadata lost") + } + saved := membership + comments + for _, failure := range []error{errors.New("partial response"), context.Canceled} { + f.err = failure + if _, err = New(f, s).Sync(ctx, opts); err == nil { + t.Fatal("failure accepted") + } + s.DB().QueryRow("select review_thread_ids_json from pull_request_review_thread_syncs where thread_id=?", head.ID).Scan(&membership) + s.DB().QueryRow("select comments_json from pull_request_review_threads where thread_id=?", head.ID).Scan(&comments) + if saved != membership+comments || before != snapshot() { + t.Fatal("failed observation overwrote retained data") + } + } + f.err = nil + f.items[0].RepositoryID = "wrong" + if _, err = New(f, s).Sync(ctx, opts); err == nil { + t.Fatal("wrong repository accepted") + } + f.items[0] = item + f.items[0].NodeID = "wrong" + if stringValue(raw["node_id"]) != "" { + if _, err = New(f, s).Sync(ctx, opts); err == nil { + t.Fatal("wrong node accepted") + } + } + // A later identity failure rolls back earlier state/history in the batch. + f.items = []gh.ReviewStateItem{item, item} + f.items[1].Number = 9 + f.items[0].Threads[0]["isResolved"] = false + opts.Numbers = []int{8, 9} + var revisionsBefore, revisionsAfter int + s.DB().QueryRow("select count(*) from pull_request_review_thread_revisions").Scan(&revisionsBefore) + if _, err = New(f, s).Sync(ctx, opts); err == nil { + t.Fatal("missing parent accepted") + } + s.DB().QueryRow("select count(*) from pull_request_review_thread_revisions").Scan(&revisionsAfter) + s.DB().QueryRow("select is_resolved from pull_request_review_threads where thread_id=?", head.ID).Scan(&resolved) + if resolved != 1 || revisionsAfter != revisionsBefore || before != snapshot() { + t.Fatal("partial batch publication or canonical mutation") + } +} diff --git a/internal/syncer/syncer.go b/internal/syncer/syncer.go index 6ab6c239..014db423 100644 --- a/internal/syncer/syncer.go +++ b/internal/syncer/syncer.go @@ -43,7 +43,11 @@ type Syncer struct { } type Options struct { - GraphQLHistory bool + GraphQLHistory bool + ReviewStateOnly bool + // ReceiptOperation separates targeted review-state recovery from verified + // core traversal; both still fetch and validate complete conversations. + ReceiptOperation string Owner string Repo string State string @@ -60,6 +64,9 @@ type Options struct { } type Stats struct { + ReviewStateOnly bool `json:"review_state_only,omitempty"` + FetchMillis int64 `json:"fetch_ms,omitempty"` + PersistMillis int64 `json:"persist_ms,omitempty"` Repository string `json:"repository"` ThreadsSynced int `json:"threads_synced"` IssuesSynced int `json:"issues_synced"` @@ -121,7 +128,12 @@ func New(client GitHubClient, st *store.Store) *Syncer { } } -func (s *Syncer) Sync(ctx context.Context, options Options) (Stats, error) { +var errAnalyticsReceipt = errors.New("persist GraphQL attempt") + +func (s *Syncer) Sync(ctx context.Context, options Options) (result Stats, resultErr error) { + if options.ReviewStateOnly && !options.GraphQLHistory { + return Stats{}, fmt.Errorf("review-state-only requires GraphQL history transport") + } startedAt := s.now() started := startedAt.Format(time.RFC3339Nano) if err := reportSyncProgress(options.Progress, SyncProgress{ @@ -143,6 +155,48 @@ func (s *Syncer) Sync(ctx context.Context, options Options) (Stats, error) { if len(options.Numbers) == 0 || !options.IncludeComments || !options.IncludePRMetadata || options.IncludePRDetails || since != "" || options.Limit != 0 || state != "all" { return Stats{}, fmt.Errorf("--graphql-history requires --numbers, --state all, --include-comments and --with pr-metadata; since/limit/pr-details are unsupported") } + operation := options.ReceiptOperation + if operation == "" { + operation = "graphql_history" + } + if operation != "graphql_history" && operation != "review_state" { + return Stats{}, fmt.Errorf("unsupported GraphQL receipt operation") + } + if options.ReviewStateOnly && operation != "review_state" { + return Stats{}, fmt.Errorf("review-state-only fetch requires review recovery operation") + } + allowed, err := s.store.FilterExcludedNumbers(ctx, options.Owner+"/"+options.Repo, uniquePositiveNumbers(options.Numbers)) + if err != nil { + return Stats{}, err + } + options.Numbers = allowed + if len(allowed) == 0 { + return Stats{Repository: options.Owner + "/" + options.Repo, StartedAt: started, FinishedAt: s.now().Format(time.RFC3339Nano)}, nil + } + // Fetch/validation failures happen before conversation transactions and + // were previously invisible to durable run tables. Keep a receipt even + // when the request is cancelled; accepted content remains untouched. + defer func() { + receiptCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + finished := s.now().Format(time.RFC3339Nano) + status, class, message := "success", "", "" + evidence, _ := json.Marshal(result) + if resultErr != nil { + status = "failed" + class, message, evidence = gh.HistoryFailureDetails(resultErr) + } + for _, number := range uniquePositiveNumbers(options.Numbers) { + err := s.store.RecordAnalyticsAttempt(receiptCtx, store.AnalyticsAttempt{Repository: options.Owner + "/" + options.Repo, Number: number, Operation: operation, StartedAt: started, FinishedAt: finished, Status: status, ErrorClass: class, ErrorText: message, Evidence: evidence}) + if err != nil { + resultErr = errors.Join(resultErr, fmt.Errorf("%w: %w", errAnalyticsReceipt, err)) + return + } + } + }() + if options.ReviewStateOnly { + return s.syncReviewState(ctx, options, started) + } client, ok := s.client.(interface { FetchGraphQLHistory(context.Context, string, string, []int, gh.Reporter) (gh.HistoryBatch, error) }) @@ -156,6 +210,16 @@ func (s *Syncer) Sync(ctx context.Context, options Options) (Stats, error) { history = &batch repoRaw = batch.Repository } else { + if len(options.Numbers) > 0 { + allowed, err := s.store.FilterExcludedNumbers(ctx, options.Owner+"/"+options.Repo, uniquePositiveNumbers(options.Numbers)) + if err != nil { + return Stats{}, err + } + options.Numbers = allowed + if len(allowed) == 0 { + return Stats{Repository: options.Owner + "/" + options.Repo, StartedAt: started, FinishedAt: s.now().Format(time.RFC3339Nano)}, nil + } + } repoRaw, err = s.client.GetRepo(ctx, options.Owner, options.Repo, options.Reporter) } if err != nil { @@ -263,6 +327,13 @@ func (s *Syncer) Sync(ctx context.Context, options Options) (Stats, error) { for _, row := range rows { payload := threadSyncPayload{row: row} number := intValue(row["number"]) + excluded, err := s.store.ThreadExcluded(ctx, options.Owner+"/"+options.Repo, number) + if err != nil { + return Stats{}, err + } + if excluded { + continue + } kind := issueKind(row) if history != nil { // Keep legacy REST identity stable when revisiting a previously saved @@ -290,6 +361,8 @@ func (s *Syncer) Sync(ctx context.Context, options Options) (Stats, error) { if item.Pull != nil { payload.hasPullDetails = true payload.pullDetails = pullRequestDetailRows{pull: item.Pull, fetchedAt: s.now().Format(time.RFC3339Nano)} + payload.reviewThreads = item.ReviewThreads + payload.reviewThreadsFetchedAt = payload.pullDetails.fetchedAt } } received.CommentsReceived += len(payload.commentRows) @@ -516,6 +589,11 @@ func (s *Syncer) Sync(ctx context.Context, options Options) (Stats, error) { if err := reserveChild(store.ThreadChildPullRequestDetails); err != nil { return err } + if history != nil { + if err := reserveChild(store.ThreadChildReviewThreads); err != nil { + return err + } + } } if options.IncludePRDetails && thread.Kind == "pull_request" { for _, family := range []store.ThreadChildObservationFamily{ diff --git a/internal/syncer/thread_exclusions_test.go b/internal/syncer/thread_exclusions_test.go new file mode 100644 index 00000000..b6c97ca6 --- /dev/null +++ b/internal/syncer/thread_exclusions_test.go @@ -0,0 +1,61 @@ +package syncer + +import ( + "context" + "net/http" + "net/http/httptest" + "path/filepath" + "sync/atomic" + "testing" + + gh "github.com/openclaw/gitcrawl/internal/github" + "github.com/openclaw/gitcrawl/internal/store" +) + +func TestOwnerExcludedSelectionNeverReachesProviderOrRetryLedger(t *testing.T) { + ctx := context.Background() + s, err := store.Open(ctx, filepath.Join(t.TempDir(), "archive.db")) + if err != nil { + t.Fatal(err) + } + defer s.Close() + _, err = s.DB().Exec(`INSERT INTO repositories(id,owner,name,full_name,raw_json,updated_at) VALUES(1,'fixture','repo','fixture/repo','{}','2026-01-01'); + INSERT INTO threads(id,repo_id,github_id,number,kind,state,title,html_url,labels_json,assignees_json,raw_json,content_hash,updated_at) VALUES + (1,1,'one',10,'pull_request','open','removed','','[]','[]','{}','h','2026-01-01'), + (2,1,'two',20,'pull_request','open','keeper','','[]','[]','{}','h','2026-01-01')`) + if err != nil { + t.Fatal(err) + } + if _, err = s.PurgeThreads(ctx, "fixture/repo", []int{10}, "fixture"); err != nil { + t.Fatal(err) + } + var calls atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { calls.Add(1); http.Error(w, "unexpected request", 400) })) + defer server.Close() + client := gh.New(gh.Options{BaseURL: server.URL}) + for _, graphql := range []bool{false, true} { + for _, review := range []bool{false, true} { + if review && !graphql { + continue + } + operation := "graphql_history" + if review { + operation = "review_state" + } + result, err := New(client, s).Sync(ctx, Options{Owner: "fixture", Repo: "repo", Numbers: []int{10}, State: "all", GraphQLHistory: graphql, ReviewStateOnly: review, ReceiptOperation: operation, IncludeComments: true, IncludePRMetadata: true}) + if err != nil || result.ThreadsSynced != 0 { + t.Fatalf("excluded selection: %+v %v", result, err) + } + } + } + if calls.Load() != 0 { + t.Fatal("owner-excluded provider access", calls.Load()) + } + for _, table := range []string{"analytics_fetch_attempts", "analytics_retries"} { + var n int + s.DB().QueryRow("select count(*) from " + table + " where number=10").Scan(&n) + if n != 0 { + t.Fatal("owner exclusion turned into retry/success", table, n) + } + } +}