Skip to content

Add lightweight observation hooks for bounded batch execution #70

Description

@MasterOfBinary

Current outcome

Expose the information an application needs to size its queues and diagnose quota/latency problems, through one small dependency-free observation contract.

Replanned on 2026-09-06 against GoBatch master 63ef757 and ShitQuant's recorder, enrichment, paper/replay, and flow workloads. This is an implementation target, not a claim that the behavior already exists.

Required behavior

  • Report batch formation cause, actual size, queued/in-flight counts, queue delay, handler duration and distinct error categories.
  • Specify callback concurrency, blocking, panic handling, event ownership/lifetime and ordering guarantees. No unbounded internal telemetry queue.
  • Keep application payloads out of default events; applications adapt events to their preferred logger/metrics system.
  • Measure disabled and enabled overhead. Replace the old literal zero-overhead promise and competing Logger/StatsCollector APIs.
  • Establish reusable event vocabulary for request and flow runtimes without making their correctness depend on an instrumentation framework.

Verification and completion

Use synchronized formation/execution scenarios to assert useful events and counts, callback failure behavior, and bounded memory; benchmark optional-hook overhead.

Follow the repository's formatting, race-test, vet/lint, package documentation, example and changelog requirements for the changed surface. Report the actual supported behavior and migration; do not treat a passing coverage percentage as proof of these outcomes.

Scope and relationships

Depends on #73's bounded-execution state/timing contract. This is not a built-in logging backend, telemetry exporter, dashboard, or statistics database.

Design history

The earlier report/prototype remains below for provenance; the requirements above supersede conflicting prescriptions. Existing discussion is preserved.

Original issue: Optional logging & statistics (observability)

Summary

Add optional, zero-overhead-by-default observability to GoBatch: pluggable logging and runtime statistics/metrics. This captures work from two abandoned branches so it isn't lost.

Motivation

Today there's no way to see what a Batch is doing or to measure throughput/error rates without wrapping processors by hand. Two prior explorations tackled this from different angles; this issue tracks reconciling them into one design.

A. Pluggable Logger

Two candidate interface shapes were prototyped — pick one (or support both via adapters):

Minimal, dependency-free (from cursor/implement-backwards-compatible-logger-hooks-3373 @ a58bbf8) — compatible with log.Logger, slog, zap.SugaredLogger, etc.:

type Logger interface {
    Printf(format string, v ...interface{})
}
// default: silent noopLogger; attached via b := batch.New[T](cfg).WithLogger(log.Default())

Leveled (from feature/logger-stats @ e9a7c6f, local-only):

type Logger interface {
    Log(level LogLevel, format string, args ...interface{})
    Debug(format string, args ...interface{})
    Info(format string, args ...interface{})
    Warn(format string, args ...interface{})
    Error(format string, args ...interface{})
}
// built-ins: NoOpLogger (default, zero overhead), SimpleLogger (stdout/stderr + timestamps)

The batch would log: processing start/completion, source-read progress, per-batch details, and errors as they occur. A logging processor wrapper was also prototyped (processor/logging.go).

B. StatsCollector / runtime metrics

From feature/logger-stats @ e9a7c6f (batch/stats.go):

type StatsCollector interface {
    RecordBatchStart(batchSize int)
    RecordBatchComplete(batchSize int, duration time.Duration)
    RecordItemProcessed()
    RecordItemError()
    RecordSourceError()
    RecordProcessorError()
    GetStats() Stats
}
// built-ins: NoOpStatsCollector (default), BasicStatsCollector (thread-safe in-memory counters/timings)
// Stats: aggregated counts, durations, throughput, success/error rates

Plus a stats processor wrapper (processor/stats.go).

Design notes / open questions

  • Default must be noop with zero overhead (both prototypes did this).
  • Logging vs. metrics are arguably two concerns — this issue can be split into two if preferred.
  • Both prototypes predate the generics migration (feat: migrate GoBatch API to generics #60); APIs need to be made generic (Batch[T]) on adoption.
  • Decide minimal-Printf vs. leveled Logger (or ship minimal + a leveled adapter).

Source (for recovery — branches are being deleted)

  • feature/logger-stats (local commit e9a7c6f, ~2,100 LOC incl. LOGGING_STATS.md, batch/logger.go, batch/stats.go, processor/logging.go, processor/stats.go) — the fuller proposal; remote was already deleted, so this commit existed only locally.
  • cursor/implement-backwards-compatible-logger-hooks-3373 (commit a58bbf8, batch/logger.go minimal interface).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions