High-throughput classification pipeline for streams and files. Deterministic rules first, small in-process models next, expensive calls only for what's left. One Go binary.
Status: v0.1.0, working toward v1. Rules, schema, and cascade config work end to end (
make testpasses, the examples below run as documented, and the parallel engine is merged). A real model stage is next; a deterministic mock stands in until it lands.
Classifying high-volume data (logs, events, records) with an LLM per record is slow and expensive; hand-rolled regex pipelines are cheap but rot. ClassMesh is the middle path: a confidence-gated cascade where each record exits at the cheapest stage that can decide it.
source -> [ stage 1: rules ] -> [ stage 2: model ] -> [ stage N ] -> sink
│ confident? │ confident?
└── exit early ───────┴── exit early uncertain -> review sink
Everything is an interface: input sources, classification stages, and output sinks are pluggable modules. Today: text files, JSONL event streams, and stdin. Tomorrow: whatever implements Source.
For working examples of each contract, see textfile (a Source), rules (a Stage), and jsonl (a Sink). docs/architecture.md explains how the pieces fit and why the core stays payload-agnostic.
shared/: domain types and the contracts (source,stage,sink) with in-memory implementations for testingservices/cli/: theclassmeshbinary
Runnable examples live in examples/: one ruleset that classifies
both text logs and JSON events. Every block below runs from a fresh clone.
make build
# text logs, one record per line
./bin/classmesh run --rules examples/rules.yml examples/logs.txt
# JSON events, one object per line, classified on their fields
./bin/classmesh run --rules examples/rules.yml --input jsonl examples/events.jsonl
Each classified record is one JSON object on stdout with its category, confidence, the matched rule's reason, and (for events) the decoded fields. Records no rule matches are counted and reported on stderr.
The full multi-stage cascade runs from a fresh clone too. genprod.go writes
production-shaped logs (access lines, probes, app chatter, payments, auth
events, warns, errors, and a slice nothing recognizes; deterministic by seed),
and classmesh.yaml declares a two-tier cascade over them: rules first, a
gated model stand-in for the leftovers, review for what neither tier can
decide, health-check noise dropped by route:
go run examples/genprod.go -n 1000000 > prod.log
./bin/classmesh validate --config examples/classmesh.yaml
./bin/classmesh run --config examples/classmesh.yaml prod.log > classified.jsonl
With the default seed, the million lines classify in under two seconds:
the rules tier decides 88%, the model tier 6%, and 6% lands in
examples/review.jsonl. The stderr stats line reads
processed=1000000 classified=940162 review=59838 by_stage=map[model:60009 rules:880153];
classified counts the health-check records the noise route then discards, so
classified.jsonl holds 720,490 lines.
A whole multi-stage cascade can be declared in one versioned YAML file, checked
with validate and run with run --config:
./bin/classmesh validate --config examples/classmesh.yaml # parse + validate only
./bin/classmesh run --config examples/classmesh.yaml app.log # build and run it
version: 1
input: { type: text } # text | jsonl
stages:
- { id: rules, type: rules, path: rules.yml, gate: 1.0 } # gate is optional
sink: { type: jsonl, stream: stdout } # default sink for classified records
review: { type: jsonl, path: review.jsonl } # optional; the undecided go hereUnknown keys, duplicate stage ids, out-of-range gates, and unknown stage/sink
types are rejected before any input is opened. run --config executes rules,
schema, and mock stages (each honoring its per-stage gate) into the default
sink and the review sink; every stage loads its declaration from the stage's
path. The mock stage is a deterministic model stand-in that scores matched
records with declared confidences below 1.0, so per-stage gates and review
routing can be exercised before a real model stage exists. When the config declares category
routes, classified records are dispatched by category (each route to its own
sink, or drop to discard that category) with the default sink as the fallback
for unrouted categories. A top-level workers: N (or --workers N with
--rules) classifies records on N goroutines while preserving output order,
error reporting, and stats exactly; the default stays serial.
Measured on a single core (AMD Ryzen 7 3800X, make bench):
| Path | Per record | Throughput | Allocations |
|---|---|---|---|
| Rules stage, first-rule hit | 46 ns | ~22M records/sec | 0 |
| Rules stage, 20-rule regex-heavy miss (benchmark ruleset) | 7-8 µs | ~130k records/sec | 0 |
| Full pipeline (engine + rules, discard sink) | 500-550 ns | ~1.8M records/sec | 0 |
| Integrated pipeline (text source -> rules -> JSONL sink) | ~600 ns | ~1.6M records/sec | 2 |
Per-record cost depends on your ruleset: order rules by expected volume so the
hot path exits early. The miss row is the benchmark ruleset's cost, not a
general bound: a pattern with no extractable required literal defeats the
prefilter and every such rule pays its full regex on a miss
(BenchmarkClassifyRegexMissUnprefilterable: ~101 µs at 20 rules on the same
machine). The pipeline row isolates engine + rules behind a discard sink; the
integrated row adds the source read and JSON encode every CLI run pays. The
structured path (JSONL source -> field rules -> structured output) is
benchmarked in shared/pkg/engine as well.
The comparison that motivates the cascade: classifying 1M short log lines with a budget LLM API (~25 input + 5 output tokens each at $0.15/$0.60 per million tokens) costs about $6.75 per million lines and runs at API latency. The rules stage does the same volume in well under a second per core for the cost of the electricity, and the cascade design only forwards the records rules can't decide to anything that costs money.
Reproduce: make bench for the table. For an end-to-end run at volume,
go run examples/genlogs.go -n 1000000 > logs-1m.txt generates a
deterministic weighted stream matching the example ruleset; 1M lines
classify in under a second, including JSON output.
make build # compile binaries into ./bin
make test # run all tests with -race
make lint # golangci-lint across all modules
MIT