[PureGo] All streams share one grpc.ClientConn, and the penalty is about double the Rust core's

[zlata-stefanovic-db]

Summary

A single PureGo SDK multiplexes every stream it creates onto one grpc.ClientConn,
and therefore one HTTP/2 connection. That costs throughput for the same reason it
does in the Rust core (see the companion issue), but PureGo pays roughly twice
the penalty, and the extra half looks attributable to grpc-go's transport rather
than to the endpoint.

Exploratory: the measurement is reproducible, the mechanism is plausible but not
directly instrumented. Proposal at the bottom.

Where the sharing happens

One SDK owns one connection:

// purego/zerobus/sdk.go:176
conn, err := transport.Dial(target, dialOpts...)

// purego/internal/transport/transport.go:44
cc, err := grpc.NewClient(endpoint, dialOpts...)

Every stream opened from that SDK becomes an HTTP/2 stream on that one
ClientConn. There is no option to spread streams over several connections.

What we measured

Harness: an internal cross-SDK benchmark harness. Fixed 120–300
second wall-clock windows rather than a byte budget, because the staging endpoint
drifts by more than the effect under test. Two topologies at 8 streams:

• phase B — 8 SDK instances × 1 stream, so 8 connections
• phase C — 1 SDK instance × 8 streams, so 1 connection (today's default)

All rates below are normalized: 1.00 is the per-connection ceiling established
in the companion issue, the highest rate a single connection carried in these runs.
Absolute throughput is omitted deliberately; every claim here is a ratio. Figures are
rounded to two decimals.

Pooling every PureGo timing at 8 streams, excluding one day when the endpoint held
everything under 0.17:

Phase	Connections	n	Mean	Min	Max
B	8	8	1.01	0.76	1.39
C	1	6	0.56	0.17	0.78

PureGo reaches 56% of its own phase B. The Rust core, measured the same way over
n=27 and n=25, reaches 72%. Same harness, same table, same records, same windows.

Note that PureGo's phase B averages right at the per-connection ceiling, so the same
figure reads two ways: PureGo on one connection reaches 56% of what it reaches on
eight, and 56% of what the endpoint gives a single connection to the Rust core. The
second framing is the one that matters here — a shared connection alone does not
explain it.

The two sessions where PureGo ran both phases:

Session	B	C	C/B
18 Aug 10:56	0.90	0.39	0.43
18 Aug 11:55	1.39	0.71	0.51

Both are well below the Rust core's ratios. The second is the more interesting one:
in that same session all three Rust-core SDKs came out above 1.0, so endpoint
drift cannot be what pushed PureGo to 0.51 — whatever moved it was specific to
PureGo while the endpoint was, if anything, favouring phase C.

Why the extra penalty is probably the transport

For context, the per-connection ceiling itself is not a client-side artefact, and
it is not flow control. The endpoint advertises a 32 MiB connection window and a
2 MiB per-stream window, which at the measured round trip permit roughly 28x and
1.8x the observed ceiling, so there is far more window than the measured rate uses.
Details in the companion Rust issue. So the first half of PureGo's deficit is the
shared connection meeting the endpoint's per-connection limit, exactly as in the
Rust core.

The second half is where the two gRPC implementations differ:

• grpc-go serialises every stream on a connection through a single
  loopyWriter goroutine, fed by a mutex-protected controlBuffer
  (google.golang.org/grpc@v1.81.1/internal/transport/controlbuf.go). Eight streams
  contend for one writer and are drained round-robin. The endpoint does not
  advertise MAX_FRAME_SIZE, so it keeps the 16 KiB default, which puts tens of
  thousands of DATA frames per second through that single writer at the rates
  measured here.
• the h2 crate that tonic uses keeps per-stream send queues and polls them
  from one task, so streams do not contend on a shared lock to get their frames
  queued.

That writer is the only component in the stack that is PureGo's alone, and it is the
one place where a per-connection cost should scale with the number of streams
sharing the connection. It is a plausible mechanism for a penalty that is about
double the Rust core's, but we have not profiled it — no goroutine-block or mutex
profile has been taken during a phase-C run, and that is the obvious next step.

What is still open

1. Is the extra deficit really the loopyWriter? A blockprofile or
   mutexprofile from a phase-C run, plus GODEBUG=http2debug=1 frame counts,
   would show whether streams are queueing behind the control buffer. Cheap to do
   and it would move this from plausible to demonstrated.
2. How much is PureGo's buffered-payload cap? PureGo bounds buffered payload
   bytes (64 MiB by default) as well as in-flight requests, which the Rust core
   does not. At one connection that cap could bind earlier than it does at eight.
   A phase-C run with -max-buffered-bytes raised would separate the two. Note that
   raising it has previously cost throughput at one stream (about 12%, plus 4x the
   run-to-run spread, most likely GC pressure), so this is a diagnostic rather than a
   fix.
3. Only two paired sessions. The pooled figures lean on unpaired timings. Two or
   three more back-to-back B/C pairs would firm up the 56%-versus-72% comparison,
   which is currently the main quantitative claim here.

Proposed fix

Same shape as the companion issue, adapted to PureGo:

• An option to give each stream its own connection. PureGo has no equivalent of
  the v0.3.0 Rust behaviour to revert to — it has always shared — so this is new
  work rather than a revert. Concretely: let SDK hold several transport.Conns and
  assign streams round-robin, with WithMaxConnections(n) or
  WithConnectionPerStream() on the constructor.
• A pool is the better end state for the same reason as in the core: it
  decouples connection count from stream count, so a caller with 200 streams does
  not open 200 connections.

If the loopyWriter contention in (1) turns out to be real and significant, a pool
addresses it directly as well, since spreading streams across connections spreads
them across writers. That makes one change worth doing for two reasons.

Acceptance test is already scripted: a paired B/C run should bring phase C to phase
B, since with one connection per stream the two topologies are the same thing.

Worth noting the workaround available today: create one SDK per stream. That is
exactly what phase B does and what produced the phase B column above. The README
currently steers callers toward a single SDK with several streams, which is the
slower shape, so documenting this is worth doing independently of any code change.

[zlata-stefanovic-db]

Companion issue for the Rust core and its cgo, C++ and Java wrappers, where the shared channel arrived in v0.4.0: #766

[krkeshav]

Hi I was reading about Zerobus and stumbled upon this issue. Its really interesting. I had a question though, if there is a pool of connections instead of only one connection, won't this break the ordering guarantee at server end?
As per this blog, it seems the server considers ordering guarantee for each connection.

[zlata-stefanovic-db]

Hi I was reading about Zerobus and stumbled upon this issue. Its really interesting. I had a question though, if there is a pool of connections instead of only one connection, won't this break the ordering guarantee at server end? As per this blog, it seems the server considers ordering guarantee for each connection.

Hi @krkeshav, thanks for the interest! Ordering is at the stream level — the "connection" in the blog refers to the logical stream, and that same section says "Ordering lives at the stream level." A pool assigns whole streams to connections and never splits a stream across them, so per-stream ordering is unaffected. Cross-stream ordering isn't guaranteed today either, shared connection or not.

[krkeshav]

Thanks for confirming @zlata-stefanovic-db !
This looks fun to me. I have some bandwidth, can I look into this issue? (if yes, then might need some help with testing setup)