[perf] TLS handshake rate plateaus at ~640/s and HTTPS costs ~35% throughput vs HTTP #306

Summary

Measured CWIST HTTPS performance on a 12-core Linux loopback box. HTTPS costs ~35% of throughput vs plaintext on the same handler, and full TLS 1.3 handshakes plateau at ~640 handshakes/sec regardless of client concurrency (no scaling past ~P32). That is low for a 12-core machine — nginx/OpenSSL-class servers do several thousand full handshakes/sec.

Measured numbers

Benchmark server: cwist_app_use_https() with example/othello-web/server.{crt,key}, one route returning a 20-byte JSON. Tooling: wrk -t4 -c100 -d10s, curl --no-keepalive for handshake rate.

Throughput (keep-alive):

Full TLS 1.3 handshakes/sec (curl, loopback):

CWIST beats the sequential openssl s_server baseline but stops scaling at ~640/s, suggesting the handshake pipeline (not crypto) is the bottleneck — consistent with open issue #294.

Code findings (src/net/http/https.c)

Good: cert/key loaded once into SSL_CTX (no per-request reload); session resumption + shared pre-fork ticket keys enabled; non-blocking handshakes on sharded epoll shepherd threads.

Suspected bottlenecks:

1. Handshake shard ceiling: default CWIST_HTTPS_HS_MAX_SHARDS 16 (https.c:541). Fixed shard count likely explains the ~640/s plateau. See [perf] Evaluate raising HTTPS handshake shard default #294.
2. 16 KiB read buffer (include/cwist/net/http/http.h:138) — every SSL_read caps at 16 KiB (https.c:1477-1478); no SSL_set_buffer_size/BIO enlargement anywhere, so large request bodies need many small encrypted reads.
3. Connection teardown cost: every connection does SSL_shutdown + up to 16×2 KiB of recv(MSG_DONTWAIT) draining + close (https.c:1383-1399) — thousands of extra syscalls/sec at handshake-churn rates (relates to Performance: reduce syscalls and serialization overhead to close the Actix gap #297).
4. Blocking handshake fallback: non-Linux path does a blocking SSL_accept retry loop with poll waits (https.c:1309-1346).

Suggested next steps

• Raise/autotune handshake shard count; re-measure handshakes/sec vs shards.
• Enlarge TLS read buffer for large bodies.
• Reduce teardown syscalls (skip/shorten shutdown drain, reuse connections).
• Add a CI perf gate for HTTPS throughput + handshake rate so this regresses loudly.

Environment: Linux x86_64, 12 cores, loopback, CWIST built via make libcwist.a (vendored BoringSSL), test cert from example/othello-web.