run-non-validator: node stays pinned to a degraded block-stream peer indefinitely (liveness-only stream health, no throughput check)

[KunHcz]

Summary

A non-validator's block stream health appears to be checked only for liveness (nv_stream_msg read deadline). If the node lands on a peer that keeps delivering blocks but far below the chain's production rate, the stream never times out, the node never rotates, and it falls behind real time indefinitely. Operators writing data (--write-fills etc.) then face a bad trade-off: keep falling behind, or restart and accept a permanent local gap (resync bootstrap only backfills a limited window).

Environment

• Mainnet, hl-visor managed non-validator (binary index 102, 2026-07-18 release)
• run-non-validator --replica-cmds-style recent-actions --serve-eth-rpc --serve-info --write-fills --write-order-statuses --write-misc-events --disable-output-file-buffering
• override_gossip_config.json with the published root node IPs, try_new_peers: true
• Host: 8 cores / 128 GB, ample headroom throughout the incident (hl-node ~1.5 cores, no swap pressure)

Observed (2026-07-21, UTC)

1. 07:20:08 — upstream stream peer stops delivering:

   WARN hl-node @@ client_block_reader error, ending task: lu::timeout nv_stream_msg: deadline has elapsed @@ [tcp_stream.peer_addr(): Ok(13.193.202.9:4001)]
   WARN hl-node @@ forward_client_blocks no more blocks from reader, reconnecting to a new peer
   

2. Rotation hunt: several candidates reject the connection; the node settles on the first peer that accepts.
3. That stream delivers ~2 blocks/s while the chain produces ~12 blocks/s. Applied cadence stays at 100 blocks per 33–64 s for 48 minutes with zero further rotations — the dribbling stream always beats the read deadline, so the liveness check passes forever:

   08:04:57 WARN hl-node @@ applied block 1080545000
   08:05:53 WARN hl-node @@ applied block 1080545100
   08:06:28 WARN hl-node @@ applied block 1080545200
   08:07:02 WARN hl-node @@ applied block 1080545300
   

   Lag grew ~0.85 min per minute. Chain itself was healthy the whole time (public API/candles complete; no status page incident).
4. 08:10 — manual container restart. New peer selection + RPC bootstrap (100-block batches) caught up to head in ~6 minutes and returned to full-speed gossip streaming.
5. Cost of the restart: the accumulated ~38 min lag became a permanent 28-minute hole in written output (node_fills etc.), since bootstrap only backfilled the most recent portion.

Request

Any of the following would fix or greatly mitigate this class of incident:

1. Throughput-aware stream health: rotate the stream peer when the applied-block rate stays substantially below the live block rate for N seconds (the node can observe wall-clock block-time drift from block headers alone).
2. Or a config knob in the gossip/visor config: e.g. max_stream_lag_secs / min_stream_rate, so operators can opt into rotation on sustained lag.
3. Or an operator signal to force stream re-selection without a full restart (admin endpoint or process signal). This alone would remove the "restart ⇒ permanent data gap" penalty for data-writing non-validators.

Happy to provide fuller logs on request.

[tchiggins]

I see exactly the same thing as well occasionally, which requires a node restart.