Per-session WireGuard traffic stops while P2P remains responsive; paired consumer traces on 1.39.5-derived builds

[kapunakap]

We are investigating established-session traffic failures using Myst 1.39.5-
derived consumers. We have saved, session-fenced traces from the same providers
used concurrently on two environments. We are seeking help correlating these
with provider-side session/payment events, not claiming a proven Myst defect.

Environment and modifications

• OKE: one ARM64 Linux worker, 2 OCPUs / 12 GB, host-networked consumer pod.
• Mac: ARM64 Linux Docker consumer on a residential network. This is not the
  native Mysterium Dark client.
• Separate existing consumer identities; one daemon per environment, multiple
  proxy-port-keyed sessions, target 150 public US-residential proposals per side.
• Myst 1.39.5 source with local atomic consumer payment-accounting correction,
  session-reply timeout adjustment and metadata-only diagnostics. OKE also has
  a one-line outgoing-TUN-batch allocation correction. These are not stock
  binaries, and builds/load/identity differences limit causal comparisons.
• Myst source baseline: c45527af1ea80300ae3d9c92bd37255335b6140d.
• WireGuard dependency: github.com/mysteriumnetwork/wireguard-go at
  406b13e8996a (2024-04-16).
• Mac binary SHA-256:
  f29b30f1098165dea0e30386db212f4692116a607e1fe6e2fd35698c770e516c.
• OKE binary SHA-256:
  919f9cde736b51cc5ab387c67ec5c898b4ce2e0e01402b47a4d018d2ed226bff.
• Public-provider versions and internal logs are unknown.

Observed failure pattern

In a September 19, 2026 16:00–17:00 UTC comparison, 16 overlapping provider pairs
had one side fail first: eleven OKE-first, five Mac-first. In each case the same
provider's unchanged counterpart later passed HTTPS at least five minutes after
the failed side's removal. None of these paired removals occurred within one
minute of each other. Four counterparts eventually failed later, 6.7–18.8 minutes
apart. This does not look like simultaneous whole-provider outages.

For all 16 pairs we retained two minutes before and five minutes after the first
removal, matched to the exact session, proxy port, WireGuard generation and UDP
binding. Failed-side one-second traces had maximum gaps of 1.041 seconds.

• Fifteen failed sessions received no encrypted tunnel packets during the failed
  routine request through removal, while continuing to send.
• One case received two early packets and completed a fresh WG handshake, but
  the request still failed. Please treat it separately from packet silence.
• All sixteen recorded DNS operations timing out before HTTP CONNECT completed.
  Three independent curl checks through the same proxy also timed out without
  CONNECT responses before removal. We did not inject direct-IP traffic into
  these specific sessions.
• The provider still answered 8–17 P2P keepalive requests after the failed side's
  last incoming tunnel packet. Five cases had an isolated keepalive timeout but
  continued successful replies.
• No recorded endpoint change or increment in invalid TCP checksum counts
  occurred in these pre-removal spans. Proxy listeners remained present in the
  sampled pre-removal interval.

Example anonymized OKE case P02 (UTC): last incoming WG packet 16:38:53.650;
first failed check starts 16:39:23.330; removal requested 16:40:03.374. Fourteen
keepalive replies followed the last incoming packet. A successful payment-message
reply arrived at 16:39:01.625. The Mac counterpart continued passing traffic.

Payment evidence and limits

All sixteen failed-side windows contain committed consumer payment metadata and
a successful P2P payment-message reply; fourteen replies arrived after the last
incoming tunnel packet. No payment-message error replies or consumer Hermes
errors were recorded in those short windows.

We do not interpret P2P success as invoice/Hermes acceptance. In the inspected
provider source, the handler queues an exchange on paymentEngineChan and
returns before the payment engine's later validation/Hermes work. We cannot see
that later outcome on public providers. No session-destroy message was observed,
but absence does not prove that the provider kept its session open.

Prior work, to avoid repeating broad tests

• Similar failures have appeared on OKE, Mac Docker/native tests and GKE; older
  comparisons had configuration differences and do not isolate one cloud.
• Kernel versus proxy tests did not establish a general kernel-mode remedy.
• Broader UDP firewall allowance did not restore reliable operation; historical
  flow-log coverage was incomplete, so this does not clear every network layer.
• Separate controlled Mac IP-change cases are excluded from this cohort's cause
  claim. An owned-provider invoice-registration race was fixed and verified,
  but that fix is not deployed to the public providers.
• We found and corrected admission/refill and memory-pressure defects locally.
  These improve usable capacity, not proof of a tunnel-drop cure.

Specific questions

1. Does this resemble a regression or related failure from Connection tunnel collapse on failed handshakes #5492, marked fixed
   in 1.18.3? That release includes Fix resource allocation in the service #5520, sharing resource allocation across
   provider services. Our inspected 1.39.5 source already has the shared allocator;
   we have not established the public providers' versions or allocator behavior.
2. With privately supplied session IDs, provider IDs and UTC timestamps, can you
   correlate provider session-close reasons, resource allocation/release,
   invoice validation and Hermes outcomes for these pairs?
3. Can the control channel remain responsive/acknowledge queued payments after
   the relevant WG session or forwarding state has failed in current versions?
4. Is there a supported private channel for the minimal correlation mapping and
   redacted metadata? We will not post keys, signatures, payment payloads,
   customer payloads or raw wallet data.

We can provide anonymized timelines publicly and session/endpoint mappings
privately after approval. We have no deterministic reproduction of the complete
public-provider failure and are not asking that timing correlation alone be
accepted as proof.

[kapunakap]

Two controlled provider-side HTTP 500s reproduce this signature

We have now found two matching failures with our own restricted provider, so we
can inspect both consumer and provider records. In each, Hermes request_promise
returned HTTP 500; the provider then treated the payment-engine error as fatal
and immediately cleaned up the exact session's WireGuard tunnel. P2P signalling
continued responding, and consumer cancellation followed about fifty seconds
later.

All timestamps below are UTC on September 19, 2026:

Event	Case A	Case B
HTTP 500 response recorded	14:38:00.775	15:57:35.450
Request duration	289 ms	258 ms
Provider payment-engine error / session cleanup	14:38:00.776	15:57:35.451
Consumer cancellation after failed traffic checks	14:38:50.646	15:58:25.565

The error was could not unmarshal error body: received unknown error:. Our
metadata confirms HTTP 500 and the decoding stage, but it did not retain the
raw response, response headers or a server request ID. The exact outbound JSON
was also intentionally not logged. We cannot claim that the requests were valid
or invalid from those records alone, nor that all public-provider drops share
this cause.

The provider constructs request_promise with the existing exchange message,
transactor fee and encrypted R-recovery data. No explicit signature, hashlock or
balance rejection appears in the retained failure logs. HTTP-500 retries remain
disabled; we have not established replay safety after a possible partial commit.

Could you:

1. Identify the server-side failure at those times, or specify the exact private
   correlation identifiers needed to locate it?
2. Provide a supported private channel for the minimum provider/session/Hermes
   mapping? We have not attached these identifiers publicly.
3. Clarify whether an identical request_promise is safe to replay after HTTP
   500, including after processing but before a successful response? If not,
   what reconciliation/recovery procedure should the provider use?

No real-Hermes fault injection or retry change has been deployed. We are not
posting wallets, keys, signatures, payment bodies, raw captures or private
endpoint mappings.

[kapunakap]

We want to reproduce the observed Hermes request_promise HTTP 500 locally and inspect server-side logs. PR #4932 references mysteriumnetwork/hermes, but that repository is inaccessible to our account; the integration-test images mysteriumnetwork/hermes:3.3.15 and 2.3.17 also return access denied.

Is there a supported source repository or development image we can use with a local test chain? If access is restricted, who should we contact? Alternatively, can your Hermes team correlate a captured failure using its UTC timestamp and request ID, if available?

[kapunakap]

Concrete backend dependency error preceding a new Hermes 500 sequence

We captured a new sequence on our restricted provider. All timestamps below are UTC on September 21, 2026; this update covers observations through 18:16:44.

• 17:36:40.121: last captured successful request_promise before this sequence.
• 17:41:40.009: fee lookup returned HTTP 500 from https://transactor.mysterium.network/api/v1/fee/137/settle. The returned error was [err_transactor_fetch_fees] dial tcp <internal-address>:6379: connect: connection refused. No request_promise was recorded for this attempt; the provider fetches fees before sending that request.
• 17:46:40.864–18:16:44.390: seven captured request_promise HTTP 500s, approximately five minutes apart. Each response body was {"cause":"","data":"","message":""}.
• The payment-error counter increased first on the fee lookup failure, then on the seven Hermes failures, reaching eight. The control was still healthy at our check; this update does not claim a new teardown.

We now retain encrypted request evidence plus response bodies/headers. All seven requests passed our offline structural, numeric, chain and request-signature signer-consistency checks. That is not exhaustive request validation: expected signer ownership and encrypted recovery plaintext were not independently verified. The channel/provider/Hermes/chain stayed unchanged. Fee values changed, but fee changes also occurred during earlier successful calls, so we have not established a fee defect.

For the first six errors, read-only consumer/provider state snapshots immediately afterward and around five/fifteen seconds later showed no change in reported promise fields or settled totals. This does not exclude partial internal updates or server-side caching. Other sessions share the consumer payment channel, so later channel-state changes cannot be attributed to these requests alone.

Could you correlate the transactor Redis-connectivity failure at 17:41:40 UTC with the Hermes errors beginning 17:46:40 UTC? Specifically, did they share an affected dependency, or did this leave agreement/payment state requiring recovery? We have evidence of the transactor dependency failure, but cannot prove that the later Hermes 500s share its cause without server-side logs.

No payment replay, new retry behavior, or fault injection was performed. Sensitive payment material and private endpoint mappings are not included here.