Provider signs quality metrics with a foreign or empty identity → "Failed to sign metrics event"

[IanJohnsons]

Describe the bug

On every provider node, each incoming session produces one ERR core/quality/mysterium_morqa.go:212 > Failed to sign metrics event error="failed to sign metrics event: authentication needed: password or unlock".

The identity is unlocked. The node is asked to sign a metrics batch as the consumer's address, whose private key it does not hold.

The trace stage "Session validation" is the only one of 22 trace stages without a Provider/Consumer prefix. The MORQA mapper decides who owns an event from the stage-name prefix, so it attributes this provider-side stage to the consumer. The filesystem keystore answers ErrLocked for any address that is not in its unlocked map, which produces the misleading "password or unlock" message.

Previously reported as #6002 (March 2024, same error at mysterium_morqa.go:212). It was closed by the stale bot without investigation. This report provides the root cause.

Affected version: 1.39.6 (commit c45527af). The code path has been present since 1.21.0 (May 2023).
Component: core/quality (MORQA metrics) × core/service (session tracing)
Severity: Low (functional impact) / Medium (log noise and misleading error for every provider operator)

To Reproduce
Steps to reproduce the behavior:

1. Run a provider node (any version since 1.21.0) as the mysterium-node systemd service, with an unlocked identity.
2. Wait for any consumer to establish a session.
3. Run journalctl -u mysterium-node | grep "Failed to sign metrics event". One error appears per incoming session.
4. Optional: compare per-day counts with grep "session ref incr for" (one line per provider session). They match 1:1.

Expected behavior
Provider-side trace metrics, including Session validation, are attributed to and signed with the provider's own (unlocked) identity, with IsProvider: true. No Failed to sign metrics event error is logged.

Screenshots
N/A. Log excerpts are included under Additional context.

Environment (please complete the following information):

• Node version: 1.39.6 (build 35291316599, commit c45527a). The code path is unchanged since 1.21.0.
• OS: Debian 13 (trixie) x86_64 (VPS); Parrot Security 7.3 x86_64 (laptop); Debian 13 (trixie) aarch64 (Raspberry Pi). All run the mysterium-node systemd service.
• Desktop app version (if applicable): N/A
• Docker (if applicable): N/A
• Your identity (if applicable): 0x45dbb4bf16ddeaf3ed7eb32fbeb5cff28faa5ef7 (VPS), 0xaf447d38e8a51975d37d19ff9bf477ba36128663 (laptop), 0x31e1c280033fa5b37d412a85c0f1ea271d29c0a5 (Pi)
• Payment (top-op) reference (if applicable): N/A

Additional context

Root cause

1. A provider-side stage without the Provider prefix

core/service/session_manager.go#L178

go func() {
    trace := session.tracer.StartStage("Session validation")
    validationError = manager.validateSession(session, prices)
    session.tracer.EndStage(trace)
    validationWG.Done()
}()

This runs in the provider's session manager, next to "Provider session create", "Provider session create (start)", (payment) and (configure).

All trace stages in the repository (non-test code, 1.39.6):

Prefix	Stages
Consumer …	11
Provider …	10
none	1: Session validation

2. The owner is chosen by stage-name prefix

core/quality/morqa_transport.go#L278-L283

func traceEventToMetricsEvent(ctx sessionTraceContext) (string, *metrics.Event) {
    sender, target, isProvider, country := ctx.Consumer, ctx.Provider, false, ctx.ProviderCountry
    // TODO Remove this workaround by generating&signing&publishing `metrics.Event` in same place
    if strings.HasPrefix(ctx.Stage, "Provider") {
        sender, target, isProvider, country = ctx.Provider, ctx.Consumer, true, ctx.ConsumerCountry
    }
    return sender, &metrics.Event{ ... }
}

For "Session validation":

• sender is ctx.Consumer, which becomes the batch owner.
• IsProvider is false and TargetId is the provider, so the event is reported to MORQA as if the consumer had sent it.

3. The node signs as the owner

core/quality/mysterium_morqa.go#L139

signature, err := m.signer(identity.FromAddress(owner)).Sign(bin)

4. The keystore returns ErrLocked for any address it hasn't unlocked, including unknown ones

identity/keystore_filesystem.go#L218-L229

unlockedKey, found := ks.unlocked[a.Address]
if !found {
    return nil, ethKs.ErrLocked   // "authentication needed: password or unlock"
}

The consumer's address is never in the provider's keystore, so the result is always ErrLocked. The message makes operators look for a passphrase or unlock problem that does not exist.

5. The batch is sent anyway, unsigned

core/quality/mysterium_morqa.go#L210-L217

signature, err := m.signBatch(owner, batch)
if err != nil {
    log.Error().Err(err).Msg("Failed to sign metrics event")   // no return
}
sb := &metrics.SignedBatch{
    Signature: signature,   // "" on error
    Batch:     batch,
}

The request is still sent with an empty signature. On all three nodes below, Failed to sent batch metrics request never appears, so the MORQA server accepts these unsigned batches without returning an error.

Call chain

core/service/session_manager.go:178     StartStage("Session validation")       [provider]
  → trace.AppTopicTraceEvent
  → core/quality/sender.go:400          sendTraceEvent()
  → core/quality/morqa_transport.go:278 traceEventToMetricsEvent()  owner = ctx.Consumer
  → core/quality/mysterium_morqa.go:139 signer(consumer).Sign()
  → identity/keystore_filesystem.go:225 ErrLocked
  → core/quality/mysterium_morqa.go:212 ERR "Failed to sign metrics event"

---

Why the existing tests don't catch it

Test	What it does	Why it misses the bug
core/quality/morqa_transport_test.go#L44	Uses identity.SignerFake{} as signer factory	SignerFake signs successfully for any identity, so a wrong owner can never fail
identity/keystore_mock.go#L81-L92	Mock keystore returns ErrNoMatch for unknown addresses	Production returns ErrLocked for the same case, so mock and real keystore disagree
core/service/session_manager_test.go#L142, #L189	Asserts the provider trace sequence includes "Session validation"	Confirms the stage is provider-side, but nothing checks how the mapper attributes it
—	No test for traceEventToMetricsEvent	The prefix rule is untested

The session manager test treats the stage as part of the provider flow, and the mapper's own TODO calls the prefix check a workaround. Both indicate the behaviour is not intentional. The stage was added three years after the prefix convention without following it.

---

Timeline

Date	Commit	Change
2020-05-05	03df488c8	"Unlock decrypts keystore once": SignHash returns ErrLocked for any address not in the unlocked map
2020-06-29	4104f0c88	"Send session trace from the consumer": prefix-based owner workaround introduced
2023-05-16	3d4676ac	"Replace policy fetching with specific identity check request": adds StartStage("Session validation") without prefix
2023-05-22	tag 1.21.0	First release containing the bug
2026-08-17	tag 1.39.6	Still present

---

Evidence from three independent provider nodes

Node details

Host	Arch	OS	Version	Identities in keystore	--identity.passphrase	Log TZ
project-vps	x86_64	Debian 13 (trixie), 6.12.107 cloud	1.39.6 / c45527a	1	not set	UTC
parrot	x86_64	Parrot Security 7.3	1.39.6 / c45527a	1	set	CEST
crypto (Raspberry Pi)	aarch64	Debian 13 (trixie), 6.18.50+rpt-rpi-v8	1.39.6 / c45527a	1	set	CEST

All nodes are installed as the mysterium-node.service systemd unit, with persistent journald. Each keystore holds exactly one UTC--… key file (owner mysterium-node, mode 0600), and it matches the unlocked address.

One error per incoming session

The per-day count of Failed to sign metrics event is compared with session ref incr for (logged once per provider session, right after the Session validation stage starts at session_manager.go#L190).

Host	Date	Sign errors	New sessions
project-vps	2026-09-16	10	10
project-vps	2026-09-17	14	14
project-vps	2026-09-18	19	20
project-vps	2026-09-19	12	11
project-vps	2026-09-20	10	9
project-vps	2026-09-21	8	6
parrot	2026-09-19	10	10
parrot	2026-09-20	60	60
parrot	2026-09-21	26	27
crypto	2026-09-21	9	10

The counts track 1:1. The small ±1–2 offsets come from batching across midnight: sessions near day-end are flushed after 00:00, and the current day was still in progress.

Present since day one on every node

The first error in each journal appears within minutes of the oldest retained node entry: VPS +7 s, parrot +8.5 min, Pi +3 min. No error-free period precedes it.

Older logs are gone. On the VPS and the laptop, journald is at its size cap (SystemMaxUse=500M, usage 500.5M and 470.6M) and continuously rotates old entries out. On the Pi, the persistent journal only starts at the current boot. /var/log/mysterium-node/ exists on all three nodes but is empty.

The missing logs do not leave the start date open. Every identity was created long after the bug shipped in 1.21.0 (2023-05-22), as shown by the key file names:

Host	Identity created (keystore file name)
parrot	2026-01-05
project-vps	2026-03-28
crypto (Pi)	2026-05-18

Every node has therefore run an affected version since its first session. The combination of the code path (unchanged since 2023), the 1:1 session correlation, and the error being present from the first retained log line confirms the error has occurred from day one on each node. It only became noticeable recently because earlier logs had already been rotated away.

Identity is unlocked when the errors occur

project-vps (UTC)

Sep 21 12:27:55 project-vps systemd[1]: Started mysterium-node.service - Server for Mysterium - decentralised VPN Network.
Sep 21 12:28:01 project-vps myst[1184]: 2026-09-21T12:28:01.303 INF cmd/commands/service/command.go:135      > Unlocked identity: 0x45dbb4bf16ddeaf3ed7eb32fbeb5cff28faa5ef7
Sep 21 12:28:28 project-vps myst[1184]: 2026-09-21T12:28:28.863 ERR core/quality/mysterium_morqa.go:212      > Failed to sign metrics event error="failed to sign metrics event: authentication needed: password or unlock"
Sep 21 12:38:29 project-vps myst[1184]: 2026-09-21T12:38:29.781 ERR core/quality/mysterium_morqa.go:212      > Failed to sign metrics event error="failed to sign metrics event: authentication needed: password or unlock"

parrot (CEST)

Sep 21 08:37:57 parrot systemd[1]: Started mysterium-node.service - Server for Mysterium - decentralised VPN Network.
Sep 21 08:38:27 parrot myst[3316]: 2026-09-21T08:38:27.935 DBG identity/selector/handler.go:108         > Unlocked identity: 0xaf447d38e8a51975d37d19ff9bf477ba36128663
Sep 21 08:38:27 parrot myst[3316]: 2026-09-21T08:38:27.938 INF cmd/commands/service/command.go:135      > Unlocked identity: 0xaf447d38e8a51975d37d19ff9bf477ba36128663
Sep 21 14:41:52 parrot myst[3316]: 2026-09-21T14:41:52.728 ERR core/quality/mysterium_morqa.go:212      > Failed to sign metrics event error="failed to sign metrics event: authentication needed: password or unlock"
Sep 21 14:42:53 parrot myst[3316]: 2026-09-21T14:42:53.036 ERR core/quality/mysterium_morqa.go:212      > Failed to sign metrics event error="failed to sign metrics event: authentication needed: password or unlock"
Sep 21 14:47:24 parrot myst[3316]: 2026-09-21T14:47:24.140 ERR core/quality/mysterium_morqa.go:212      > Failed to sign metrics event error="failed to sign metrics event: authentication needed: password or unlock"

crypto / Raspberry Pi (CEST)

Sep 21 12:54:12 crypto systemd[1]: Started mysterium-node.service - Server for Mysterium - decentralised VPN Network.
Sep 21 12:54:17 crypto myst[2208]: 2026-09-21T12:54:17.440 DBG identity/selector/handler.go:108         > Unlocked identity: 0x31e1c280033fa5b37d412a85c0f1ea271d29c0a5
Sep 21 12:54:17 crypto myst[2208]: 2026-09-21T12:54:17.442 INF cmd/commands/service/command.go:135      > Unlocked identity: 0x31e1c280033fa5b37d412a85c0f1ea271d29c0a5
Sep 21 14:11:05 crypto myst[2208]: 2026-09-21T14:11:05.372 ERR core/quality/mysterium_morqa.go:212      > Failed to sign metrics event error="failed to sign metrics event: authentication needed: password or unlock"
Sep 21 14:17:39 crypto myst[2208]: 2026-09-21T14:17:39.963 ERR core/quality/mysterium_morqa.go:212      > Failed to sign metrics event error="failed to sign metrics event: authentication needed: password or unlock"
Sep 21 14:26:19 crypto myst[2208]: 2026-09-21T14:26:19.407 ERR core/quality/mysterium_morqa.go:212      > Failed to sign metrics event error="failed to sign metrics event: authentication needed: password or unlock"

Local causes ruled out

Hypothesis	Check	Result
Stale or multiple identities	ls -la /var/lib/mysterium-node/keystore/	Exactly 1 key file per node, matches unlocked address
Passphrase / unlock problem	--identity.passphrase presence + Unlocked identity log line	Occurs both with and without passphrase; unlock succeeds on all nodes
Startup race	Error timestamps vs. unlock	Errors up to 6+ h after unlock, same PID
Platform-specific	Arch / distro	x86_64 + aarch64, Debian 13 + Parrot 7.3
MORQA server blocked	curl https://quality.mysterium.network/api/v3/, iptables/ufw, nftables, CrowdSec, fail2ban	Reachable from all nodes (HTTP 404 on base path); both resolved IPs (188.245.149.150, 116.203.130.98) have 0 matches in every block list
Batch rejected by server	grep -c "Failed to sent batch metrics request"	0 on all nodes

Signing happens entirely locally before any network call, so network conditions cannot cause this error. They are listed for completeness.

---

Impact

• Log noise: one ERR per session on every provider node, with a message that points operators to a non-existent passphrase or unlock problem.
• Misattributed metrics: the provider's validation timing is reported with IsProvider: false, TargetId: <provider> and the consumer's ID as sender, so MORQA stores provider-side data as consumer-reported.
• Unsigned batches accepted: these batches reach MORQA without a signature, and the server does not reject them. Whether the server should accept unsigned batches is a separate question.
• Earnings and sessions: not affected.

---

Suggested fix

Minimal: prefix the stage so the existing convention holds.

- trace := session.tracer.StartStage("Session validation")
+ trace := session.tracer.StartStage("Provider session validation")

This also requires updating the expected sequences in core/service/session_manager_test.go (L142, L189). Check any MORQA-side dashboards or queries keyed on the old stage name.

Proper: resolve the existing TODO. Carry IsProvider (or the owner identity) explicitly in trace.Event / sessionTraceContext instead of deriving it from the stage name. A future stage name can then no longer silently flip ownership.

Hardening (optional):

• In sendMetrics, return or skip on signing failure instead of posting an empty signature.
• Add a test for traceEventToMetricsEvent covering every stage name, or asserting that every StartStage literal carries a Provider/Consumer prefix.
• Align keystore_mock.SignHash with production (ErrLocked vs ErrNoMatch), or use a signer factory in MORQA tests that fails for identities other than the expected owner.

---

Reported by Ian Johnsons (Telegram: @IanJohnsons). Analysis based on tag 1.39.6 source and journald output from three independent provider nodes.

[IanJohnsons]

Update: second source found, both fixed and verified live in #6221

While testing the fix on a live provider node, I found a second, independent source of the same error. It explains the remaining errors on hosts with a public IP.

Source 2: public_ip NAT event with an empty ID (one error per node start)

cmd/di.go handleNATStatusForPublicIP publishes the event during bootstrap with a hard-coded empty ID:

if outIP == pubIP && pubIP != "" {
    di.EventBus.Publish(event.AppTopicTraversal, event.BuildSuccessfulEvent("", "public_ip"))
}

• Added in c53685e (2021-04-27, 0.46.2) for the local nat/status_tracker.go, which did not need an identity.
• That tracker was removed in 7a77bce (2021-10-07, 0.67.0, "Throw away legacy status_tracker.go").
• Since then, the only subscriber of AppTopicTraversal is the quality sender. It turns the event into a MORQA metric with owner "", which cannot be signed (ErrLocked) or attributed to any provider.
• It only triggers when the outbound IP equals the public IP (VPS / dedicated hosts). Nodes behind NAT are not affected.

A temporary debug build that logs the batch owner on signing failure confirmed it, about 30 s after start (the first MORQA batch flush):

Failed to sign metrics event ... event_types=["*metrics.Event_NatMappingPayload[provider=true target=]"] events=1 owner=

This also explains the counts in my original report. On the VPS (public IP), 2026-09-21 had 8 errors = 6 sessions + 2 restarts. The two nodes behind NAT only show the per-session errors.

Fix and live verification

#6221 now contains both fixes:

1. "Session validation" → "Provider session validation", plus a regression test that enforces the Provider/Consumer prefix on every trace stage.
2. The public_ip NAT event is published after identity unlock, with the unlocked identity (the same pattern as sendNATType), plus a unit test.

Patched build on the same public-IP host, A/B on 2026-09-21 (UTC):

	Official 1.39.6	Patched
Restart	17:28:16 start → 17:29:34 Failed to sign	17:31:15 start → no error
Sessions	1 error per session	5 sessions (17:37, 18:16, 18:53, 19:11, 19:13) → 0 errors

Full details are in the PR description.