Skip to content

fix(chipingress): empty PublishBatch results with nil error counts as delivered - #2446

Open
cawthorne wants to merge 1 commit into
mainfrom
fix/chip-ingress-batch-empty-results
Open

cawthorne wants to merge 1 commit into
mainfrom
fix/chip-ingress-batch-empty-results

Conversation

@cawthorne

Copy link
Copy Markdown
Contributor

Summary

A nil-error PublishBatch response carrying no per-event results is reported to every event's callback as ErrCodeResultsMismatch since dc8a1dc removed the len(resp.Results) > 0 guard in sendBatch. Servers that accept a batch without per-event detail do exist: the local-cre chip-router v1.x forwards every event to Kafka but returns an empty results array.

With the durable emitter on top, this is not a cosmetic mismatch:

  • the durable emitter treats every event as undelivered and retransmits after 60s — forever, because the ack never comes;
  • the node re-sends its whole event stream endlessly;
  • the Kafka topic fills with duplicate retransmits and consumers drown.

Evidence (chainlink v2.67.2-rc.0 node vs chip-router v1.0.1)

Clean A/B against the same router, same environment, only the chainlink-common pin differing (v0.11.2-0.20260914191328 (chipingress v0.0.11-0.20260724142814) vs v0.11.2-0.20260929093916 (chipingress v0.0.11-0.20260915184316)):

old pin (guard present) new pin (guard removed)
DurableEmitter: failed to deliver event. Relying on retransmit. 0 301,385 on one node, first at 12s after boot, zero successful acks
client error — -1: server returned 0 results for 1/500 events
Kafka topic rate modest ~4,000 msg/s (duplicate retransmits; counters inflated ~20x)
metric-event delivery latency ~seconds gaps of 1.5–4.5 min (buffer backup)

The events were delivered to Kafka each time (the router forwards on receive) — only the acks were missing — so the retransmit storm duplicates the entire stream rather than losing data. The delivery-latency gaps are long enough to break downstream freshness-gated consumers (e.g. recording rules gating on time() - <metric>_ts < 120).

Fix

Restores the pre-dc8a1dc3 semantics in sendBatch:

  • nil error + empty results → every callback resolves with nil (delivered);
  • nil error + non-empty results → still correlated per event (unchanged);
  • partial results → still RESULTS_MISMATCH for the missing tail (unchanged, existing test covers it).

Adds a regression test for the zero-results case.

Alternatives considered

  • Router side: chip-router populates per-event results — also valid, but any deployed router without them turns every producer into an eternal retransmit loop, so the client tolerance is the robust side to fix.
  • Config knob: Telemetry.ChipIngressBatchEmitterEnabled = false falls back to the legacy per-event emitter and avoids the contract — but it's a throughput regression at scale and the field is framework-generated in CRE environments, so it isn't reachable for affected deployments.

… delivered

A nil-error PublishBatch response carrying no per-event results is
reported to every event's callback as ErrCodeResultsMismatch since
dc8a1dc removed the len(resp.Results) > 0 guard in sendBatch. Servers
that accept a batch without per-event detail do exist: local-cre
chip-router v1.x forwards every event to Kafka but returns an empty
results array.

With the durable emitter on top this is not a cosmetic mismatch: every
event is treated as undelivered and retransmitted after 60s, forever —
the ack never comes, so the node re-sends its whole event stream
endlessly. On a chainlink v2.67.2-rc.0 node against chip-router v1.0.1
this produced 301k 'failed to deliver event. Relying on retransmit.'
warnings on a single node within minutes of startup, a ~20x duplicated
Kafka topic stream, and metric-event delivery latency gaps long enough
to break downstream freshness gates.

Restore the pre-dc8a1dc3 semantics: a nil-error response with an empty
results array resolves every callback with nil (delivered); a non-empty
array is still correlated per event, and a partial array still mismatches
for the missing tail (unchanged, covered by the existing test). Adds a
regression test for the zero-results case.
Copilot AI balanced review requested due to automatic review settings October 6, 2026 23:59
@cawthorne
cawthorne requested a review from a team as a code owner October 6, 2026 23:59
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

👋 cawthorne, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

📊 API Diff Results

No changes detected for module github.com/smartcontractkit/chainlink-common/pkg/chipingress

View full report

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The focused compatibility fix matches the stated semantics and includes appropriate regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Restores compatibility with batch servers that acknowledge delivery without per-event results, preventing unnecessary durable-emitter retransmissions.

Changes:

  • Treats successful empty PublishBatch responses as delivered.
  • Adds regression coverage while preserving partial-result mismatch handling.
File Description
pkg/​chipingress/​batch/​client.go Handles empty successful responses as batch success.
pkg/​chipingress/​batch/​client_test.go Tests callbacks for empty-result responses.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@cawthorne

Copy link
Copy Markdown
Contributor Author

@pkcll

It seems this might have been intentional:
https://ainv.smartcontract.com/findings/79362753-0fef-4a50-934c-572e8f7c504d

This might actually be the fix:
smartcontractkit/chainlink-testing-framework#2861

c.c. @jmank88

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants