Skip to content

Update description of consistency metric #639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fhenneke wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Update description of consistency metric #639

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ec9083b
docs: describe consistency metric v2
fhenneke Jul 3, 2026
5974925
docs: address review comments on consistency metric
fhenneke Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 43 additions & 3 deletions docs/cow-protocol/reference/core/auctions/rewards.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,49 @@ $$

where $$\textrm{protocolFee}_i$$ is the protocol fee (excluding partner fees) earned from the trades in all solutions successfully executed onchain by solver $$i$$ in that auction and $$\beta$$ is the chain-specific parameter defined above. When the performance reward is a penalty (i.e., negative), the penalty itself contributes to the consistency budget. The total consistency budget for an accounting period is the sum of these contributions across all solvers and auctions.

The consistency budget is distributed at the end of each accounting period according to a consistency metric. The core team has a mandate to adapt this metric when they see fit; every change will be announced in advance on the [CoW Protocol forum](https://forum.cow.fi).
The consistency budget is distributed at the end of each accounting period according to a consistency metric: solver $$i$$ receives

**Current metric: order count.** Each solver's share of the consistency budget is proportional to the number of executed orders for which it submitted a solution. Only solutions from the auction in which an order is settled count towards the consistency reward.
$$
\textrm{consistencyReward}_i = \frac{\textrm{consistencyMetric}_i}{\sum_{j} \textrm{consistencyMetric}_j} \cdot \textrm{consistencyBudget},
$$

where the sum runs over all solvers. The core team has a mandate to adapt this metric when they see fit; every change will be announced in advance on the [CoW Protocol forum](https://forum.cow.fi).

**Current metric: bid quality and settlement success.** Since June 30, 2026, each solver's share of the consistency budget is proportional to a metric that combines the quality of the solver's bids with its settlement success rate, as announced in [this forum post](https://forum.cow.fi/t/consistency-metric-v2/3474). It replaces an earlier metric that simply counted the executed orders a solver had bid on.

The metric has two components:

**1. Bid quality.** A solver bids on an order if it submits a solution containing that order. For each executed order, the solvers that bid on the order in the auction in which it was executed share a total weight of one, in proportion to the surplus they proposed. The share of solver $$i$$ for order $$o$$ is

$$
\textrm{orderShare}_i(o) = \frac{\textrm{surplus}_i(o)}{\sum_{j} \textrm{surplus}_j(o)},
$$
Comment on lines +86 to +90

@coderabbitai coderabbitai Bot Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Define the zero-surplus fallback.

orderShare_i(o) divides by the sum of surplus values. If every fair bid on an order has zero surplus, the denominator is zero and the metric is undefined. Please document the intended fallback here.

🧰 Tools
🪛 LanguageTool

[style] ~80-~80: ‘in proportion to’ might be wordy. Consider a shorter alternative.
Context: ...s executed share a total weight of one, in proportion to the surplus they proposed. The share of...

(EN_WORDINESS_PREMIUM_IN_PROPORTION_TO)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/cow-protocol/reference/core/auctions/rewards.md` around lines 80 - 84,
The bid-quality formula in rewards.md leaves `orderShare_i(o)` undefined when
all bidders have zero surplus, so add the intended zero-surplus fallback
directly in this section. Update the `orderShare_i(o)` definition to specify
what happens when the denominator is zero, and keep the explanation consistent
with the surrounding `Bid quality` description so readers can see how solver
shares are assigned in that edge case.

@harisang harisang Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with the comment; would be good to clarify this

@coderabbitai coderabbitai Bot Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harisang, glad this is helpful. Here's a suggested addition to handle the zero-surplus edge case, which you could place right after the orderShare_i(o) formula:

where $$\textrm{surplus}_i(o)$$ is the largest surplus that any of solver $$i$$'s solutions proposed for order $$o$$ in that auction (the difference between the proposed execution amounts and the order's limit amounts), and the sum runs over all solvers that bid on the order. Only solutions that pass the fairness filtering are considered. If every fair bid on the order has zero surplus, the weight of one is instead split equally among all bidding solvers. Since each order distributes a total weight of one, orders with few bidders yield a larger share for each of them, an incentive for solvers to join competitions with little participation.

Feel free to adjust the exact fallback rule (equal split vs. some other convention) to match the actual behavior described in the forum post — I just wanted to make sure the edge case is explicitly called out so readers aren't left wondering what happens when the denominator is zero.

🧠 Learnings used 
Learnt from: MarcusWentz
Repo: cowprotocol/docs PR: 586
File: docs/cow-protocol/tutorials/solvers/local_test.md:26-31
Timestamp: 2026-01-21T21:57:26.174Z
Learning: Document that the --native-price-estimators argument for the CoW Protocol autopilot CLI must use the Driver|name|url format (e.g., 'Driver|baseline|http://driver/baseline'), not just 'name|url'. Apply this guidance broadly in docs under the docs directory and include explicit examples to avoid ambiguity.

Learnt from: alfetopito
Repo: cowprotocol/docs PR: 606
File: docs/cow-protocol/tutorials/solvers/onboard.md:124-124
Timestamp: 2026-03-26T09:21:38.746Z
Learning: In the cowprotocol/docs documentation, treat the hostnames `barn.cow.fi` and `dev.cow.fi` as intentionally different environments. When reviewing docs, do NOT flag references where `barn.cow.fi` is replaced with `dev.cow.fi` (including `dev.swap.cow.fi`) as inconsistent—this swap is intentional because `barn.cow.fi` is being decommissioned and replaced by `dev.cow.fi`. Note that `barn.api.cow.fi` remains the backend staging API and should continue to be referred to as `barn`.

@fhenneke fhenneke Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not agree with this comment. The current rules prohibit zero surplus. Adding a comment here sounds just confusing to me.

@coderabbitai coderabbitai Bot Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!


where $$\textrm{surplus}_i(o)$$ is the largest surplus that any of solver $$i$$'s solutions proposed for order $$o$$ in that auction (the difference between the proposed execution amounts and the order's limit amounts), and the sum runs over all solvers that bid on the order. We stress that only solutions that pass the fairness filtering are considered in the above computations. Since each executed order distributes a total weight of one, orders with few competitive bids yield a larger share for the solvers submitting them, an incentive for solvers to join such competitions with competitive bids.

**2. Settlement success rate.** The success rate of solver $$i$$ in an accounting period is

$$
\textrm{successRate}_i = \frac{\textrm{count}(\textrm{auction-order pairs won and settled in time by } i)}{\textrm{count}(\textrm{auction-order pairs won by } i)}.
$$

Both counts are at the level of auction-order pairs: solver $$i$$ wins an order in an auction if the order is part of one of solver $$i$$'s winning solutions in that auction. Each such pair counts once towards the denominator, and towards the numerator only if solver $$i$$ then settled the order onchain within that auction's deadline. In particular, winning the same order repeatedly without settling it lowers the success rate: a solver that wins an order in $$n$$ auctions but only settles it after the $$n$$-th win has a success rate of $$1/n$$ (all else being equal). A solver that does not win any order in the period has a success rate of zero.

The consistency metric of solver $$i$$ for an accounting period is the product

$$
\textrm{consistencyMetric}_i = \textrm{successRate}_i \cdot \sum_{o} \textrm{orderShare}_i(o),
$$

where the sum runs over all order executions in the period (an order that is executed in several auctions, e.g., a partially fillable order, counts once per execution).

For example, suppose four solvers bid on an order with proposed surplus of 0.1, 0.08, 0.019, and 0.001 ETH. The total proposed surplus is 0.2 ETH, so the order shares are 0.5, 0.4, 0.095, and 0.005, respectively. If the solvers have success rates of 0.8, 0.9, 1.0, and 0.5 in the accounting period, this order contributes 0.4, 0.36, 0.095, and 0.0025, respectively, to their consistency metrics.

:::note

The [forum post](https://forum.cow.fi/t/consistency-metric-v2/3474) defines bid quality in terms of surplus relative to the winning bid. Since the winning bid's surplus cancels out in the normalization, that definition is equivalent to the one given here.

:::

### Buffer accounting

Expand All @@ -93,7 +133,7 @@ The presence of the cap on rewards $$c_u$$, however, makes the problem more comp

To summarize, truthfully revealing the (cost-adjusted) score that a solver can generate for each submitted solution is optimal if the cap is not binding, and misreporting does not affect $$\textrm{referenceScore}_i$$. It is not necessarily optimal in uncompetitive auctions when the difference between the best and second-best solution may be large, and in some edge cases in which a solver may benefit from making the filtering step less stringent. However, in these cases, deriving the optimal strategy is a very complex problem.

Consistency rewards introduce an additional strategic dimension; for example, the order-count metric rewards solvers for submitting solutions to auctions in which orders are ultimately executed, solvers have an incentive to participate broadly across auctions, even in cases where they do not expect to win the performance reward.
Consistency rewards introduce an additional strategic dimension; since the consistency metric rewards competitive bids on executed orders, solvers have an incentive to participate broadly across auctions with bids close to the winning one, even in cases where they do not expect to win the performance reward. At the same time, since bid quality is discounted by the settlement success rate, solvers should only submit bids they are prepared to settle.

## Price estimation competition rewards (CIPs 27, 36, 57, 72)

Expand Down
Loading