Cohort discipline: the agent's join shape and its choice of k both change the answer

[rohan-hotdata]

Two measured behaviours where the agent produces valid SQL that answers the wrong question, or
picks a number that decides the conclusion. Neither is a code bug. Both need a decision about what
we tell the model before anyone writes a fix, which is why they are together.

Both were measured with one model on one dataset. Treat the shapes as real and the
frequencies as indicative.

1. Multi-way breakdowns fan out — 2 of 5 runs wrong, 1 of 5 answers a different question

Asked to break a cohort-vs-population comparison down by neighbourhood, across five fresh threads:

Runs	Shape	Correct?
2/5	join two row-level relations on neighbourhood_cleansed, then GROUP BY	no
1/5	aggregated the population only, dropped the cohort entirely	no
2/5	UNION ALL of two independently aggregated sides	yes

The failing shape joins on a non-unique key before aggregating, so every population row is
duplicated once per cohort row in the same group. Verified exactly: Bernal Heights has 237 rated
listings and 25 in the cohort, and the query reported n_listings_all = 5925 = 237 × 25.

The nasty part: the averages stay correct, because uniform duplication preserves a mean. Only
the counts are wrong, so the output looks plausible and is half right.

This is not caused by making bm25_search composable, but it is invited by it. With ids in hand
the model aggregated correctly; thinking in relations, it reaches for a join. Both correct samples
used UNION ALL over independently aggregated sides. Aggregating each side first and then joining
on the group key would also be correct and was never tried.

Open question: should the SQL tool description name the safe shape? Untested — and there is a real
risk of over-instructing a model that gets it right 40% of the time unaided.

2. k is chosen arbitrarily and the conclusion moves with it

k came out as 100, 200, 500 and 5000 across runs — the last exceeding the corpus size. And the
measured effect depends on it:

Cohort	Avg rating
k=100, tight matches	4.8236
k=500	4.8023
population	4.8029

So the conclusion flips from "quiet-garden listings rate higher" to "no difference" purely on the
model's choice of k. BM25 k is a hard cutoff, not a relevance threshold, so the model is
choosing the answer when it chooses the cohort boundary — with nothing to base it on.

More damaging to credibility than the fan-out, because every individual run looks confident and
internally consistent.

Options, none obviously right:

• Guidance on k in the tool description.
• A score threshold instead of, or alongside, a hard k.
• Surfacing the score distribution so a cohort boundary can be justified rather than guessed.
• Accepting it, and requiring that any demo or doc built on a cohort states its k.

Suggested approach

Run the experiment before changing anything. Both problems are description-level hypotheses, and
we already have a repeatable harness downstream (same six questions, fresh thread each time). A
wording change that is not measured against the current 40% baseline is not a fix, it is a guess.

Related: #58 (retrieval eval harness) would give this a proper measurement surface, though these
two are agent-behaviour questions rather than retrieval-quality ones.

[rohan-hotdata]

k saturates, which gives BM25 a principled cohort boundary. Measured 2026-08-13.

This issue treats k as an arbitrary knob with nothing to base it on. That is right for small
k, but there turns out to be a real boundary in the data, and it changes what the fix should
be.

Because the BM25 score does not depend on k, the ranked list is fixed and k only slides the
cutoff — the top-50 is a prefix of the top-500. So the numbers across runs are the cumulative
mean of one sequence, read wherever the model happened to cut it, not independent samples:

k	cohort avg	n rated
50	4.7900	49
100	4.8236	94
200	4.8282	183
500	4.8023	446
1,426 — the whole match pool	4.8311	1,266
population	4.8029	5,862

Non-monotonic: below the population, above, above, level, above.

BM25 is sparse, so the list ends. Only documents containing a query term score at all.
bm25_search(..., 'quiet garden', k) returns 1,426 rows for any k ≥ 1426 (scores
1.2189–6.7374) out of 7,535. Above the pool the cutoff is inert, so the cumulative mean
converges on the match-pool mean, not the population mean.

Two things follow:

1. k at or above saturation is defensible where every smaller k is an arbitrary slice.
   Reporting the pool size — or the saturation point — would let a model ask for the whole
   cohort deliberately instead of guessing. That is a concrete option this issue did not have.
2. The effect is real. At saturation the cohort is 4.8311 on 1,266 rated listings versus
   4.8029 on 5,862 — the best-powered comparison available, and it sits above the population.
   An earlier reading, that BM25 dilutes a small-cohort artifact as k grows, does not survive.

A score threshold is still the interesting knob, because the match pool is documents
containing quiet or garden — a lexical boundary, not the concept asked about.
bm25_search already returns score, so WHERE score > <x> is expressible today; BM25 scores
are not normalised, so what a defensible threshold is per query and corpus is unmeasured.
[hypothesis]

The vector side has no such escape — every document has a distance, so there the cumulative
mean does converge on the population mean as k grows, and a 500-document corpus is small
enough for a model's k to reach it. Untested. [hypothesis] Relevant to #39.

Also measured: the wasted search call persists

Three of three turns called hotdata_search_text first, ignored the result, then composed
bm25_search in SQL anyway. One of those calls came back capped at 100 rows while the query
written next used k=500. So the "prefer this whenever the answer aggregates" wording changes
the SQL but does not stop the redundant tool call — which is finding 9's open question about
whether the row-returning tool and the SQL tool should coexist, now with evidence.

Source: an internal exploratory run log Run 4.