Skip to content

test(fuzz): exercise live icomplete resizing - #23

Draft
dangduc wants to merge 7 commits into
jojojames:mainfrom
dangduc:codex/fzfa-fuzz-live-icomplete
Draft

test(fuzz): exercise live icomplete resizing#23
dangduc wants to merge 7 commits into
jojojames:mainfrom
dangduc:codex/fzfa-fuzz-live-icomplete

Conversation

@dangduc

@dangduc dangduc commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

What this adds

This is stack PR 3 of 3. It drives a real icomplete-vertical minibuffer through
Emacs's command loop and checks the window after actual renders.

Batch Emacs cannot enter a live minibuffer, so CI runs terminal Emacs inside a
pseudo-terminal supplied by script.

There are no runtime changes in this PR.

This PR is based on #22, which is based on #21. Until those PRs merge,
GitHub's full diff includes the earlier commits. Review this PR's incremental
change here:

dangduc/fzfa@codex/fzfa-fuzz-native-producers...codex/fzfa-fuzz-live-icomplete

Simple example

One case starts with a target such as alpha and many distractors:

alpha
xxxxx-00
xxxxx-01
...

It then does this through Emacs's real input loop:

open the fzfa completion
wait for the empty query to render
type a
wait for "a" to render
type l
wait for "al" to render
continue through "alpha"
delete one character at a time, waiting after each deletion
wait for a new empty-query render
press RET

Cases choose among alpha, quartz, violet, and mango, and vary the
number of distractors.

What the oracle checks

The test records each fit call after the real function runs. One observation
contains:

  • a sequence number;
  • the current query;
  • the logical candidates in icomplete's cache;
  • candidate and displayed-line counts;
  • mini-window height before, requested target height, and height afterward;
  • whether the minibuffer still owns an fzfa session; and
  • the plain displayed string for failure diagnosis.

The handshake has three checkpoints.

1. Initial empty query

The logical candidates must equal the complete input set, the display must use
more than one line, and the real fit must satisfy:

height before < requested height <= height after

This distinguishes a working fit from a no-op. In the historical failure
shape, a one-line mini-window could remain short even when icomplete had many
rows to display.

2. Full narrow query

The target must still be present, at least one candidate must remain, and the
logical set must be smaller than the initial set. This prevents a display that
ignores the query from passing just because it has multiple lines.

3. Fresh empty query after deletion

The observation's sequence number must be newer than the narrow observation.
The full candidate set and multiline display must be restored, and the window
must be at least as tall as it was after the initial fit. Production uses a
grow-only session policy, so the contract is “restore the display without
collapsing,” not “shrink and grow again.”

Keys are queued only after the expected query has rendered. There is no fixed
sleep between keys. A watchdog sends C-g and reports the current phase and
expected query if the handshake stops making progress.

Oracle qualification

Before generated cases run, three controlled canaries use the real minibuffer:

  1. skip the mini-window fit call;
  2. replace the narrow observation with the complete candidate set; and
  3. replace the restored empty observation with the stale narrow set.

The intended oracle must reject each canary. This qualifies FZFA-C14 in the
contract catalog. It does not cover Vertico, Emacs 31's built-in eager
*Completions* frontend, or pixel-level rendering.

The temporary observer advice is removed after every case. Each ordinary case
also requires fzfa's icomplete advice count and membership to return to their
starting values. A separate check installs the advice twice and removes it
twice to exercise nested-session reference counting.

CI failure output

CI writes the final live result to a file outside the pseudo-terminal, always
prints that file, and then returns script's original exit status. A failure
includes its seed, target query, generated event list, and every recorded
observation, so the case can be replayed without reading terminal control
sequences.

CI and local use

Pull requests run eight live sessions on Emacs 29.1, 30.1, and snapshot. The
weekly job runs 40 sessions on each version.

make -C fuzz live LIVE_EMACS_FLAGS=-nw
FZFA_FUZZ_SEED=123 make -C fuzz live LIVE_CASES=1 LIVE_EMACS_FLAGS=-nw

Verification performed

  • all three controlled live defects rejected through a real pseudo-terminal;
  • four generated live sessions, root seed 1;
  • fuzz harness byte compilation with warnings treated as errors; and
  • the full aggregate repository suite: 261 expected, 0 unexpected.

Stack

@dangduc
dangduc force-pushed the codex/fzfa-fuzz-live-icomplete branch 2 times, most recently from 1df9a26 to cd7760e Compare September 3, 2026 03:12
@dangduc

dangduc commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

I ran a two-round adversarial review of the new fuzz oracles using six independent, persona-inspired review lenses. Each item below was reproduced with a temporary local mutation and then challenged by another reviewer. These are test-harness or CI-observability issues, not claims that the current production code is broken.

Requested before merge

1. Tie the live observations to the behavior being tested

fzfa-fuzz-live--case (fuzz/fzfa-fuzz-live.el:175-234) accepts any nonempty-query observation followed by an empty query with multiple lines, then checks only after >= target. It never proves that alpha narrowed the display, that the later empty observation is a fresh render, or that fzfa--icomplete-fit-mini-window changed the height.

Both halves fail negative controls:

  • Returning the complete collection for every query—so alpha did not narrow anything—still passed.
  • Making fzfa--icomplete-fit-mini-window a no-op still passed 8 live PTY cases because the selected later observation was already tall.
  • With a discriminating fixture, alpha displayed 3 lines and the later empty query still displayed the same stale 3 lines after a delay; that also passed because the predicate requires only more than one line.

There is a deterministic causal check that matches production's grow-only policy: enable matches on no input, wait for the initial empty-input exhibit, and require before < target <= after. Locally the normal helper produced 1 -> 10, while the no-op helper stayed 1 -> 1. Then type alpha and require a smaller displayed set; after deleting it, require a fresh empty render that restores the full/multiline set while the window does not collapse.

Please use an observation-driven handshake rather than a fixed sleep. Since production intentionally pins the session to grow-only, do not require a shrink-then-regrow height cycle; test initial growth and later no-collapse/display recovery as separate contracts. Keep the no-narrowing and no-fit mutations as negative controls.

2. Preserve live failures and their seeds in CI

The runner supports FZFA_FUZZ_RESULT_FILE, but .github/workflows/fuzz.yaml:111-122 neither sets nor prints it. A forced failure at seed 424242 returned nonzero through script, but the captured terminal transcript contained neither the error nor the seed; it showed only make's exit line.

Please set a result file, capture script's exit status, always print the file, and then return the original status. Also include the event/observation trace in the recorded failure so a CI failure can be replayed directly.

Nonblocking follow-up

Every case currently types the same alpha, five DELs, and RET; seeds vary only filler names/counts. Once the causal assertions are fixed, either keep one or two explicit boundary cases per Emacs version or generate genuinely different transitions. Increasing the same shape from 8 to 40 cases adds runtime but little behavioral diversity.

Baseline live runs passed on the tested Emacs versions before mutations, including 40 local PTY cases. The isolated worktree was restored and verified clean after every probe.

@dangduc
dangduc force-pushed the codex/fzfa-fuzz-live-icomplete branch from cd7760e to 0ad940b Compare September 3, 2026 09:14
@dangduc
dangduc force-pushed the codex/fzfa-fuzz-live-icomplete branch from 0ad940b to aa1e8bd Compare September 3, 2026 13:24
@dangduc

dangduc commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Phase-two oracle work is now on this branch at aa1e8bd.

The requested findings are addressed:

  • Input is observation-driven: the next key is queued only after the expected
    query has rendered. There is no fixed inter-key sleep.
  • Each observation records a sequence number and the logical candidate set
    from icomplete's cache, plus query, line count, heights, session state, and
    plain display text.
  • The initial empty render must show the full multiline set and satisfy
    before < target <= after. The full query must reduce the set. The later
    empty render must be newer, restore the full set, and not collapse the
    grow-only window.
  • Three real-minibuffer canaries skip fitting, fake an unfiltered narrow set,
    or reuse stale narrow candidates for the final empty render. The intended
    oracle rejects all three.
  • CI now always prints the out-of-terminal result file and returns script's
    original status. Failures include the seed and complete observation trace.

The canaries and four generated cases pass locally through a pseudo-terminal.
CI passes for the live lane on Emacs 29.1, 30.1, and snapshot, and the aggregate
261-test ERT suite passes.

This qualifies the icomplete contract C14. Vertico, Emacs 31's built-in eager
completion frontend, and pixel-level rendering remain outside this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant