Skip to content

test(fuzz): add versioned trace replay - #24

Draft
dangduc wants to merge 8 commits into
jojojames:mainfrom
dangduc:codex/fzfa-fuzz-versioned-traces
Draft

test(fuzz): add versioned trace replay#24
dangduc wants to merge 8 commits into
jojojames:mainfrom
dangduc:codex/fzfa-fuzz-versioned-traces

Conversation

@dangduc

@dangduc dangduc commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

What this adds

This is a follow-up to the three fuzz-lane PRs in #21, #22, and #23. It gives
all seven fuzz targets one versioned trace format, exact file-based replay, and
replayable CI failure artifacts.

There are no runtime changes. This PR changes only fuzz/ and the fuzz
workflow.

This PR is based on #23, which is based on #22 and #21. Until those PRs merge,
GitHub's full diff includes the earlier fuzz lanes. Review this PR's
incremental change here:

dangduc/fzfa@codex/fzfa-fuzz-live-icomplete...codex/fzfa-fuzz-versioned-traces

Why save a trace instead of only a seed?

A seed tells a generator how to make a case. It is not the case itself.

For example, seed 123 may make this today:

fetch "a"
fetch "ab"
deliver callback 0 with ("old")
stop

If the generator changes next month, seed 123 may make different actions. A
saved trace keeps the original actions, candidate strings, byte chunks, timing
choices, and settings. The driver reads those values directly. It does not ask
the random generator to choose them again.

Seed replay is still useful for exploring nearby cases. A trace is the better
answer to: “Can I run the exact failing input again?”

What a trace records

Format 1 is an Emacs Lisp data file with:

  • the target name;
  • the campaign seed and case seed;
  • Emacs, OS, locale, fzfa, and fzf-native versions and Git revisions;
  • the complete initial state; and
  • the ordered actions sent to the driver.

The seven target drivers are:

  1. completion-list ownership;
  2. Elisp producer lifecycle;
  3. stale poll publication;
  4. message ownership;
  5. native producer bytes;
  6. the ugrep fixture; and
  7. live icomplete input.

fuzz/TRACE_FORMAT.md documents each target with a small example. Format 1
rejects missing, repeated, or unknown top-level fields. Each driver also checks
its own initial state and action shapes.

The reader disables reader evaluation and circular reader syntax, rejects a
second form after the trace, and does not execute Lisp from the file. The ugrep
driver also rejects absolute fixture paths and .. paths that would leave its
temporary directory.

Simple examples

A late producer callback

The saved trace contains the actual schedule:

(:initial-state (:source-name "state" :step-budget 5)
 :actions
 ((fetch "a")
  (fetch "ab")
  (deliver 0 ("old"))
  (stop)))

Replay runs those four actions in that order. It does not regenerate a callback
selector or a candidate list from the seed.

The driver copies the delivered candidates before giving them to production
code. A controlled mutation can therefore change the delivered list without
changing the trace that will be written to disk.

Bytes split in the middle of café

The native trace stores raw bytes as hexadecimal text:

((emit :bytes-hex "636166c3" :pause 0.01)
 (emit :bytes-hex "a90a" :pause 0.01)
 (exit 0)
 (poll-until-terminal)
 (redraw 3)
 (stop))

The first write ends after the first byte of é. The second write starts with
the remaining byte. Hexadecimal storage also preserves malformed UTF-8, NUL,
CRLF, and partial ANSI sequences without asking the current locale to decode
them first.

Live keys wait for real renders

A live action stores both the key and the query that must appear afterward:

((key 97 "a")
 (key 108 "al")
 (key 127 "a")
 (key 127 "")
 (key 13 ""))

The next key is released only after the expected query has rendered. This
preserves the causal handshake instead of replacing it with sleeps.

Ambient policy stays ambient

If a producer case used the ambient line limit, the trace records both:

(:max-line-source ambient :max-line-length 4)

Replay restores the value 4 around the case but still leaves the per-case
option absent. That keeps testing the bridge from fzfa's ambient setting into
fzf-native. An explicit nil remains different: it means unlimited.

Failure artifacts and stable signatures

An oracle failure wraps the executable trace with:

(:artifact-format 1
 :failure
 (:signature "state-producer/0123456789abcdef"
  :oracle "snapshot is different from expected after %S"
  :seed 127
  :message "snapshot is different from expected after (deliver ...)"
  :expected (...)
  :observed (...))
 :trace (:format 1 ...))

The human message may contain the generated values. The signature for a normal
oracle uses the target and the oracle's stable format string, so two failures
of the same assertion group together. An unexpected condition also includes
its condition type and message in the signature so a different crash does not
count as the same reproduction.

Replaying a plain trace succeeds when the driver completes. Replaying a failure
artifact succeeds only when the recorded signature fails again. It reports an
error if the case now passes or a different assertion fails first.

Commands

# Check the format, raw-byte round trip, strict reader, no-RNG replay,
# and same-signature reproduction.
make -C fuzz trace-selftest

# Replay a state, producer, or ugrep trace or failure artifact.
make -C fuzz replay-trace TRACE=/path/to/failure.sexp

# Replay a live trace through a real terminal minibuffer.
make -C fuzz replay-trace-live \
  TRACE=/path/to/live-failure.sexp LIVE_EMACS_FLAGS=-nw

Replay prints the environment saved in the trace. If the current environment
is different, it prints that environment too.

CI behavior

Each lane gets a separate artifact path. When a lane fails, the workflow
uploads the .sexp file:

  • fzfa-state-trace-<emacs version>;
  • fzfa-producer-trace; or
  • fzfa-live-trace-<emacs version>.

The upload step ignores a missing file. This matters for failures that happen
before a trace can run, such as a compiler or dependency failure.

Harness defects found while building this

The trace self-audit found and fixed test-harness defects before publication:

  • native :expected-present fields initially held a plist tail instead of the
    documented boolean t;
  • an automatic first icomplete render could consume a saved key while the setup
    timer still occupied the delivery slot; and
  • the producer alias canary could mutate the candidate list inside the trace,
    which would have changed the saved reproducer.

The self-tests now check boolean presence markers, reserve the live timer slot
before consuming a key, and require producer execution to leave its trace
unchanged.

Verification performed on a0511f7

  • all fuzz files byte-compiled with warnings treated as errors;
  • trace and artifact round trips, exact raw-byte round trip, reader-eval and
    circular-form rejection, target-shape checks, and unsafe ugrep path rejection;
  • exact replay with the random-choice functions replaced by errors;
  • a saved state trace replayed through make replay-trace;
  • a saved live trace replayed through make replay-trace-live in terminal Emacs;
  • a saved failure artifact reproduced the same stable signature;
  • eight state, five native/tool, and three live controlled defects were rejected;
  • 300 generated state cases x 40 steps, root seed 1;
  • 300 generated native producer cases, root seed 1;
  • eight generated live terminal cases, root seed 1; and
  • the full repository suite: 261 expected, 0 unexpected.

The local make tools run skipped because ugrep is not installed. The Linux CI
job installs ugrep and remains the execution gate for that integration.

Stack

@dangduc

dangduc commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Final verification for a0511f7:

  • Fuzz workflow: 7/7 jobs passed. This covers state fuzzing and live icomplete fuzzing on Emacs 29.1, 30.1, and snapshot, plus the native producer and installed-ugrep boundary job.
  • Test workflow: 6/6 jobs passed. This covers Ubuntu, macOS, and Windows on Emacs 30.1 and snapshot.
  • Local checks passed: 261 ERT tests; trace self-tests; 300 state cases with 40 steps each; 300 producer cases; exact file and live replay; and reproduction of a generated failure with the same stable signature.

The local machine did not have ugrep, so that local target was skipped. The CI job installed ugrep and exercised the boundary successfully.

No regressions were detected within this test envelope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant