test(fuzz): add versioned trace replay - #24
Draft
dangduc wants to merge 8 commits into
Draft
Conversation
Collaborator
Author
|
Final verification for
The local machine did not have ugrep, so that local target was skipped. The CI job installed ugrep and exercised the boundary successfully. No regressions were detected within this test envelope. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
This is a follow-up to the three fuzz-lane PRs in #21, #22, and #23. It gives
all seven fuzz targets one versioned trace format, exact file-based replay, and
replayable CI failure artifacts.
There are no runtime changes. This PR changes only
fuzz/and the fuzzworkflow.
This PR is based on #23, which is based on #22 and #21. Until those PRs merge,
GitHub's full diff includes the earlier fuzz lanes. Review this PR's
incremental change here:
dangduc/fzfa@codex/fzfa-fuzz-live-icomplete...codex/fzfa-fuzz-versioned-traces
Why save a trace instead of only a seed?
A seed tells a generator how to make a case. It is not the case itself.
For example, seed 123 may make this today:
If the generator changes next month, seed 123 may make different actions. A
saved trace keeps the original actions, candidate strings, byte chunks, timing
choices, and settings. The driver reads those values directly. It does not ask
the random generator to choose them again.
Seed replay is still useful for exploring nearby cases. A trace is the better
answer to: “Can I run the exact failing input again?”
What a trace records
Format 1 is an Emacs Lisp data file with:
The seven target drivers are:
fuzz/TRACE_FORMAT.mddocuments each target with a small example. Format 1rejects missing, repeated, or unknown top-level fields. Each driver also checks
its own initial state and action shapes.
The reader disables reader evaluation and circular reader syntax, rejects a
second form after the trace, and does not execute Lisp from the file. The ugrep
driver also rejects absolute fixture paths and
..paths that would leave itstemporary directory.
Simple examples
A late producer callback
The saved trace contains the actual schedule:
Replay runs those four actions in that order. It does not regenerate a callback
selector or a candidate list from the seed.
The driver copies the delivered candidates before giving them to production
code. A controlled mutation can therefore change the delivered list without
changing the trace that will be written to disk.
Bytes split in the middle of
caféThe native trace stores raw bytes as hexadecimal text:
The first write ends after the first byte of
é. The second write starts withthe remaining byte. Hexadecimal storage also preserves malformed UTF-8, NUL,
CRLF, and partial ANSI sequences without asking the current locale to decode
them first.
Live keys wait for real renders
A live action stores both the key and the query that must appear afterward:
The next key is released only after the expected query has rendered. This
preserves the causal handshake instead of replacing it with sleeps.
Ambient policy stays ambient
If a producer case used the ambient line limit, the trace records both:
Replay restores the value 4 around the case but still leaves the per-case
option absent. That keeps testing the bridge from fzfa's ambient setting into
fzf-native. An explicit
nilremains different: it means unlimited.Failure artifacts and stable signatures
An oracle failure wraps the executable trace with:
The human message may contain the generated values. The signature for a normal
oracle uses the target and the oracle's stable format string, so two failures
of the same assertion group together. An unexpected condition also includes
its condition type and message in the signature so a different crash does not
count as the same reproduction.
Replaying a plain trace succeeds when the driver completes. Replaying a failure
artifact succeeds only when the recorded signature fails again. It reports an
error if the case now passes or a different assertion fails first.
Commands
Replay prints the environment saved in the trace. If the current environment
is different, it prints that environment too.
CI behavior
Each lane gets a separate artifact path. When a lane fails, the workflow
uploads the
.sexpfile:fzfa-state-trace-<emacs version>;fzfa-producer-trace; orfzfa-live-trace-<emacs version>.The upload step ignores a missing file. This matters for failures that happen
before a trace can run, such as a compiler or dependency failure.
Harness defects found while building this
The trace self-audit found and fixed test-harness defects before publication:
:expected-presentfields initially held a plist tail instead of thedocumented boolean
t;timer still occupied the delivery slot; and
which would have changed the saved reproducer.
The self-tests now check boolean presence markers, reserve the live timer slot
before consuming a key, and require producer execution to leave its trace
unchanged.
Verification performed on
a0511f7circular-form rejection, target-shape checks, and unsafe ugrep path rejection;
make replay-trace;make replay-trace-livein terminal Emacs;The local
make toolsrun skipped because ugrep is not installed. The Linux CIjob installs ugrep and remains the execution gate for that integration.
Stack