tests: add Session/Memory/Summary multi-backend replay consistency testing framework#152
Open
DNKYr wants to merge 7 commits into
Open
tests: add Session/Memory/Summary multi-backend replay consistency testing framework#152DNKYr wants to merge 7 commits into
DNKYr wants to merge 7 commits into
Conversation
Add DESIGN.md (English) and DESIGN.zh_CN.md (Chinese) for the Session/Memory/Summary multi-backend replay consistency testing framework. Covers normalization strategy, summary comparison approach, allowed difference rules, backend integration, and harness self-testing strategy. Updates trpc-group#89 RELEASE NOTES: NONE
Add ReplayOp / ReplayCase Pydantic models and 10 standardized replay trajectory files covering: - 01 single-turn chat - 02 multi-turn conversation - 03 tool call (function_call + function_response) - 04 state update (write and overwrite) - 05 memory store and search - 06 summary generation and retrieval - 07 summary with event truncation - 08 error recovery (duplicate append) - 09 injected anomaly (extra event detection) - 10 injected anomaly (state corruption detection) Updates trpc-group#89 RELEASE NOTES: NONE
Implement normalization utilities that strip auto-generated and backend-dependent fields (timestamps, UUIDs, invocation IDs) from events, state, summaries, and memory entries before comparison. 24 unit tests cover: event field stripping, timestamp-to-index replacement, function_call/response extraction, state deep-sort determinism, summary timestamp removal, memory entry normalization, and the full normalize_backend_result orchestrator. Updates trpc-group#89 RELEASE NOTES: NONE
Implement pairwise comparison of normalized backend results with field-level precision. Covers events (author, text, function calls, responses, state deltas), state (deep recursive diff), summaries (loss, overwrite, wrong-session-affiliation detection), and memory. Includes AllowedDiff rules to suppress known-acceptable backend differences via pattern matching. 30 unit tests verify: identical detection, count mismatches, field- level diffs, event-index precision, nested state diff paths, summary loss / overwrite / mis-affiliation, memory diffs, and allow-rule suppression with backend-pair scoping. Updates trpc-group#89 RELEASE NOTES: NONE
Implement DiffReport generation with run metadata, per-case results, and aggregate summary statistics including false-positive rate calculation. Supports JSON serialization and file output via write_report. 21 unit tests cover: ReportMetadata defaults, CaseResult fields, ReportSummary, false-positive rate computation (edge cases for no-passing, all-pass-no-diffs, mixed, failing-only), generate_report with empty/all-pass/all-fail/mixed scenarios, report_to_dict JSON round-trip, and write_report file creation. Updates trpc-group#89 RELEASE NOTES: NONE
Implement ReplayEngine that executes replay cases against session and memory backends, with support for: - Building Events from replay ops (text, function calls, responses) - Summary injection into SummarizerSessionManager cache - Memory store/search integration - Duplicate append error recovery simulation - Anomaly injection for detection verification Add test_replay_consistency.py with 11 E2E tests: - 10 parametrized tests running all replay cases through InMemory vs SQL (sqlite) backend pair - 1 test generating the session_memory_summary_diff_report.json All 87 tests pass (24 normalizer + 30 comparator + 22 report + 11 E2E). Diff report shows 0% false positive rate on normal cases and 100% detection on injected anomaly cases. Fix false-positive-rate calculation to exclude anomaly cases where diffs are expected. Updates trpc-group#89 RELEASE NOTES: NONE
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
Author
|
I have read the CLA Document and I hereby sign the CLA |
Rook1ex
added a commit
to trpc-group/cla-database
that referenced
this pull request
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Resolved #89
Summary
Build a replay consistency harness that validates InMemory, SQL, and Redis
backends produce equivalent results when driven by identical operation sequences.
Drives the same trajectory through multiple backends and generates a structured
diff report with field-level precision.
Changes
Design notes
docs/replay_harness/DESIGN.md(English) andDESIGN.zh_CN.md(Chinese)backend integration.
Replay cases (10 JSON trajectory files)
tests/sessions/replay_cases/_base.py—ReplayOp/ReplayCasedata modelswith JSON loader.
Replay harness (3 modules + unit tests, 75 tests)
Normalizer (
_normalizer.py+ 24 tests)Strips auto-generated fields (timestamps, UUIDs, invocation IDs) and replaces
timestamps with sequential indices for deterministic comparison.
Comparator (
_comparator.py+ 30 tests)Pairwise field-level diff across events, state, memory, and summaries. Detects
summary loss, overwrite errors, and wrong-session-affiliation. Supports
AllowedDiffrules for known-acceptable backend differences.Report (
_report.py+ 22 tests)Generates
session_memory_summary_diff_report.jsonwith run metadata, per-caseresults, aggregate statistics, and false-positive rate (excluding anomaly cases).
E2E tests (11 tests)
Results
TRPC_REDIS_URLenv varBackward Compatibility
No API changes. All new code lives under
tests/sessions/. No effect on theproduction SDK.