Skip to content

test(sessions): 新增 Session/Memory/Summary 多后端回放一致性测试框架#153

Open
guocfu wants to merge 1 commit into
trpc-group:mainfrom
guocfu:feature/session-replay-test
Open

test(sessions): 新增 Session/Memory/Summary 多后端回放一致性测试框架#153
guocfu wants to merge 1 commit into
trpc-group:mainfrom
guocfu:feature/session-replay-test

Conversation

@guocfu

@guocfu guocfu commented Jul 10, 2026

Copy link
Copy Markdown

Session / Memory / Summary Replay Consistency 设计说明

Description | 描述

本变更为 Session / Memory / Summary 后端新增 replay consistency harness。框架会用同一组确定性 JSONL replay case 驱动后端执行 Session、Memory、Summary 操作,读取后归一化为稳定 snapshot,并输出结构化 diff report,用于发现不同存储后端之间的事件顺序、state、memory、summary 行为漂移。

Closes #89

Files Changed

  • tests/sessions/replay_consistency/:模块化 replay consistency 框架,包括 case loader、fixture builder、normalizer、comparator、reporter、backend factory 和 assertions。
  • tests/sessions/replay_consistency/replay_cases/:10 个 JSONL replay fixture,覆盖单轮文本、多轮追加、工具调用与响应、state 覆盖、memory 写入读取、summary 生成/更新、summary 截断、异常恢复、重复写入和分支 metadata。
  • tests/sessions/test_replay_consistency.py:主 E2E replay 测试,覆盖后端矩阵、snapshot 归一化比较、summary 内容/metadata 检查、diff report fixture。
  • tests/sessions/test_replay_mutations.py:真实 replay snapshot mutation 检测,验证事件、state、tool call、memory、summary 丢失/篡改/覆盖/归属错误能被检出。

Backend Behavior | 后端行为

默认本地模式不要求外部 SQL / Redis,只运行 InMemory 轻量 replay。设置 TRPC_AGENT_REPLAY_SQL_URL 后尝试外部 SQL 后端;设置 TRPC_AGENT_REPLAY_REDIS_URL 后尝试外部 Redis 后端。外部 SQL 不可用时降级到临时 SQLite,外部 Redis 不可用时降级到 mock Redis,避免本地或 CI 因缺少真实 MySQL/Redis 失败。

Comparison Strategy | 比较策略

normalizer 会固定时间戳、自动生成 ID、序列化顺序等非业务字段,对事件内容、工具参数/响应、state 值、memory 文本/作用域、summary 文本和 summary 归属保持严格比较。Summary 比较区分内容语义和存储 metadata:摘要文本做规范化比较,session_id、manager session、summary event count、compressed count 等 metadata 严格比较。真实 summary 路径使用 deterministic fake summarizer,避免真实 LLM 随机性。

Reports | 报告

已提交示例报告:

  • tests/sessions/replay_consistency/session_memory_summary_diff_report.json
    • normal replay report
    • 10 个 replay case
    • 记录 backend pair/status、case 级 diff 统计、session id、event index、summary id、字段路径以及左右值
    • 默认 InMemory-only 轻量模式下无跨后端比较;配置 SQL/Redis URL 后生成对应后端比较结果

Test Coverage | 测试覆盖

  • 10 个 replay case fixture 完整性检查。
  • Session replay E2E:事件、state、工具调用、summary 生成/更新/截断。
  • Memory replay E2E:memory 写入、读取、文本和作用域比较。
  • Summary 专项:summary 丢失、stale/错误 overwrite、绑定到错误 session。
  • Mutation tests:向归一化后的 replay snapshot 注入故障,并验证 precise diff context。

Self-test Checklist | 自测清单

  • python -m json.tool tests/sessions/replay_consistency/session_memory_summary_diff_report.json
  • python -m pytest tests/sessions/test_replay_consistency.py tests/sessions/test_replay_mutations.py -q
  • $env:TRPC_AGENT_REPLAY_SQL_URL='sqlite:///:memory:'; python -m pytest tests/sessions/test_replay_consistency.py -q
  • $env:TRPC_AGENT_REPLAY_REDIS_URL='redis://localhost:6379/15'; python -m pytest tests/sessions/test_replay_consistency.py -q

@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@cbb6979). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             main        #153   +/-   ##
==========================================
  Coverage        ?   87.53692%           
==========================================
  Files           ?         467           
  Lines           ?       44018           
  Branches        ?           0           
==========================================
  Hits            ?       38532           
  Misses          ?        5486           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@guocfu

guocfu commented Jul 10, 2026

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@guocfu

guocfu commented Jul 10, 2026

Copy link
Copy Markdown
Author

recheck

@guocfu guocfu force-pushed the feature/session-replay-test branch from 0b98617 to c796fd0 Compare July 10, 2026 04:32
Rook1ex added a commit to trpc-group/cla-database that referenced this pull request Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

构建 Session / Memory 多后端回放一致性测试框架

1 participant