Python: fix: prevent superlinear history growth by deduplicating messages in save_messages

[PratikWayase]

Motivation & Context

1. Why is this change required?
   The current history providers (InMemoryHistoryProvider, FileHistoryProvider, and RedisHistoryProvider) blindly append messages on every service call without checking if they already exist in the store.
2. What problem does it solve?
   In looped runs (e.g., AG-UI stateless clients or harness todo loops), the transport passes the full accumulated conversation as input on every request. This caused the entire conversation history to be re-persisted every round, leading to superlinear store growth, corrupted context, repeated tool calls, and massive token bloat (up to 3× the real conversation size).
3. What scenario does it contribute to?
   Multi-iteration flows where each service call's message list carries the accumulated conversation rather than a delta, particularly when using stateless chat clients over the AG-UI endpoint.
4. Issue link: Fixes Python: [Bug]: per-service-call history persistence re-appends already-persisted messages #7211

Description & Review Guide

• What are the major changes?

  • Added a new _get_message_identity helper function that generates a stable identity for a message (using message.id if available, otherwise falling back to a deterministic hash of role + serialized contents).
  • Updated save_messages in InMemoryHistoryProvider, FileHistoryProvider, and RedisHistoryProvider to build a set of existing message identities and filter out duplicates before appending/pushing new messages.
  • Added comprehensive regression tests for all three providers to ensure identical messages are ignored, mixed old/new messages only append the new ones, and messages with the same text but different roles are kept separate.

• What is the impact of these changes?
  This hardens the system at the lowest storage level. It completely prevents duplicate message persistence regardless of how the middleware or transport layer constructs the message list. This stops token bloat, prevents context corruption, and ensures graceful degradation in long-running loops without requiring changes to the middleware routing logic.

• What do you want reviewers to focus on?

  1. The stability and correctness of the _get_message_identity fallback logic (ensuring it handles missing IDs and serialization edge cases gracefully).
  2. The RedisHistoryProvider.save_messages implementation, specifically ensuring it correctly fetches existing messages via lrange to build the identity set before pushing only the new_messages via the pipeline.
  3. The new unit tests to ensure they adequately cover the deduplication edge cases.

Related Issue

Fixes #7211

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

[Copilot]

Pull request overview

This PR fixes a Python history-persistence defect where per-service-call history providers re-persist the full accumulated conversation every round, causing superlinear growth and duplicated context. It introduces message-level deduplication in history providers (in-memory, file, Redis) and adds regression tests to ensure repeated inputs don’t bloat stored history.

Changes:

• Added a message-identity helper and used it to filter out already-persisted messages before appending/pushing.
• Updated save_messages behavior across in-memory, file (JSONL), and Redis history providers to skip duplicates.
• Added unit/regression tests covering identical-message deduplication and mixed old/new message lists.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
python/packages/core/agent_framework/_sessions.py	Adds message identity + deduplication to in-memory and file history providers.
python/packages/core/tests/core/test_sessions.py	Adds regression tests for deduplication behavior in core providers and looped runs.
python/packages/redis/agent_framework_redis/_history_provider.py	Adds message identity + deduplication to Redis history persistence.
python/packages/redis/tests/test_providers.py	Adds Redis-provider tests validating deduplication behavior.

[eavanvalkenburg]

Please check the failing tests, and make sure the comments have a reply or are marked as resolved @PratikWayase

[github-actions]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/core/agent_framework
_sessions.py	914	64	93%	161, 173–174, 198, 204–205, 243, 254, 268, 292, 337, 342, 344, 354, 386, 398, 408, 517, 586–587, 720, 1073–1077, 1092, 1122, 1159–1160, 1174, 1176, 1195, 1197, 1280, 1320, 1391, 1395, 1405, 1622, 1655–1656, 1661, 1676, 1754–1755, 1757, 1841, 1944, 2017, 2032, 2037–2039, 2063, 2082, 2085, 2093–2094, 2106–2107, 2119, 2129, 2159
packages/redis/agent_framework_redis
_history_provider.py	72	1	98%	205
TOTAL	47389	4535	90%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
9754	34 💤	0 ❌	0 🔥	2m 35s ⏱️

[eavanvalkenburg]

Still some checks failing @PratikWayase

[eavanvalkenburg]

and a new merge conflict @PratikWayase

[PratikWayase]

@eavanvalkenburg, @moonbox3 @giles17 the merge conflict seems to be addressed now. Lmk if there's anything else