Timestamp index silently drops in-range messages when timestamps carry different UTC offsets

[KRRT7]

Severity: High — wrong search results, fails silently
Area: storage/memory/timestampindex.py, emails/email_import.py

Summary

TimestampToTextRangeIndex sorts and bisects ISO-8601 strings, relying on the
assumption that lexicographic order equals chronological order. That assumption
only holds if every timestamp in the index renders with the same UTC offset
suffix. Email ingestion produces timestamps with the sender's own offset, so a
mailbox with senders in different timezones yields an index where the ordering
invariant is broken and range lookups return the wrong set of messages — with no
error.

Reproduction

import asyncio
from datetime import timezone, timedelta
from typeagent.knowpro.interfaces import DateRange, Datetime
from typeagent.storage.memory.timestampindex import TimestampToTextRangeIndex

idx = TimestampToTextRangeIndex()
# 12:00 in +05:00 == 07:00 UTC
msg = Datetime(2024, 1, 1, 12, 0, 0, tzinfo=timezone(timedelta(hours=5)))
asyncio.run(idx.add_timestamp(0, msg.isoformat()))

start = Datetime(2024, 1, 1, 6, 0, 0, tzinfo=timezone.utc)
end   = Datetime(2024, 1, 1, 8, 0, 0, tzinfo=timezone.utc)
print(start <= msg < end)                                    # True
print(asyncio.run(idx.lookup_range(DateRange(start=start, end=end))))

stored  : 2024-01-01T12:00:00+05:00  (== 2024-01-01T07:00:00+00:00)
query   : [2024-01-01T06:00:00+00:00, 2024-01-01T08:00:00+00:00)
in range chronologically? True
lookup_range -> []

Expected: the message is returned.
Actual: empty result.

Root cause

storage/memory/timestampindex.py:104-108 stores the timestamp as
Datetime.fromisoformat(timestamp).isoformat(), preserving whatever offset the
input carried:

timestamp_datetime = Datetime.fromisoformat(timestamp)
entry = TimestampedTextRange(
    range=text_range_from_message_chunk(message_ordinal),
    # This string is formatted to be lexically sortable.
    timestamp=timestamp_datetime.isoformat(),
)

The comment on line 107 states the intent, but nothing enforces it. Insertion
(bisect_left, line 111) and lookup (_lookup_range, lines 58-66, via
get_in_range) both compare these strings directly.

"2024-01-01T12:00:00+05:00" sorts after "2024-01-01T08:00:00+00:00"
lexicographically but before it chronologically.

Offsets other than UTC reach the index in practice:

# emails/email_import.py:103
timestamp = parsedate_to_datetime(timestamp_date).isoformat()

'Mon, 1 Jan 2024 12:00:00 +0500' -> '2024-01-01T12:00:00+05:00'
'Mon, 1 Jan 2024 10:00:00 -0800' -> '2024-01-01T10:00:00-08:00'

A related boundary case: a naive stored timestamp ("...T00:00:00") is a strict
prefix of an aware query bound ("...T00:00:00+00:00"), so it sorts before it
and a message sitting exactly on the range start is excluded.

Suggested fix

Normalize to a single representation on both the write and the read path — e.g.
convert to UTC and render with a fixed suffix in _insert_timestamp, and apply
the same conversion to date_range.start/date_range.end in _lookup_range.
universal_message.format_timestamp_utc already does exactly this conversion
(and rejects naive datetimes); reusing it would make the two paths agree by
construction.

Worth also asserting the invariant rather than only commenting it: a debug check
that self._ranges is sorted after insert would have caught this.