A scanner that watches everything sees nothing

[fivestarspicy]

Changes

• Adds a draft guide to building Replay Vision scanners that produce trustworthy observations.
• Grounds the advice in PostHog's own scanner examples.
• Includes concrete prompts for PostHog AI and coding agents.

Authorship note: brain-written by Cory. AI helped edit, structure, and tighten parts of the draft.

Before publishing

• Set Cory Slater as the author.
• Add the hero image.
• Replace the screenshot placeholders.
• Refresh the observation counts and as-of date.
• Link the merged PR for the scanner-wizard fix.
• Review the Vercel preview.
• Get Editorial Team approval.

Checklist

• I've read the content style guide.
• Words use American English.
• Internal links use relative URLs.
• I've checked the page in the Vercel preview build.

---

Created with PostHog Desktop

[SaraMiteva]

Suggested change

	Nobody was going to watch the vast majority of those millions of recordings anyway. That is the problem [Natalia wrote about](/blog/nobody-watches-session-replays). Session replay contains useful evidence. Then the evidence sits in a giant list while everyone goes back to their dashboards.
	[Nobody was going to watch](/blog/nobody-watches-session-replays) the vast majority of those millions of recordings anyway. Session replay contains useful evidence. Then the evidence sits in a giant list while everyone goes back to their dashboards.

[SaraMiteva]

Suggested change

	Replay Vision fixes the watching part. It does not fix the thinking part.
	We learned this the annoying (but expected) way. Some scanners pointed us to bugs we could then verify in the linked recordings. Some showed us exactly where a flow became confusing. Others produced plausible descriptions of sessions that nobody wanted to read. The model was not the main difference. The scanner either had a real job or it did not.
	Replay Vision fixes the watching part but not the thinking part. We learned this the annoying (but expected) way. Some scanners pointed us to bugs we could then verify in the linked recordings, while others showed us exactly where a flow became confusing. Some even produced plausible session descriptions that nobody wanted to read.
	In fact, the model wasn't the main difference. The scanner either had a real job or it did not.

[SaraMiteva]

Suggested change

	Different surfaces with the same shape: a filtered set of recordings, one visible question, and a defined answer.
	The recording query and the prompt do different work. The query decides which sessions deserve inspection. The prompt decides what judgment to make inside each one. A high-intent event, relevant URL, useful cohort, or minimum duration usually improves quality more than another paragraph of instructions.
	Same shape on every surface: a filtered set of recordings, one visible question, and a defined answer. The recording query and the prompt do different work:
	- The query decides which sessions deserve inspection.
	- The prompt decides what judgment to make inside each one.
	A high-intent event, relevant URL, useful cohort, or minimum duration usually improves quality more than another paragraph of instructions.

[SaraMiteva]

Suggested change

	One watches first sessions from high-fit signups and writes the trip report we would never produce consistently by hand. What did the person evaluate? How far did they get? What slowed them down? Another reconstructs how people investigate Error Tracking issues. It records the entry point, data consulted, impact assessment, queries, actions, and outcome.
	One watches the first sessions from high-fit signups and writes the trip report we would never produce consistently by hand:
	- What did the person evaluate?
	- How far did they get?
	- What slowed them down?
	Another reconstructs how people investigate Error Tracking issues. It records the entry point, data consulted, impact assessment, queries, actions, and outcome.

[SaraMiteva]

Suggested change

	An event can say someone abandoned signup. An observation can show the contradictory copy they read before leaving. An event can say someone clicked summarize. An observation can show a populated trace that the product incorrectly summarized as empty. An event can say setup completed. An observation can show that the person got there only after painfully hunting, scrolling, and backtracking.
	- An event can say someone abandoned signup. An observation can show the contradictory copy they read before leaving.
	- An event can say someone clicked **summarize.** An observation can show a populated trace that the product incorrectly summarized as empty.
	- An event can say setup completed. An observation can show that the person got there only after painfully hunting, scrolling, and backtracking.

[SaraMiteva]

Suggested change

	Use Gemini 3.5 Flash Lite for high-volume jobs where you care about the distribution, not one observation. Our broken-render classifier is the obvious example. It tags recordings as `nothing_broken`, `media_failed`, `clipped_layout`, or `horizontal_overflow`. One wrong tag nudges a trend instead of opening a ticket.
	Use **Gemini 3.5 Flash Lite** for high-volume jobs where you care about the distribution, not one observation. Our broken-render classifier is the obvious example. It tags recordings as `nothing_broken`, `media_failed`, `clipped_layout`, or `horizontal_overflow`. One wrong tag nudges a trend instead of opening a ticket.

[SaraMiteva]

Suggested change

	Use Gemini 3 Flash when the job combines a fixed rubric with some judgment. An experiment classifier typically fits well here. It can tag each post-exposure recording as `smooth`, `hesitation`, `confusion`, `error_or_dead_end`, or `inconclusive`. The scanner judges one recording, and the Digest or Scout compares the pattern across variants.
	Use **Gemini 3 Flash** when the job combines a fixed rubric with some judgment. An experiment classifier typically fits well here. It can tag each post-exposure recording as `smooth`, `hesitation`, `confusion`, `error_or_dead_end`, or `inconclusive`. The scanner judges one recording, and the Digest or Scout compares the pattern across variants.

[SaraMiteva]

Suggested change

Use Gemini 3.7 Flash when a person may act on a single observation. Our ghost-bug scanner must verify both halves of a contradiction before it says yes. A high-ICP onboarding scanner reconstructs one signup journey that a product team may inspect immediately. In both cases, a plausible wrong answer wastes someone's time. Calibration will tell you whether the more expensive model earns its keep.

Use **Gemini 3.7 Flash** when a person may act on a single observation. Our ghost-bug scanner must verify both halves of a contradiction before it says yes. A high-ICP onboarding scanner reconstructs one signup journey that a product team may inspect immediately. In both cases, a plausible wrong answer wastes someone's time. Calibration will tell you whether the more expensive model earns its keep.

[SaraMiteva]

Suggested change

	title: "A scanner that watches everything sees nothing"
	title: "Watching PostHog watch itself: A scanner that watches everything sees nothing"

[SaraMiteva]

Suggested change

	In the last few weeks, our own [Replay Vision](/replay-vision) scanners have produced 251,351 succeeded observations across the millions of sessions we have of people using PostHog.
	A month ago we launched [Replay Vision](https://github.com/replay-vision), our AI layer over Session Replay. It watches session recordings in batches and writes up what it found, so nobody has to sit through them. You give it work by setting up a scanner: a recording query that picks which sessions to watch, and a prompt that says what to look for.
	In the last few weeks, our own [scanners](/docs/replay-vision/scanner-types) have produced 251,351 successful observations across the millions of sessions we have of people using PostHog.