Skip to content

Show what the browser tag never saw #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lbesecker195 merged 1 commit into from
Sep 19, 2026
Merged

Show what the browser tag never saw #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions lib/web_analytics/analytics.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2166,6 +2166,124 @@ defmodule WebAnalytics.Analytics do
)
end

# -- tag coverage --------------------------------------------------------
#
# What the browser tag missed, which is only answerable now that a server-side
# plug reports the same pageviews the tag does. Where both reported one, they
# merge into a single row; where only the server did, the row is missing
# everything a browser has to supply.
#
# Two markers, either of which settles it. The tag sends `window.innerHeight`
# on every pageview it opens, and it heartbeats afterwards; the plug sends
# neither, because a server has no viewport and does not stay on a page. So a
# pageview with no viewport height *and* no tick is one no tag ever reported.
#
# Either signal on its own would be wrong in a case that really happens: a
# visitor who leaves inside a second is gone before the first heartbeat, and a
# browser reporting no viewport height is unusual but not impossible.

# Crawlers are the subject here, not noise in front of it, so the usual
# exclusion is deliberately not applied. A report about what the tag missed
# that hid the largest thing it misses would be worse than no report.
defp coverage_scope(f) do
from(p in Pageview,
join: s in assoc(p, :session),
as: :session,
where: p.site_id == ^f.site_id,
where: p.entered_at >= ^f.from and p.entered_at < ^f.to
)
|> filter_joined_anomalies(f)
|> filter_joined_dwell(f)
|> filter_joined_origins(f)
|> filter_joined_sessions(f)
|> filter_joined_project(f)
|> filter_joined_host(f)
|> filter_joined_user(f)
end

@doc """
How much of this site's traffic the browser tag actually saw.

Splits what it missed into automated and everything else, because the two
mean different things. Crawlers missing the tag is expected and is the reason
server-side recording exists. People missing it is a finding: blocked
scripts, a failed asset, a page the tag was never added to.
"""
def tag_coverage(f) do
totals =
Repo.one(
from [p, session: s] in coverage_scope(f),
select: %{
pageviews: count(p.id),
tagged: filter(count(p.id), not is_nil(p.viewport_h) or p.tick_count > 0),
untagged: filter(count(p.id), is_nil(p.viewport_h) and p.tick_count == 0),
untagged_crawler:
filter(count(p.id), is_nil(p.viewport_h) and p.tick_count == 0 and s.crawler),
untagged_human:
filter(count(p.id), is_nil(p.viewport_h) and p.tick_count == 0 and not s.crawler),
human_pageviews: filter(count(p.id), not s.crawler)
}
) || %{}

totals
|> Map.put(:coverage, rate(Map.get(totals, :tagged, 0), Map.get(totals, :pageviews, 0)))
|> Map.put(
:human_coverage,
rate(
Map.get(totals, :human_pageviews, 0) - Map.get(totals, :untagged_human, 0),
Map.get(totals, :human_pageviews, 0)
)
)
end

@doc """
The pages the tag never reported, most-missed first.

Always grouped by path, never by title, because a title is one of the things
only the tag can supply — grouping these by title would return one unnamed
row holding everything.
"""
def untagged_pages(f, limit \\ 25) do
Repo.all(
from [p, session: s] in coverage_scope(f),
where: is_nil(p.viewport_h) and p.tick_count == 0,
group_by: p.path,
order_by: [desc: count(p.id)],
limit: ^limit,
select: %{
name: p.path,
count: count(p.id),
crawler: filter(count(p.id), s.crawler),
human: filter(count(p.id), not s.crawler),
sessions: count(p.session_id, :distinct),
last_seen: max(p.entered_at)
}
)
end

@doc """
What was reading the pages the tag never saw, by user agent.

Answers the question the coverage number raises: if a tenth of this site is
invisible to the tag, who is that?
"""
def untagged_clients(f, limit \\ 15) do
Repo.all(
from [p, session: s] in coverage_scope(f),
where: is_nil(p.viewport_h) and p.tick_count == 0,
group_by: [s.crawler_name, s.crawler_kind, s.crawler],
order_by: [desc: count(p.id)],
limit: ^limit,
select: %{
name: s.crawler_name,
kind: s.crawler_kind,
crawler: s.crawler,
count: count(p.id),
sessions: count(p.session_id, :distinct)
}
)
end

# -- breakdowns ----------------------------------------------------------

@doc "Top values of a session dimension, e.g. `:browser` or `:referrer_host`."
Expand Down
10 changes: 9 additions & 1 deletion lib/web_analytics_web/live/dashboard_live.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ defmodule WebAnalyticsWeb.DashboardLive do
alias WebAnalytics.Ingest.Crawler
alias WebAnalytics.Sites

@tabs ~w(live overview users pages events metrics flow locations clicks forms sessions anomalies crawlers)
@tabs ~w(live overview users pages events metrics flow locations clicks forms sessions anomalies crawlers coverage)
@click_groups ~w(name id class text selector tag)
@location_levels ~w(country region county city)
@flow_modes ~w(pages events)
Expand Down Expand Up @@ -622,6 +622,14 @@ defmodule WebAnalyticsWeb.DashboardLive do
}
end

defp tab_data("coverage", filters, _assigns) do
%{
tag_coverage: Analytics.tag_coverage(filters),
untagged_pages: Analytics.untagged_pages(filters),
untagged_clients: Analytics.untagged_clients(filters)
}
end

# Loaded by assign_live/1 on its own interval rather than here, so the five
# second refresh does not run it too.
defp tab_data("live", _filters, _assigns), do: %{}
Expand Down
115 changes: 115 additions & 0 deletions lib/web_analytics_web/live/dashboard_live.html.heex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2130,6 +2130,121 @@
</div>
</div>

<!-- ============================= COVERAGE ============================ -->
<div :if={@tab == "coverage"} class="space-y-6">
<% cov = @data[:tag_coverage] || %{} %>

<div class="text-sm text-base-content/60">
What the browser tag missed. The tag has to load and run before it can report, so
anything that never runs JavaScript is invisible to it — crawlers and AI agents by
nature, and real visitors when a script is blocked or a page was never tagged. These
pages are here because the server-side plug recorded them anyway.
</div>

<div class="grid grid-cols-2 md:grid-cols-4 gap-3">
<.stat label="Pageviews recorded" value={number(cov[:pageviews])} />
<.stat
label="Seen by the tag"
value={percent(cov[:coverage])}
hint={"#{number(cov[:tagged])} of #{number(cov[:pageviews])}"}
tone={if (cov[:coverage] || 0) >= 90, do: "success", else: "neutral"}
/>
<.stat
label="Missed — automated"
value={number(cov[:untagged_crawler])}
hint="Expected. This is why the plug exists."
/>
<.stat
label="Missed — not automated"
value={number(cov[:untagged_human])}
hint={"#{percent(100 - (cov[:human_coverage] || 100))} of non-bot pageviews"}
tone={if (cov[:untagged_human] || 0) > 0, do: "warning", else: "success"}
/>
</div>

<div
:if={(cov[:untagged_human] || 0) > 0}
class="rounded-box border border-warning/40 bg-warning/10 px-4 py-3 text-sm"
>
<span class="font-medium">{number(cov[:untagged_human])} pageviews</span>
came from something that was not automated and still never ran the tag. That is usually
a blocked script, a failed asset, or a page the snippet was never added to — worth
checking, because those visits would be missing entirely without the plug.
</div>

<div class="grid md:grid-cols-2 gap-4">
<.bar_list
title="Pages the tag never saw"
rows={@data[:untagged_pages] || []}
empty="The tag saw everything"
/>

<.bar_list
title="What was reading them"
rows={
Enum.map(@data[:untagged_clients] || [], fn row ->
%{name: row.name || "Not automated", count: row.count}
end)
}
empty="Nothing went unreported"
/>
</div>

<div class="rounded-box bg-base-100 border border-base-300 overflow-x-auto">
<div class="px-4 py-3 border-b border-base-300 font-medium text-sm">
Untagged pages in full
</div>
<table class="table table-sm">
<thead>
<tr>
<th>Path</th>
<th class="text-right">Pageviews</th>
<th class="text-right">Automated</th>
<th class="text-right">Not automated</th>
<th class="text-right">Sessions</th>
<th class="text-right">Last seen</th>
</tr>
</thead>
<tbody>
<tr :for={page <- @data[:untagged_pages] || []}>
<td class="font-mono text-xs">{page.name}</td>
<td class="text-right tabular-nums">{number(page.count)}</td>
<td class="text-right tabular-nums text-base-content/60">{number(page.crawler)}</td>
<td class={[
"text-right tabular-nums",
page.human > 0 && "text-warning font-medium"
]}>
{number(page.human)}
</td>
<td class="text-right tabular-nums">{number(page.sessions)}</td>
<td class="text-right text-base-content/60 whitespace-nowrap">
{page.last_seen && Calendar.strftime(page.last_seen, "%d %b %H:%M")}
</td>
</tr>
<tr :if={(@data[:untagged_pages] || []) == []}>
<td colspan="6" class="text-center py-8 text-base-content/50">
Nothing went unreported
</td>
</tr>
</tbody>
</table>
</div>

<div class="rounded-box bg-base-100 border border-base-300 px-4 py-3 text-sm">
<span class="font-medium">Not seeing anything here?</span>
This tab only fills up once server-side recording is installed —
<.link
href="https://github.com/lbesecker195/Phoenix-Analytics"
class="link link-primary"
target="_blank"
rel="noopener"
>
the Phoenix plug
</.link>
reports the pages your tag cannot, on the same account and the same visits.
</div>
</div>

<%!-- The instructions live on /getting-started now. An account with traffic
scrolled past them on every visit, and an account with none had to scroll
past every empty chart to reach the only thing it needed. --%>
Expand Down
Loading
Loading