feat(analytics): anonymous storefront telemetry for the shopper funnel

docs/storefront-analytics.md

@@ -0,0 +1,135 @@
+# Storefront Analytics
+
+Anonymous shopper telemetry, so the funnel can start before an order exists.
+
+`GET /v1/admin/analytics/funnel` begins at order creation. It answers "how many
+orders were paid" but not "how many people looked and left" — and the second
+question is usually the more useful one. Browsing, product views and abandoned
+carts leave no trace in the order tables, because nothing happened there.
+
+## How it fits together
+
+```
+storefront  ──POST /v1/analytics/events──▶  storefront_events
+   (public, rate limited, opt-in)                  │
+                                                   ▼
+                              GET /v1/admin/analytics/storefront
+                                                   │
+                                     joins the paid step from `orders`
+```
+
+## Privacy
+
+The table holds no column that could identify a person: no customer id, no
+email, no IP address, no user agent. A shopper is represented only by
+`session_id`, an opaque value their own browser generates and discards when the
+tab closes.
+
+This is not a policy someone has to remember. It is enforced in three places:
+
+**The schema.** There is nowhere to put an IP address.
+`test_the_table_has_no_column_that_could_identify_a_person` fails if a column
+named for an address, an identity or a device ever appears.
+
+**The request schema.** `extra="forbid"`, so a client sending `email` or
+`ip_address` gets a `422` rather than having the field quietly dropped. Silently
+discarding it would let a frontend believe it was collecting something it was
+not — which is worse than refusing, because nobody finds out.
+
+**The path validator.** Query strings are stripped before storage. A query
+string is where personal data arrives by accident — an email in a share link, a
+token in a redirect — and removing it at the boundary means it cannot be stored
+even if a client sends it.
+
+Because nothing here identifies anyone and nothing is stored in the browser
+beyond a per-tab session id, this needs no consent banner in the EU. That is not
+a happy accident; it is why the design is shaped this way. A banner costs
+40-60% of sessions to opt-outs, which would make the funnel it feeds mostly
+fiction.
+
+## Off unless the operator turns it on
+
+`STOREFRONT_ANALYTICS_ENABLED` defaults to `false` and the ingest endpoint
+returns `404` while it is. Cloning OpenTaberna must not silently start
+collecting anything, even something that identifies nobody.
+
+`404` rather than `403` so a deployment that has not opted in does not advertise
+a capability it is not offering.
+
+The admin endpoint still answers when collection is off — it reports
+`enabled: false` with zeroes, which distinguishes "nobody visited" from "we are
+not counting". Those look identical otherwise, and an operator staring at an
+empty funnel deserves to know which they are looking at.
+
+## The ingest endpoint is public
+
+Anyone who can load the shop can post to it. That shapes everything:
+
+| Guard | Why |
+|---|---|
+| Rate limited to 120/minute per address | An open write endpoint otherwise fills a table |
+| Batch capped at 50 events | One request cannot be a bulk insert |
+| Closed event vocabulary | A client cannot write arbitrary strings into a table an admin reads |
+| `extra="forbid"` | Unexpected fields are refused, not ignored |
+| Every field length-bounded | No unbounded text reaches storage |
+| Returns `202` | The browser must not wait on it, and must not retry into a queue during an incident |
+
+The worst an abusive client can achieve is noise in a report. It cannot store
+anything about anyone, and it cannot affect an order.
+
+## Timestamps are not trusted
+
+Events carry the browser's `occurred_at`, and browser clocks are wrong often
+enough that discarding all skew would lose real data. Events more than 24 hours
+either side of server time are dropped and counted in `rejected`.
+
+The window exists so a client cannot write into a period an administrator has
+already reported on. `created_at` records when the API stored the event and is
+trustworthy; `occurred_at` is what the browser claimed.
+
+## The last step is not taken on trust
+
+The funnel's `paid` step is **not** read from the events table. A browser
+reporting `checkout_started` means a button was pressed; whether money arrived
+is knowable only from `orders`.
+
+So `checkout_started` carries an `order_id`, and the API counts how many of
+those orders actually reached a revenue-producing status. A client claiming a
+checkout it never paid for inflates one step and cannot touch the next.
+
+`order_id` is deliberately **not** a foreign key. The event records what a
+browser reported, and must survive the order being deleted rather than vanishing
+with it and silently improving the conversion rate.
+
+## What the numbers mean
+
+Sessions are counted distinctly. Ten product views from one shopper is one
+person considering a purchase, not ten.
+
+| Step | Source |
+|---|---|
+| Visited the shop | Any event in the window |
+| Viewed a product | A `product_view` |
+| Added to cart | An `add_to_cart` |
+| Started checkout | A `checkout_started` |
+| Paid | `orders`, joined on the reported `order_id` |
+
+**The pre-order steps are a floor, not a count.** Blocked scripts, a tab closed
+before the batch flushed and disabled JavaScript all lose events. The paid step
+is exact. They are labelled differently in the UI for that reason rather than
+presented as one continuous measurement — a funnel whose first step undercounts
+and whose last step does not will overstate conversion, and a reader should know
+which end is soft.
+
+`add_to_cart_rate` per SKU is the figure worth watching: a product viewed often
+and added rarely means the listing draws people in and something — price, stock,
+photography — turns them away. Sales figures cannot show this, because they only
+ever contain what did sell.
+
+## Testing
+
+- `tests/test_storefront_analytics_unit.py` — the schema as a boundary: query
+  string stripping, closed vocabulary, forbidden extras, batch caps.
+- `tests/test_storefront_analytics_integration.py` — ingest without auth, the
+  PII-column assertion, clock-skew rejection, session-distinct counting, and
+  that a fabricated `order_id` cannot inflate the paid step.

src/app/db_models.py

@@ -41,3 +41,8 @@
 
 # Returns (RMA)
 from app.services.returns.models.returns_db_models import ReturnDB  # noqa: F401
+
+# Storefront analytics (S2)
+from app.services.storefront_analytics.models.storefront_events_db_models import (  # noqa: F401
+    StorefrontEventDB,
+)

src/app/main.py

@@ -16,6 +16,7 @@
 from app.services.inventory import inventory_api_router
 from app.services.orders import orders_api_router, webhooks_api_router
 from app.services.returns import admin_returns_api_router, returns_api_router
+from app.services.storefront_analytics import storefront_analytics_api_router
 from app.shared.exceptions import AppException, InternalError
 from app.shared.logger import get_logger
 from app.shared.middleware import CorrelationIDMiddleware
@@ -156,6 +157,9 @@ async def generic_exception_handler(request: Request, exc: Exception) -> JSONRes
 # Include analytics service router (S1)
 app.include_router(analytics_api_router, prefix="/v1")
 
+# Include storefront analytics ingest (S2) — public, rate limited, opt-in
+app.include_router(storefront_analytics_api_router, prefix="/v1")
+
 # Include fulfillment service router (Phase 3)
 app.include_router(fulfillment_api_router, prefix="/v1")
 

src/app/services/analytics/models/__init__.py

@@ -2,6 +2,7 @@
 
 from .analytics_models import (
     AnalyticsFunnelResponse,
+    AnalyticsStorefrontResponse,
     AnalyticsProductsResponse,
     AnalyticsSummaryResponse,
     AnalyticsTimeseriesResponse,
@@ -11,13 +12,17 @@
     CurrencyTotalsPrevious,
     FunnelStep,
     NeverSoldItem,
+    PathViews,
     PeriodInfo,
+    ProductInterest,
     ProductPerformance,
     SeriesPoint,
+    StorefrontStep,
 )
 
 __all__ = [
     "AnalyticsFunnelResponse",
+    "AnalyticsStorefrontResponse",
     "AnalyticsProductsResponse",
     "AnalyticsSummaryResponse",
     "AnalyticsTimeseriesResponse",
@@ -27,7 +32,10 @@
     "CurrencyTotalsPrevious",
     "FunnelStep",
     "NeverSoldItem",
+    "PathViews",
     "PeriodInfo",
+    "ProductInterest",
     "ProductPerformance",
     "SeriesPoint",
+    "StorefrontStep",
 ]

src/app/services/analytics/models/analytics_models.py

@@ -241,4 +241,71 @@ class AnalyticsFunnelResponse(BaseResponse):
     cancelled: int = Field(description="Orders cancelled in the window")
 
 
+# ============================================================================
+# Storefront funnel (S2)
+# ============================================================================
+
+
+class StorefrontStep(BaseModel):
+    """One stage of the shopper journey, counted in sessions."""
+
+    step: str
+    label: str
+    sessions: int
+    conversion_from_start: float | None = Field(
+        default=None, description="Share of all sessions that reached this step"
+    )
+    drop_off_from_previous: int | None = Field(
+        default=None, description="Sessions lost between the previous step and this"
+    )
+
+
+class PathViews(BaseModel):
+    """Traffic to one route."""
+
+    path: str
+    views: int
+    sessions: int
+
+
+class ProductInterest(BaseModel):
+    """
+    How a product fares before the checkout.
+
+    A SKU viewed often and added rarely is the useful case: the listing draws
+    people in and something then turns them away. Sales figures cannot show
+    this, because they only ever contain what did sell.
+    """
+
+    sku: str
+    name: str | None = None
+    sessions_viewed: int
+    sessions_added: int
+    add_to_cart_rate: float | None = None
+
+
+class AnalyticsStorefrontResponse(BaseResponse):
+    """
+    The shopper funnel, from arriving at the shop through to a paid order.
+
+    Sessions are counted distinctly, so ten product views by one shopper are one
+    person considering a purchase rather than ten.
+
+    **This depends on what browsers reported.** Blocked scripts, a closed tab
+    before the batch flushed and disabled JavaScript all lose events, so the
+    pre-order steps are a floor rather than an exact count. The paid step is
+    read from the orders table and is exact — which is why the two are labelled
+    differently rather than presented as one continuous measurement.
+    """
+
+    period: PeriodInfo
+    enabled: bool = Field(
+        description="Whether the deployment is collecting storefront events at all"
+    )
+    page_views: int
+    steps: list[StorefrontStep] = Field(default_factory=list)
+    top_paths: list[PathViews] = Field(default_factory=list)
+    product_interest: list[ProductInterest] = Field(default_factory=list)
+
+
 CurrencyTotals.model_rebuild()

src/app/services/analytics/routers/analytics_router.py

@@ -27,6 +27,7 @@
 from ..functions import Interval, Period, build_period, percent_change
 from ..models import (
     AnalyticsFunnelResponse,
+    AnalyticsStorefrontResponse,
     AnalyticsProductsResponse,
     AnalyticsSummaryResponse,
     AnalyticsTimeseriesResponse,
@@ -36,16 +37,21 @@
     CurrencyTotalsPrevious,
     FunnelStep,
     NeverSoldItem,
+    PathViews,
     PeriodInfo,
+    ProductInterest,
     ProductPerformance,
     SeriesPoint,
+    StorefrontStep,
 )
 from ..responses import (
     FUNNEL_RESPONSES,
     PRODUCTS_RESPONSES,
     SUMMARY_RESPONSES,
     TIMESERIES_RESPONSES,
 )
+from app.services.storefront_analytics.services import StorefrontEventRepository
+
 from ..services import AnalyticsRepository, fill_series_gaps
 
 logger = get_logger(__name__)
@@ -368,3 +374,91 @@ async def get_funnel(
         payment_unresolved=counts["payment_unresolved"],
         cancelled=counts["cancelled"],
     )
+
+
+# ---------------------------------------------------------------------------
+# GET /admin/analytics/storefront
+# ---------------------------------------------------------------------------
+
+
+@router.get(
+    "/storefront",
+    response_model=AnalyticsStorefrontResponse,
+    summary="Shopper funnel (admin)",
+    description=(
+        "The journey before an order exists: sessions, product views, carts, "
+        "checkouts and paid orders.\n\n"
+        "Sessions are counted distinctly, so ten product views by one shopper "
+        "count once.\n\n"
+        "The pre-order steps come from what browsers reported and are a **floor**, "
+        "not an exact count — blocked scripts and closed tabs lose events. The "
+        "paid step is read from the orders table and is exact. Returns "
+        "`enabled: false` with zeroes when the deployment is not collecting."
+    ),
+    responses=FUNNEL_RESPONSES,
+    dependencies=[Depends(require_admin)],
+)
+async def get_storefront_funnel(
+    date_from: date | None = Query(
+        None, alias="from", description="First day, inclusive"
+    ),
+    date_to: date | None = Query(None, alias="to", description="Last day, inclusive"),
+    session: AsyncSession = Depends(get_session_dependency),
+) -> AnalyticsStorefrontResponse:
+    period = await _resolve_period(date_from, date_to)
+    settings = get_settings()
+
+    events = StorefrontEventRepository(session)
+    counts = await events.browse_funnel(period.start, period.end)
+    page_views = await events.page_views(period.start, period.end)
+
+    # The final step is not taken from the browser's word for it. A reported
+    # checkout says the shopper pressed the button; only the orders table knows
+    # whether money arrived.
+    checkout_orders = await events.checkout_order_ids(period.start, period.end)
+    paid = await AnalyticsRepository(session).count_paid_orders(checkout_orders)
+
+    definitions = [
+        ("sessions", "Visited the shop", counts["sessions"]),
+        ("viewed_product", "Viewed a product", counts["viewed_product"]),
+        ("added_to_cart", "Added to cart", counts["added_to_cart"]),
+        ("started_checkout", "Started checkout", counts["started_checkout"]),
+        ("paid", "Paid", paid),
+    ]
+
+    total = counts["sessions"]
+    steps: list[StorefrontStep] = []
+    previous: int | None = None
+    for key, label, value in definitions:
+        steps.append(
+            StorefrontStep(
+                step=key,
+                label=label,
+                sessions=value,
+                conversion_from_start=round(value / total, 4) if total else None,
+                drop_off_from_previous=(
+                    None if previous is None else max(previous - value, 0)
+                ),
+            )
+        )
+        previous = value
+
+    interest = await events.product_interest(period.start, period.end)
+    names = await AnalyticsRepository(session).item_names(
+        [row["sku"] for row in interest]
+    )
+
+    return AnalyticsStorefrontResponse(
+        success=True,
+        message="Storefront funnel retrieved successfully",
+        period=_period_info(period),
+        enabled=settings.storefront_analytics_enabled,
+        page_views=page_views,
+        steps=steps,
+        top_paths=[
+            PathViews(**row) for row in await events.top_paths(period.start, period.end)
+        ],
+        product_interest=[
+            ProductInterest(**row, name=names.get(row["sku"])) for row in interest
+        ],
+    )