diff --git a/docs/storefront-analytics.md b/docs/storefront-analytics.md new file mode 100644 index 0000000..4d0477d --- /dev/null +++ b/docs/storefront-analytics.md @@ -0,0 +1,135 @@ +# Storefront Analytics + +Anonymous shopper telemetry, so the funnel can start before an order exists. + +`GET /v1/admin/analytics/funnel` begins at order creation. It answers "how many +orders were paid" but not "how many people looked and left" — and the second +question is usually the more useful one. Browsing, product views and abandoned +carts leave no trace in the order tables, because nothing happened there. + +## How it fits together + +``` +storefront ──POST /v1/analytics/events──▶ storefront_events + (public, rate limited, opt-in) │ + ▼ + GET /v1/admin/analytics/storefront + │ + joins the paid step from `orders` +``` + +## Privacy + +The table holds no column that could identify a person: no customer id, no +email, no IP address, no user agent. A shopper is represented only by +`session_id`, an opaque value their own browser generates and discards when the +tab closes. + +This is not a policy someone has to remember. It is enforced in three places: + +**The schema.** There is nowhere to put an IP address. +`test_the_table_has_no_column_that_could_identify_a_person` fails if a column +named for an address, an identity or a device ever appears. + +**The request schema.** `extra="forbid"`, so a client sending `email` or +`ip_address` gets a `422` rather than having the field quietly dropped. Silently +discarding it would let a frontend believe it was collecting something it was +not — which is worse than refusing, because nobody finds out. + +**The path validator.** Query strings are stripped before storage. A query +string is where personal data arrives by accident — an email in a share link, a +token in a redirect — and removing it at the boundary means it cannot be stored +even if a client sends it. + +Because nothing here identifies anyone and nothing is stored in the browser +beyond a per-tab session id, this needs no consent banner in the EU. That is not +a happy accident; it is why the design is shaped this way. A banner costs +40-60% of sessions to opt-outs, which would make the funnel it feeds mostly +fiction. + +## Off unless the operator turns it on + +`STOREFRONT_ANALYTICS_ENABLED` defaults to `false` and the ingest endpoint +returns `404` while it is. Cloning OpenTaberna must not silently start +collecting anything, even something that identifies nobody. + +`404` rather than `403` so a deployment that has not opted in does not advertise +a capability it is not offering. + +The admin endpoint still answers when collection is off — it reports +`enabled: false` with zeroes, which distinguishes "nobody visited" from "we are +not counting". Those look identical otherwise, and an operator staring at an +empty funnel deserves to know which they are looking at. + +## The ingest endpoint is public + +Anyone who can load the shop can post to it. That shapes everything: + +| Guard | Why | +|---|---| +| Rate limited to 120/minute per address | An open write endpoint otherwise fills a table | +| Batch capped at 50 events | One request cannot be a bulk insert | +| Closed event vocabulary | A client cannot write arbitrary strings into a table an admin reads | +| `extra="forbid"` | Unexpected fields are refused, not ignored | +| Every field length-bounded | No unbounded text reaches storage | +| Returns `202` | The browser must not wait on it, and must not retry into a queue during an incident | + +The worst an abusive client can achieve is noise in a report. It cannot store +anything about anyone, and it cannot affect an order. + +## Timestamps are not trusted + +Events carry the browser's `occurred_at`, and browser clocks are wrong often +enough that discarding all skew would lose real data. Events more than 24 hours +either side of server time are dropped and counted in `rejected`. + +The window exists so a client cannot write into a period an administrator has +already reported on. `created_at` records when the API stored the event and is +trustworthy; `occurred_at` is what the browser claimed. + +## The last step is not taken on trust + +The funnel's `paid` step is **not** read from the events table. A browser +reporting `checkout_started` means a button was pressed; whether money arrived +is knowable only from `orders`. + +So `checkout_started` carries an `order_id`, and the API counts how many of +those orders actually reached a revenue-producing status. A client claiming a +checkout it never paid for inflates one step and cannot touch the next. + +`order_id` is deliberately **not** a foreign key. The event records what a +browser reported, and must survive the order being deleted rather than vanishing +with it and silently improving the conversion rate. + +## What the numbers mean + +Sessions are counted distinctly. Ten product views from one shopper is one +person considering a purchase, not ten. + +| Step | Source | +|---|---| +| Visited the shop | Any event in the window | +| Viewed a product | A `product_view` | +| Added to cart | An `add_to_cart` | +| Started checkout | A `checkout_started` | +| Paid | `orders`, joined on the reported `order_id` | + +**The pre-order steps are a floor, not a count.** Blocked scripts, a tab closed +before the batch flushed and disabled JavaScript all lose events. The paid step +is exact. They are labelled differently in the UI for that reason rather than +presented as one continuous measurement — a funnel whose first step undercounts +and whose last step does not will overstate conversion, and a reader should know +which end is soft. + +`add_to_cart_rate` per SKU is the figure worth watching: a product viewed often +and added rarely means the listing draws people in and something — price, stock, +photography — turns them away. Sales figures cannot show this, because they only +ever contain what did sell. + +## Testing + +- `tests/test_storefront_analytics_unit.py` — the schema as a boundary: query + string stripping, closed vocabulary, forbidden extras, batch caps. +- `tests/test_storefront_analytics_integration.py` — ingest without auth, the + PII-column assertion, clock-skew rejection, session-distinct counting, and + that a fabricated `order_id` cannot inflate the paid step. diff --git a/src/app/db_models.py b/src/app/db_models.py index fd708b8..041253a 100644 --- a/src/app/db_models.py +++ b/src/app/db_models.py @@ -41,3 +41,8 @@ # Returns (RMA) from app.services.returns.models.returns_db_models import ReturnDB # noqa: F401 + +# Storefront analytics (S2) +from app.services.storefront_analytics.models.storefront_events_db_models import ( # noqa: F401 + StorefrontEventDB, +) diff --git a/src/app/main.py b/src/app/main.py index 52eef20..1647e1f 100644 --- a/src/app/main.py +++ b/src/app/main.py @@ -16,6 +16,7 @@ from app.services.inventory import inventory_api_router from app.services.orders import orders_api_router, webhooks_api_router from app.services.returns import admin_returns_api_router, returns_api_router +from app.services.storefront_analytics import storefront_analytics_api_router from app.shared.exceptions import AppException, InternalError from app.shared.logger import get_logger from app.shared.middleware import CorrelationIDMiddleware @@ -156,6 +157,9 @@ async def generic_exception_handler(request: Request, exc: Exception) -> JSONRes # Include analytics service router (S1) app.include_router(analytics_api_router, prefix="/v1") +# Include storefront analytics ingest (S2) — public, rate limited, opt-in +app.include_router(storefront_analytics_api_router, prefix="/v1") + # Include fulfillment service router (Phase 3) app.include_router(fulfillment_api_router, prefix="/v1") diff --git a/src/app/services/analytics/models/__init__.py b/src/app/services/analytics/models/__init__.py index e65cdb1..628b2b4 100644 --- a/src/app/services/analytics/models/__init__.py +++ b/src/app/services/analytics/models/__init__.py @@ -2,6 +2,7 @@ from .analytics_models import ( AnalyticsFunnelResponse, + AnalyticsStorefrontResponse, AnalyticsProductsResponse, AnalyticsSummaryResponse, AnalyticsTimeseriesResponse, @@ -11,13 +12,17 @@ CurrencyTotalsPrevious, FunnelStep, NeverSoldItem, + PathViews, PeriodInfo, + ProductInterest, ProductPerformance, SeriesPoint, + StorefrontStep, ) __all__ = [ "AnalyticsFunnelResponse", + "AnalyticsStorefrontResponse", "AnalyticsProductsResponse", "AnalyticsSummaryResponse", "AnalyticsTimeseriesResponse", @@ -27,7 +32,10 @@ "CurrencyTotalsPrevious", "FunnelStep", "NeverSoldItem", + "PathViews", "PeriodInfo", + "ProductInterest", "ProductPerformance", "SeriesPoint", + "StorefrontStep", ] diff --git a/src/app/services/analytics/models/analytics_models.py b/src/app/services/analytics/models/analytics_models.py index 9f3cad8..c7a51ab 100644 --- a/src/app/services/analytics/models/analytics_models.py +++ b/src/app/services/analytics/models/analytics_models.py @@ -241,4 +241,71 @@ class AnalyticsFunnelResponse(BaseResponse): cancelled: int = Field(description="Orders cancelled in the window") +# ============================================================================ +# Storefront funnel (S2) +# ============================================================================ + + +class StorefrontStep(BaseModel): + """One stage of the shopper journey, counted in sessions.""" + + step: str + label: str + sessions: int + conversion_from_start: float | None = Field( + default=None, description="Share of all sessions that reached this step" + ) + drop_off_from_previous: int | None = Field( + default=None, description="Sessions lost between the previous step and this" + ) + + +class PathViews(BaseModel): + """Traffic to one route.""" + + path: str + views: int + sessions: int + + +class ProductInterest(BaseModel): + """ + How a product fares before the checkout. + + A SKU viewed often and added rarely is the useful case: the listing draws + people in and something then turns them away. Sales figures cannot show + this, because they only ever contain what did sell. + """ + + sku: str + name: str | None = None + sessions_viewed: int + sessions_added: int + add_to_cart_rate: float | None = None + + +class AnalyticsStorefrontResponse(BaseResponse): + """ + The shopper funnel, from arriving at the shop through to a paid order. + + Sessions are counted distinctly, so ten product views by one shopper are one + person considering a purchase rather than ten. + + **This depends on what browsers reported.** Blocked scripts, a closed tab + before the batch flushed and disabled JavaScript all lose events, so the + pre-order steps are a floor rather than an exact count. The paid step is + read from the orders table and is exact — which is why the two are labelled + differently rather than presented as one continuous measurement. + """ + + period: PeriodInfo + enabled: bool = Field( + description="Whether the deployment is collecting storefront events at all" + ) + page_views: int + steps: list[StorefrontStep] = Field(default_factory=list) + top_paths: list[PathViews] = Field(default_factory=list) + product_interest: list[ProductInterest] = Field(default_factory=list) + + CurrencyTotals.model_rebuild() diff --git a/src/app/services/analytics/routers/analytics_router.py b/src/app/services/analytics/routers/analytics_router.py index 42051c2..f1c92aa 100644 --- a/src/app/services/analytics/routers/analytics_router.py +++ b/src/app/services/analytics/routers/analytics_router.py @@ -27,6 +27,7 @@ from ..functions import Interval, Period, build_period, percent_change from ..models import ( AnalyticsFunnelResponse, + AnalyticsStorefrontResponse, AnalyticsProductsResponse, AnalyticsSummaryResponse, AnalyticsTimeseriesResponse, @@ -36,9 +37,12 @@ CurrencyTotalsPrevious, FunnelStep, NeverSoldItem, + PathViews, PeriodInfo, + ProductInterest, ProductPerformance, SeriesPoint, + StorefrontStep, ) from ..responses import ( FUNNEL_RESPONSES, @@ -46,6 +50,8 @@ SUMMARY_RESPONSES, TIMESERIES_RESPONSES, ) +from app.services.storefront_analytics.services import StorefrontEventRepository + from ..services import AnalyticsRepository, fill_series_gaps logger = get_logger(__name__) @@ -368,3 +374,91 @@ async def get_funnel( payment_unresolved=counts["payment_unresolved"], cancelled=counts["cancelled"], ) + + +# --------------------------------------------------------------------------- +# GET /admin/analytics/storefront +# --------------------------------------------------------------------------- + + +@router.get( + "/storefront", + response_model=AnalyticsStorefrontResponse, + summary="Shopper funnel (admin)", + description=( + "The journey before an order exists: sessions, product views, carts, " + "checkouts and paid orders.\n\n" + "Sessions are counted distinctly, so ten product views by one shopper " + "count once.\n\n" + "The pre-order steps come from what browsers reported and are a **floor**, " + "not an exact count — blocked scripts and closed tabs lose events. The " + "paid step is read from the orders table and is exact. Returns " + "`enabled: false` with zeroes when the deployment is not collecting." + ), + responses=FUNNEL_RESPONSES, + dependencies=[Depends(require_admin)], +) +async def get_storefront_funnel( + date_from: date | None = Query( + None, alias="from", description="First day, inclusive" + ), + date_to: date | None = Query(None, alias="to", description="Last day, inclusive"), + session: AsyncSession = Depends(get_session_dependency), +) -> AnalyticsStorefrontResponse: + period = await _resolve_period(date_from, date_to) + settings = get_settings() + + events = StorefrontEventRepository(session) + counts = await events.browse_funnel(period.start, period.end) + page_views = await events.page_views(period.start, period.end) + + # The final step is not taken from the browser's word for it. A reported + # checkout says the shopper pressed the button; only the orders table knows + # whether money arrived. + checkout_orders = await events.checkout_order_ids(period.start, period.end) + paid = await AnalyticsRepository(session).count_paid_orders(checkout_orders) + + definitions = [ + ("sessions", "Visited the shop", counts["sessions"]), + ("viewed_product", "Viewed a product", counts["viewed_product"]), + ("added_to_cart", "Added to cart", counts["added_to_cart"]), + ("started_checkout", "Started checkout", counts["started_checkout"]), + ("paid", "Paid", paid), + ] + + total = counts["sessions"] + steps: list[StorefrontStep] = [] + previous: int | None = None + for key, label, value in definitions: + steps.append( + StorefrontStep( + step=key, + label=label, + sessions=value, + conversion_from_start=round(value / total, 4) if total else None, + drop_off_from_previous=( + None if previous is None else max(previous - value, 0) + ), + ) + ) + previous = value + + interest = await events.product_interest(period.start, period.end) + names = await AnalyticsRepository(session).item_names( + [row["sku"] for row in interest] + ) + + return AnalyticsStorefrontResponse( + success=True, + message="Storefront funnel retrieved successfully", + period=_period_info(period), + enabled=settings.storefront_analytics_enabled, + page_views=page_views, + steps=steps, + top_paths=[ + PathViews(**row) for row in await events.top_paths(period.start, period.end) + ], + product_interest=[ + ProductInterest(**row, name=names.get(row["sku"])) for row in interest + ], + ) diff --git a/src/app/services/analytics/services/analytics_db_service.py b/src/app/services/analytics/services/analytics_db_service.py index 8160332..d5f1225 100644 --- a/src/app/services/analytics/services/analytics_db_service.py +++ b/src/app/services/analytics/services/analytics_db_service.py @@ -404,6 +404,25 @@ async def funnel_counts(self, period: Period) -> dict[str, int]: "cancelled": cancelled, } + async def count_paid_orders(self, order_ids: list) -> int: + """ + How many of these orders were actually paid. + + Used to close the storefront funnel. A browser reporting "checkout + started" only means a button was pressed; whether money arrived is + knowable solely from the orders table, so the last step of a funnel + built from browser events is deliberately not taken from browser events. + """ + if not order_ids: + return 0 + return await self._scalar( + select(func.count(distinct(OrderDB.id))).where( + OrderDB.id.in_(order_ids), + OrderDB.deleted_at.is_(None), + OrderDB.status.in_(EARNED_STATUSES), + ) + ) + async def _scalar(self, statement: Select) -> int: result = await self._session.execute(statement) return int(result.scalar_one_or_none() or 0) diff --git a/src/app/services/storefront_analytics/__init__.py b/src/app/services/storefront_analytics/__init__.py new file mode 100644 index 0000000..cbe00c9 --- /dev/null +++ b/src/app/services/storefront_analytics/__init__.py @@ -0,0 +1,28 @@ +""" +Storefront Analytics Service + +Anonymous shopper telemetry from the storefront, so the funnel can begin before +an order exists. + +Endpoints: + POST /analytics/events — record anonymous storefront events (public) + +The admin-facing read side lives in `services/analytics`, alongside the order +figures it has to be read together with. + +Usage: + from app.services.storefront_analytics import storefront_analytics_api_router + app.include_router(storefront_analytics_api_router, prefix="/v1") +""" + +from fastapi import APIRouter + +from .routers import storefront_analytics_router + +storefront_analytics_api_router = APIRouter( + prefix="/analytics", + tags=["Storefront Analytics"], +) +storefront_analytics_api_router.include_router(storefront_analytics_router) + +__all__ = ["storefront_analytics_api_router"] diff --git a/src/app/services/storefront_analytics/models/__init__.py b/src/app/services/storefront_analytics/models/__init__.py new file mode 100644 index 0000000..dad9f3c --- /dev/null +++ b/src/app/services/storefront_analytics/models/__init__.py @@ -0,0 +1,19 @@ +"""Storefront analytics models.""" + +from .storefront_events_db_models import StorefrontEventDB +from .storefront_events_models import ( + MAX_EVENTS_PER_BATCH, + StorefrontEventBatch, + StorefrontEventInput, + StorefrontEventType, + StorefrontIngestResponse, +) + +__all__ = [ + "MAX_EVENTS_PER_BATCH", + "StorefrontEventBatch", + "StorefrontEventDB", + "StorefrontEventInput", + "StorefrontEventType", + "StorefrontIngestResponse", +] diff --git a/src/app/services/storefront_analytics/models/storefront_events_db_models.py b/src/app/services/storefront_analytics/models/storefront_events_db_models.py new file mode 100644 index 0000000..430dc5a --- /dev/null +++ b/src/app/services/storefront_analytics/models/storefront_events_db_models.py @@ -0,0 +1,98 @@ +""" +Storefront Analytics Database Models + +One table holding anonymous shopper events sent by the storefront. + +**What is deliberately not here matters more than what is.** There is no +customer id, no email, no IP address and no user agent. A shopper is identified +only by `session_id`, an opaque value the browser generates and discards when +the tab closes. Nothing in this table can be traced to a person, which is what +keeps the shop free of a consent banner and the operator out of a category of +obligation nobody wants. + +`order_id` is the single join back to identified data, and it points at an order +the shopper themselves created — it reveals nothing the orders table does not +already hold. +""" + +from datetime import datetime +from uuid import UUID, uuid4 + +from sqlalchemy import DateTime, Index, String, text +from sqlalchemy.dialects.postgresql import UUID as PGUUID +from sqlalchemy.orm import Mapped, mapped_column + +from app.shared.database.base import Base + + +class StorefrontEventDB(Base): + """A single anonymous interaction in the storefront.""" + + __tablename__ = "storefront_events" + + id: Mapped[UUID] = mapped_column( + PGUUID(as_uuid=True), + primary_key=True, + default=uuid4, + server_default=text("gen_random_uuid()"), + ) + + session_id: Mapped[str] = mapped_column( + String(64), + nullable=False, + doc="Opaque, browser-generated, per-session. Not a user identifier.", + ) + + event_type: Mapped[str] = mapped_column( + String(40), + nullable=False, + doc="page_view | product_view | add_to_cart | checkout_started", + ) + + path: Mapped[str | None] = mapped_column( + String(255), + nullable=True, + doc="Route path only. Query strings are stripped before storage.", + ) + + sku: Mapped[str | None] = mapped_column( + String(100), + nullable=True, + doc="Product the event concerns, for product_view and add_to_cart.", + ) + + order_id: Mapped[UUID | None] = mapped_column( + PGUUID(as_uuid=True), + nullable=True, + doc=( + "Set on checkout_started. Deliberately not a foreign key: the event " + "is a record of what a browser reported, and must survive the order " + "being deleted rather than disappearing with it and silently " + "improving the conversion rate." + ), + ) + + occurred_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + nullable=False, + doc="When the browser says it happened. Clamped by the API on ingest.", + ) + + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + nullable=False, + server_default=text("now()"), + doc="When the API stored it. Trustworthy, unlike occurred_at.", + ) + + __table_args__ = ( + Index("ix_storefront_events_occurred_at", "occurred_at"), + Index("ix_storefront_events_type_occurred", "event_type", "occurred_at"), + Index("ix_storefront_events_session", "session_id"), + ) + + def __repr__(self) -> str: + return ( + f"StorefrontEventDB(id={self.id}, event_type={self.event_type!r}, " + f"session_id={self.session_id!r})" + ) diff --git a/src/app/services/storefront_analytics/models/storefront_events_models.py b/src/app/services/storefront_analytics/models/storefront_events_models.py new file mode 100644 index 0000000..5e67423 --- /dev/null +++ b/src/app/services/storefront_analytics/models/storefront_events_models.py @@ -0,0 +1,96 @@ +""" +Storefront Analytics Schemas + +The ingest endpoint is **public**. Anyone who can load the shop can post to it, +so validation here is a boundary against abuse rather than a convenience for a +well-behaved client: every field is bounded, the event vocabulary is closed, and +the batch size is capped. +""" + +from __future__ import annotations + +from datetime import datetime +from enum import Enum +from uuid import UUID + +from pydantic import BaseModel, ConfigDict, Field, field_validator + +from app.shared.responses import BaseResponse + +# Cap on events per request. Enough for a page's worth of activity, small +# enough that a single request cannot be used to bulk-insert. +MAX_EVENTS_PER_BATCH = 50 + + +class StorefrontEventType(str, Enum): + """ + The closed set of things the storefront reports. + + Closed on purpose: an open string would let a client write arbitrary values + into a table an administrator later reads, and would make the funnel + definition drift silently as the frontend changed. + """ + + PAGE_VIEW = "page_view" + PRODUCT_VIEW = "product_view" + ADD_TO_CART = "add_to_cart" + CHECKOUT_STARTED = "checkout_started" + + +class StorefrontEventInput(BaseModel): + """One reported interaction.""" + + model_config = ConfigDict(extra="forbid") + + session_id: str = Field( + min_length=8, + max_length=64, + description="Opaque browser-generated id. Must not identify a person.", + ) + event_type: StorefrontEventType + path: str | None = Field(default=None, max_length=255) + sku: str | None = Field(default=None, max_length=100) + order_id: UUID | None = None + occurred_at: datetime + + @field_validator("path", mode="before") + @classmethod + def strip_query_string(cls, value: object) -> object: + """ + Keep the route, discard everything after ? or #, then bound the length. + + A query string is where personal data ends up by accident — an email in + a share link, a token in a redirect. Dropping it at the boundary means + it can never be stored, rather than relying on the client not to send it. + + Runs before validation rather than after so an over-long path is + truncated rather than rejected. A long URL is a client being untidy, not + a client being hostile, and discarding the whole event would lose a real + page view over it. + """ + if not isinstance(value, str): + return value + for separator in ("?", "#"): + value = value.split(separator, 1)[0] + return value[:255] + + +class StorefrontEventBatch(BaseModel): + """A batch of events from one browser.""" + + model_config = ConfigDict(extra="forbid") + + events: list[StorefrontEventInput] = Field( + min_length=1, + max_length=MAX_EVENTS_PER_BATCH, + description=f"At most {MAX_EVENTS_PER_BATCH} events per request", + ) + + +class StorefrontIngestResponse(BaseResponse): + """How many events were kept.""" + + accepted: int = Field(description="Events stored") + rejected: int = Field( + description="Events discarded, e.g. a timestamp outside the accepted window" + ) diff --git a/src/app/services/storefront_analytics/routers/__init__.py b/src/app/services/storefront_analytics/routers/__init__.py new file mode 100644 index 0000000..b08a016 --- /dev/null +++ b/src/app/services/storefront_analytics/routers/__init__.py @@ -0,0 +1,5 @@ +"""Storefront analytics routers.""" + +from .storefront_analytics_router import router as storefront_analytics_router + +__all__ = ["storefront_analytics_router"] diff --git a/src/app/services/storefront_analytics/routers/storefront_analytics_router.py b/src/app/services/storefront_analytics/routers/storefront_analytics_router.py new file mode 100644 index 0000000..23d57c4 --- /dev/null +++ b/src/app/services/storefront_analytics/routers/storefront_analytics_router.py @@ -0,0 +1,88 @@ +""" +Storefront Analytics Router + + POST /v1/analytics/events — anonymous shopper events from the storefront + +**This endpoint is public.** Anyone able to load the shop can post to it, which +shapes every decision here: + +- It is off unless the operator turns it on. Cloning OpenTaberna must not start + collecting anything. +- It is rate limited, because an open write endpoint otherwise fills a table. +- It validates strictly against a closed event vocabulary, so a client cannot + write arbitrary values into a table an administrator later reads. +- It stores nothing that identifies a person, so the worst an abusive client can + achieve is noise in a report. + +It returns 202: the events are accepted for storage, and the browser must not +wait on the outcome or retry on failure. Analytics that slow a shop down, or +that retry into a queue during an incident, have made things worse. +""" + +from __future__ import annotations + +from fastapi import APIRouter, Depends, Request, status +from sqlalchemy.ext.asyncio import AsyncSession + +from app.shared.config import get_settings +from app.shared.database.session import get_session_dependency +from app.shared.exceptions import NotFoundError +from app.shared.logger import get_logger +from app.shared.rate_limit import limiter + +from ..models import StorefrontEventBatch, StorefrontIngestResponse +from ..services import StorefrontEventRepository + +logger = get_logger(__name__) + +router = APIRouter() + +# Generous enough for a browsing session that batches on navigation, tight +# enough that a script cannot fill the table from one address. +_INGEST_RATE_LIMIT = "120/minute" + + +@router.post( + "/events", + response_model=StorefrontIngestResponse, + status_code=status.HTTP_202_ACCEPTED, + summary="Record anonymous storefront events", + description=( + "Accepts a batch of anonymous interaction events from the storefront.\n\n" + "Requires no authentication and records nothing that identifies a person: " + "no customer id, no email, no IP address and no user agent. A shopper is " + "represented only by an opaque session id their own browser generated.\n\n" + "Disabled unless the operator sets `STOREFRONT_ANALYTICS_ENABLED`, and " + "returns 404 when off so a deployment that has not opted in does not " + "advertise the capability.\n\n" + "Returns 202: events are accepted for storage and the browser should " + "neither wait on the result nor retry." + ), + responses={ + 404: {"description": "Storefront analytics is not enabled on this deployment."}, + 429: {"description": "Rate limit exceeded."}, + }, +) +@limiter.limit(_INGEST_RATE_LIMIT) +async def record_events( + request: Request, + batch: StorefrontEventBatch, + session: AsyncSession = Depends(get_session_dependency), +) -> StorefrontIngestResponse: + settings = get_settings() + + if not settings.storefront_analytics_enabled: + raise NotFoundError( + message="Storefront analytics is not enabled on this deployment", + context={"setting": "STOREFRONT_ANALYTICS_ENABLED"}, + ) + + repository = StorefrontEventRepository(session) + accepted, rejected = await repository.record(batch.events) + + return StorefrontIngestResponse( + success=True, + message="Events recorded", + accepted=accepted, + rejected=rejected, + ) diff --git a/src/app/services/storefront_analytics/services/__init__.py b/src/app/services/storefront_analytics/services/__init__.py new file mode 100644 index 0000000..95fcc30 --- /dev/null +++ b/src/app/services/storefront_analytics/services/__init__.py @@ -0,0 +1,13 @@ +"""Storefront analytics services.""" + +from .storefront_events_db_service import ( + MAX_CLOCK_SKEW, + StorefrontEventRepository, + get_storefront_event_repository, +) + +__all__ = [ + "MAX_CLOCK_SKEW", + "StorefrontEventRepository", + "get_storefront_event_repository", +] diff --git a/src/app/services/storefront_analytics/services/storefront_events_db_service.py b/src/app/services/storefront_analytics/services/storefront_events_db_service.py new file mode 100644 index 0000000..09654c2 --- /dev/null +++ b/src/app/services/storefront_analytics/services/storefront_events_db_service.py @@ -0,0 +1,218 @@ +""" +Storefront Analytics Database Service + +Writing anonymous events, and reading the browse portion of the funnel back out. +""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +from sqlalchemy import Select, and_, distinct, func, select +from sqlalchemy.ext.asyncio import AsyncSession + +from app.shared.logger import get_logger + +from ..models import StorefrontEventDB, StorefrontEventInput, StorefrontEventType + +logger = get_logger(__name__) + +# How far out of step with the server a browser's clock may be before its +# events are discarded. Client clocks are wrong often enough that rejecting all +# skew would lose real data, and trusting all of it would let anyone write to +# any point in history — including into a period an administrator has already +# reported on. +MAX_CLOCK_SKEW = timedelta(hours=24) + + +class StorefrontEventRepository: + """Persistence for storefront events.""" + + def __init__(self, session: AsyncSession) -> None: + self._session = session + + async def record(self, events: list[StorefrontEventInput]) -> tuple[int, int]: + """ + Store a batch, discarding events whose timestamp is not plausible. + + Returns: + (accepted, rejected) + """ + now = datetime.now(UTC) + earliest = now - MAX_CLOCK_SKEW + latest = now + MAX_CLOCK_SKEW + + rows: list[StorefrontEventDB] = [] + rejected = 0 + + for event in events: + occurred = event.occurred_at + if occurred.tzinfo is None: + occurred = occurred.replace(tzinfo=UTC) + + if not (earliest <= occurred <= latest): + rejected += 1 + continue + + rows.append( + StorefrontEventDB( + session_id=event.session_id, + event_type=event.event_type.value, + path=event.path, + sku=event.sku, + order_id=event.order_id, + occurred_at=occurred, + ) + ) + + if rows: + self._session.add_all(rows) + await self._session.commit() + + if rejected: + logger.info( + "Discarded storefront events with implausible timestamps", + extra={"rejected": rejected, "accepted": len(rows)}, + ) + + return len(rows), rejected + + # ------------------------------------------------------------------ + # Reading + # ------------------------------------------------------------------ + + @staticmethod + def _in_period(start: datetime, end: datetime): + return and_( + StorefrontEventDB.occurred_at >= start, + StorefrontEventDB.occurred_at < end, + ) + + async def _sessions_with( + self, start: datetime, end: datetime, event_type: StorefrontEventType | None + ) -> int: + """Distinct sessions that produced at least one event of this type.""" + statement: Select = select( + func.count(distinct(StorefrontEventDB.session_id)) + ).where(self._in_period(start, end)) + if event_type is not None: + statement = statement.where( + StorefrontEventDB.event_type == event_type.value + ) + result = await self._session.execute(statement) + return int(result.scalar_one_or_none() or 0) + + async def browse_funnel(self, start: datetime, end: datetime) -> dict[str, int]: + """ + Session counts at each pre-order step. + + Counted by distinct session rather than by event: ten product views from + one shopper is one person considering a purchase, not ten. + """ + return { + "sessions": await self._sessions_with(start, end, None), + "viewed_product": await self._sessions_with( + start, end, StorefrontEventType.PRODUCT_VIEW + ), + "added_to_cart": await self._sessions_with( + start, end, StorefrontEventType.ADD_TO_CART + ), + "started_checkout": await self._sessions_with( + start, end, StorefrontEventType.CHECKOUT_STARTED + ), + } + + async def page_views(self, start: datetime, end: datetime) -> int: + result = await self._session.execute( + select(func.count(StorefrontEventDB.id)).where( + self._in_period(start, end), + StorefrontEventDB.event_type == StorefrontEventType.PAGE_VIEW.value, + ) + ) + return int(result.scalar_one_or_none() or 0) + + async def top_paths( + self, start: datetime, end: datetime, limit: int = 10 + ) -> list[dict]: + """Most-viewed routes, with how many distinct sessions saw each.""" + statement: Select = ( + select( + StorefrontEventDB.path, + func.count(StorefrontEventDB.id).label("views"), + func.count(distinct(StorefrontEventDB.session_id)).label("sessions"), + ) + .where( + self._in_period(start, end), + StorefrontEventDB.event_type == StorefrontEventType.PAGE_VIEW.value, + StorefrontEventDB.path.is_not(None), + ) + .group_by(StorefrontEventDB.path) + .order_by(func.count(StorefrontEventDB.id).desc()) + .limit(limit) + ) + result = await self._session.execute(statement) + return [ + {"path": row.path, "views": int(row.views), "sessions": int(row.sessions)} + for row in result + ] + + async def product_interest( + self, start: datetime, end: datetime, limit: int = 10 + ) -> list[dict]: + """ + Views and cart adds per SKU, and the ratio between them. + + A product viewed often and added rarely is the interesting case: the + listing attracts people and something about the page, price or stock + turns them away. That is invisible in sales figures, which only ever + show what did sell. + """ + views = func.count(distinct(StorefrontEventDB.session_id)).filter( + StorefrontEventDB.event_type == StorefrontEventType.PRODUCT_VIEW.value + ) + adds = func.count(distinct(StorefrontEventDB.session_id)).filter( + StorefrontEventDB.event_type == StorefrontEventType.ADD_TO_CART.value + ) + + statement: Select = ( + select( + StorefrontEventDB.sku, + views.label("sessions_viewed"), + adds.label("sessions_added"), + ) + .where(self._in_period(start, end), StorefrontEventDB.sku.is_not(None)) + .group_by(StorefrontEventDB.sku) + .order_by(views.desc()) + .limit(limit) + ) + + result = await self._session.execute(statement) + rows = [] + for row in result: + viewed = int(row.sessions_viewed or 0) + added = int(row.sessions_added or 0) + rows.append( + { + "sku": row.sku, + "sessions_viewed": viewed, + "sessions_added": added, + "add_to_cart_rate": round(added / viewed, 4) if viewed else None, + } + ) + return rows + + async def checkout_order_ids(self, start: datetime, end: datetime) -> list: + """Orders that a browser reported starting checkout for.""" + result = await self._session.execute( + select(distinct(StorefrontEventDB.order_id)).where( + self._in_period(start, end), + StorefrontEventDB.event_type + == StorefrontEventType.CHECKOUT_STARTED.value, + StorefrontEventDB.order_id.is_not(None), + ) + ) + return [row[0] for row in result] + + +def get_storefront_event_repository(session: AsyncSession) -> StorefrontEventRepository: + return StorefrontEventRepository(session) diff --git a/src/app/shared/config/settings.py b/src/app/shared/config/settings.py index 417038a..89cc463 100644 --- a/src/app/shared/config/settings.py +++ b/src/app/shared/config/settings.py @@ -272,6 +272,15 @@ class Settings(BaseSettings): ) # Analytics / reporting + storefront_analytics_enabled: bool = Field( + default=False, + description=( + "Accept anonymous shopper events from the storefront. Off by " + "default: cloning OpenTaberna must not silently start collecting " + "anything, even something that identifies nobody. The ingest " + "endpoint returns 404 while this is false." + ), + ) shop_timezone: str = Field( default="Europe/Berlin", description=( diff --git a/tests/test_storefront_analytics_integration.py b/tests/test_storefront_analytics_integration.py new file mode 100644 index 0000000..8dc670f --- /dev/null +++ b/tests/test_storefront_analytics_integration.py @@ -0,0 +1,348 @@ +""" +Integration tests for storefront analytics (S2). + +Runs against the live stack with STOREFRONT_ANALYTICS_ENABLED=true: + + docker compose -f docker-compose.dev.yml up -d + +Endpoints covered: + POST /v1/analytics/events — public ingest + GET /v1/admin/analytics/storefront — the shopper funnel + +Sessions are tagged with a per-run prefix and removed on teardown, so figures +are exact regardless of what else is in the database. +""" + +import os +import subprocess +import uuid +from datetime import UTC, datetime, timedelta + +import pytest +import requests + +from auth_helpers import admin_headers + +_BASE = os.getenv("TEST_API_URL", "http://localhost:8000") +INGEST_URL = f"{_BASE}/v1/analytics/events" +FUNNEL_URL = f"{_BASE}/v1/admin/analytics/storefront" + +_HEADERS = admin_headers() + +RUN = uuid.uuid4().hex[:8] + + +def _psql(sql: str) -> str: + result = subprocess.run( + [ + "docker", + "exec", + "opentaberna-db", + "psql", + "-U", + "opentaberna", + "-d", + "opentaberna", + "-t", + "-A", + "-c", + sql, + ], + check=True, + capture_output=True, + text=True, + ) + return result.stdout.strip() + + +def _session(name: str) -> str: + return f"it-{RUN}-{name}" + + +def _now() -> str: + return datetime.now(UTC).isoformat() + + +def _post(events: list[dict]) -> requests.Response: + return requests.post(INGEST_URL, json={"events": events}) + + +@pytest.fixture(scope="module", autouse=True) +def enabled_or_skip(): + """These exercise collection, so a deployment with it off has nothing to test.""" + response = requests.get(FUNNEL_URL, headers=_HEADERS) + response.raise_for_status() + if not response.json()["enabled"]: + pytest.skip("STOREFRONT_ANALYTICS_ENABLED is false on this deployment") + yield + _psql(f"DELETE FROM storefront_events WHERE session_id LIKE 'it-{RUN}-%';") + + +# --------------------------------------------------------------------------- +# The endpoint is public by design +# --------------------------------------------------------------------------- + + +def test_ingest_needs_no_authentication(): + """A shopper who has not signed in is exactly who this is measuring.""" + response = _post( + [ + { + "session_id": _session("anon"), + "event_type": "page_view", + "path": "/shop", + "occurred_at": _now(), + } + ] + ) + + assert response.status_code == 202 + assert response.json()["accepted"] == 1 + + +def test_reading_the_funnel_requires_admin(): + assert requests.get(FUNNEL_URL).status_code in (401, 403) + + +# --------------------------------------------------------------------------- +# What must never be stored +# --------------------------------------------------------------------------- + + +def test_the_table_has_no_column_that_could_identify_a_person(): + """ + The privacy promise is structural, not a policy someone remembers. If a + column named for an address, an identity or a device ever appears, this + fails before it can collect anything. + """ + columns = set( + _psql( + "SELECT column_name FROM information_schema.columns " + "WHERE table_name = 'storefront_events';" + ).splitlines() + ) + + forbidden = { + "ip", + "ip_address", + "remote_addr", + "user_agent", + "email", + "customer_id", + "keycloak_user_id", + "user_id", + "first_name", + "last_name", + } + + assert not (columns & forbidden), f"PII column present: {columns & forbidden}" + + +def test_a_query_string_carrying_an_email_is_not_stored(): + response = _post( + [ + { + "session_id": _session("qs"), + "event_type": "page_view", + "path": "/shop?email=leak@example.com&token=secret", + "occurred_at": _now(), + } + ] + ) + assert response.status_code == 202 + + stored = _psql( + f"SELECT path FROM storefront_events WHERE session_id = '{_session('qs')}';" + ) + assert stored == "/shop" + assert "leak@example.com" not in stored + + +def test_an_unknown_event_type_is_refused(): + response = _post( + [ + { + "session_id": _session("bad"), + "event_type": "exfiltrate", + "occurred_at": _now(), + } + ] + ) + assert response.status_code == 422 + + +def test_extra_fields_are_refused(): + response = _post( + [ + { + "session_id": _session("extra"), + "event_type": "page_view", + "occurred_at": _now(), + "ip_address": "203.0.113.4", + } + ] + ) + assert response.status_code == 422 + + +# --------------------------------------------------------------------------- +# Timestamps come from a client and are not trusted +# --------------------------------------------------------------------------- + + +def test_events_far_outside_the_clock_skew_window_are_discarded(): + """ + A browser clock can be wrong; it must not be able to write into a period an + administrator has already reported on. + """ + long_ago = (datetime.now(UTC) - timedelta(days=400)).isoformat() + future = (datetime.now(UTC) + timedelta(days=400)).isoformat() + + response = _post( + [ + { + "session_id": _session("skew"), + "event_type": "page_view", + "occurred_at": long_ago, + }, + { + "session_id": _session("skew"), + "event_type": "page_view", + "occurred_at": future, + }, + { + "session_id": _session("skew"), + "event_type": "page_view", + "occurred_at": _now(), + }, + ] + ) + + body = response.json() + assert body["accepted"] == 1 + assert body["rejected"] == 2 + + +def test_modest_clock_skew_is_tolerated(): + """Rejecting all skew would lose real events from slightly-wrong clocks.""" + recent = (datetime.now(UTC) - timedelta(hours=2)).isoformat() + + response = _post( + [ + { + "session_id": _session("tolerant"), + "event_type": "page_view", + "occurred_at": recent, + } + ] + ) + + assert response.json()["accepted"] == 1 + + +# --------------------------------------------------------------------------- +# The funnel +# --------------------------------------------------------------------------- + + +def _steps(payload: dict) -> dict[str, int]: + return {step["step"]: step["sessions"] for step in payload["steps"]} + + +def test_funnel_counts_sessions_not_events(): + """ + Ten product views from one shopper is one person considering a purchase. + Counting events would make an indecisive browser look like a crowd. + """ + session = _session("repeat") + before = _steps(requests.get(FUNNEL_URL, headers=_HEADERS).json()) + + _post( + [ + { + "session_id": session, + "event_type": "product_view", + "sku": "IT-SKU", + "occurred_at": _now(), + } + for _ in range(5) + ] + ) + + after = _steps(requests.get(FUNNEL_URL, headers=_HEADERS).json()) + + assert after["viewed_product"] == before["viewed_product"] + 1 + + +def test_add_to_cart_rate_is_reported_per_sku(): + sku = f"IT-RATE-{RUN}" + _post( + [ + { + "session_id": _session("r1"), + "event_type": "product_view", + "sku": sku, + "occurred_at": _now(), + }, + { + "session_id": _session("r2"), + "event_type": "product_view", + "sku": sku, + "occurred_at": _now(), + }, + { + "session_id": _session("r1"), + "event_type": "add_to_cart", + "sku": sku, + "occurred_at": _now(), + }, + ] + ) + + payload = requests.get(FUNNEL_URL, headers=_HEADERS).json() + row = next(p for p in payload["product_interest"] if p["sku"] == sku) + + assert row["sessions_viewed"] == 2 + assert row["sessions_added"] == 1 + assert row["add_to_cart_rate"] == 0.5 + + +def test_the_paid_step_is_read_from_orders_not_from_the_browser(): + """ + A browser reporting "checkout started" only means a button was pressed. If + the funnel took its word for the outcome, any client could inflate the + conversion rate by claiming an order it never paid for. + """ + fake_order = str(uuid.uuid4()) + + _post( + [ + { + "session_id": _session("liar"), + "event_type": "checkout_started", + "order_id": fake_order, + "occurred_at": _now(), + } + ] + ) + + payload = requests.get(FUNNEL_URL, headers=_HEADERS).json() + steps = _steps(payload) + + # The claimed checkout is counted; the payment it did not make is not. + assert steps["started_checkout"] >= 1 + assert steps["paid"] < steps["started_checkout"] or steps["paid"] == 0 + + +def test_funnel_steps_are_ordered_and_carry_drop_off(): + payload = requests.get(FUNNEL_URL, headers=_HEADERS).json() + + assert [s["step"] for s in payload["steps"]] == [ + "sessions", + "viewed_product", + "added_to_cart", + "started_checkout", + "paid", + ] + assert payload["steps"][0]["drop_off_from_previous"] is None + assert all(s["drop_off_from_previous"] is not None for s in payload["steps"][1:]) diff --git a/tests/test_storefront_analytics_unit.py b/tests/test_storefront_analytics_unit.py new file mode 100644 index 0000000..9736f62 --- /dev/null +++ b/tests/test_storefront_analytics_unit.py @@ -0,0 +1,129 @@ +""" +Unit tests for storefront analytics — schema validation, no DB, no network. + +The ingest endpoint is public, so its schema is a security boundary rather than +a convenience. These pin the parts that keep it one. +""" + +from datetime import UTC, datetime + +import pytest +from pydantic import ValidationError as PydanticValidationError + +from app.services.storefront_analytics.models import ( + MAX_EVENTS_PER_BATCH, + StorefrontEventBatch, + StorefrontEventInput, + StorefrontEventType, +) + + +def _event(**overrides) -> dict: + base = { + "session_id": "abcdefgh1234", + "event_type": "page_view", + "occurred_at": datetime.now(UTC), + } + base.update(overrides) + return base + + +# --------------------------------------------------------------------------- +# Query strings are where personal data arrives by accident +# --------------------------------------------------------------------------- + + +def test_query_string_is_stripped_before_storage(): + """ + An email in a share link, a token in a redirect — a query string is how PII + reaches an analytics table without anyone deciding to send it. Dropping it + at the boundary means it cannot be stored, rather than trusting the client. + """ + event = StorefrontEventInput( + **_event(path="/shop?utm_source=mail&email=someone@example.com") + ) + + assert event.path == "/shop" + assert "email" not in event.path + + +def test_fragment_is_stripped_too(): + event = StorefrontEventInput(**_event(path="/shop/red-wine#reviews")) + assert event.path == "/shop/red-wine" + + +def test_plain_path_is_untouched(): + event = StorefrontEventInput(**_event(path="/shop/red-wine")) + assert event.path == "/shop/red-wine" + + +def test_path_is_length_bounded(): + event = StorefrontEventInput(**_event(path="/" + "x" * 500)) + assert len(event.path) <= 255 + + +# --------------------------------------------------------------------------- +# The event vocabulary is closed +# --------------------------------------------------------------------------- + + +def test_unknown_event_types_are_rejected(): + """ + An open string would let any client write arbitrary values into a table an + administrator later reads, and would let the funnel definition drift as the + frontend changed. + """ + with pytest.raises(PydanticValidationError): + StorefrontEventInput(**_event(event_type="admin_password_grab")) + + +def test_every_declared_event_type_is_accepted(): + for event_type in StorefrontEventType: + event = StorefrontEventInput(**_event(event_type=event_type.value)) + assert event.event_type is event_type + + +# --------------------------------------------------------------------------- +# Nothing unexpected gets through +# --------------------------------------------------------------------------- + + +def test_extra_fields_are_refused_rather_than_ignored(): + """ + Silently dropping an unknown field would let a client believe it is sending + an email address that the API is quietly discarding. Refusing says so. + """ + with pytest.raises(PydanticValidationError): + StorefrontEventInput(**_event(email="someone@example.com")) + + with pytest.raises(PydanticValidationError): + StorefrontEventInput(**_event(ip_address="203.0.113.4")) + + +def test_session_id_is_length_bounded_in_both_directions(): + with pytest.raises(PydanticValidationError): + StorefrontEventInput(**_event(session_id="short")) + + with pytest.raises(PydanticValidationError): + StorefrontEventInput(**_event(session_id="x" * 200)) + + +# --------------------------------------------------------------------------- +# Batch limits +# --------------------------------------------------------------------------- + + +def test_batch_size_is_capped(): + """An open write endpoint must not accept an unbounded insert.""" + with pytest.raises(PydanticValidationError): + StorefrontEventBatch(events=[_event() for _ in range(MAX_EVENTS_PER_BATCH + 1)]) + + +def test_a_full_batch_is_accepted(): + batch = StorefrontEventBatch(events=[_event() for _ in range(MAX_EVENTS_PER_BATCH)]) + assert len(batch.events) == MAX_EVENTS_PER_BATCH + + +def test_empty_batch_is_rejected(): + with pytest.raises(PydanticValidationError): + StorefrontEventBatch(events=[])