diff --git a/.gitignore b/.gitignore index 14ff8cd..af9ecff 100644 --- a/.gitignore +++ b/.gitignore @@ -14,7 +14,7 @@ dist/ downloads/ eggs/ .eggs/ -lib/ +/lib/ lib64/ parts/ sdist/ diff --git a/README.md b/README.md index ff1bc98..41d47d6 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ The platform deliberately separates source facts, normalized source values, dete | Capability | Status | Current evidence / limitation | |---|---|---| | Cached UK organization ingestion | Complete | 65 normalized UK-side organizations in the current rebuilt database | -| Cached 360Giving grant ingestion | Complete | 34,478 observed transactions with source provenance; original currency is retained per grant | +| Cached 360Giving grant ingestion | Sampled | 200,000 observed 360Giving transactions with source provenance; the active database is a reproducible capped sample, not complete 360Giving coverage | | Cached Philea organization ingestion | Complete | 299 records; organization-level only, with no grants assigned | | DACH foundation intelligence | Partial | Philea and deterministic geography normalization provide some European/DACH discoverability; this is not a complete DACH registry or grant dataset | | Organization directory and detail | Complete | SQLite-backed API and UI, with source/type/coverage labels | @@ -69,7 +69,7 @@ The checked-in source caches currently contain 62 Charity Commission records, a The current regenerated database contains: - 364 organizations: 65 primarily from the Charity Commission/360Giving and 299 from Philea. -- 34,478 observed 360Giving grants in GBP, USD, EUR, ILS, and CHF. Original amount and currency remain source facts. +- 200,000 observed 360Giving grants in GBP, USD, EUR, ILS, and CHF. Original amount and currency remain source facts; this is the current reproducible ingestion cap, not the complete 360Giving dataset. - Auto EUR display uses the official daily ECB EXR reference rate for the award date; weekends and ECB holidays use the previous published business-day rate. Each converted grant stores the conversion status, rate, rate date, and source. 34 pre-1999 GBP grants remain explicitly unconverted because no ECB EUR reference rate exists for their award dates. - 299 Philea organizations marked `organization_level_only`; no grant is attached to a synthetic Philea ID. @@ -124,17 +124,29 @@ Geographic concepts are intentionally distinct: - Geographic focus: where an organization says it works or funds; source and inferred values remain separate. - Beneficiary/project geography: a transaction's destination; used by recipient-region filters and the map. -The `Global Grant Distribution` map resolves beneficiary-country geography for 13,123 of the 34,478 currently ingested grants (38.06%), above the default 30% display threshold. It uses `beneficiary_geography_normalized` first and falls back only to explicit ISO country codes or explicit country names in the original `beneficiary_geography` source field. It never consults funder headquarters, recipient registered offices, or inferred operating regions. England, Scotland, Wales, and Northern Ireland roll up to the United Kingdom shape while their original labels are retained in country detail. The remaining 21,355 records are reported as unmapped rather than assigned a fabricated country. +The `Global Grant Distribution` map currently resolves beneficiary-country geography for 71,286 of the 200,000 ingested grants (35.64%). These produce 71,403 country associations; 62 grants have multiple associated countries. It uses `beneficiary_geography_normalized` first and falls back only to explicit ISO country codes or explicit country names in the original `beneficiary_geography` source field. It never consults funder headquarters, recipient registered offices, or inferred operating regions. England, Scotland, Wales, and Northern Ireland roll up to the United Kingdom shape while their original labels are retained in country detail. The remaining 128,714 records are reported as unmapped rather than assigned a fabricated country. The Overview `Filters` button opens a non-layout-shifting, scrollable global grant-filter drawer for award date, currency, beneficiary geography, programme area, donor, recipient, and time aggregation. `Auto · EUR converted` includes every eligible source currency using stored historical ECB rates; selecting `GBP`, `USD`, `EUR`, `CHF`, or `ILS` instead shows only grants originally recorded in that currency. One applied grant scope drives the Overview KPIs, map, trends, and programme allocation; coverage counters explicitly change from ingested to filtered grants. The map header retains only display controls and the illustrative-connections toggle. Organization-directory search, income/expenditure, and headquarters filters remain independent because they describe organization records rather than the filtered grant population. -Selecting a country exposes a direct handoff to the Organization Directory. The handoff preserves the active organization filters and applies the selected country as beneficiary geography. Directory geography matching uses the same canonical resolver as the map, including explicit raw ISO-code/name fallback and UK constituent-country roll-up. A country can still have no Directory result when every observed 360Giving funder for that country is source-only and lacks a matched organization profile; the UI explains that state rather than displaying a blank grid. +Selecting a country opens the primary **Donor Directory** rather than applying the country as an organization-profile location filter. It uses the map's canonical beneficiary-country association (including the UK roll-up) and preserves currency, date, beneficiary geography, programme area, donor, recipient, and selected sources through a shared typed URL contract. This means a country with observed funding leads to the source-funder evidence that produced the map instead of an often-empty registered-address lookup. + +### Observed Donor Directory + +`GET /api/charities/grants/funders` is a derived, paginated aggregation over the stored grant population; it does not create a new organization profile or use external data. A source funder has a deterministic identity made from its source namespace plus `funding_org_source_id`; only where that ID is unavailable does it fall back to the normalized `funding_name`. The identity deliberately does not change when an optional enriched-profile link is later added. A narrow reproducible `grant_source_funder_facts` table keeps list filtering, SQL aggregation, sorting, and pagination away from the wide raw-JSON grant column. + +The endpoint requires `beneficiary_country` as a canonical ISO alpha-2 code and accepts the same grant-scope fields as the Overview (`currency`, `date_from`, `date_to`, `beneficiary_geographies`, `programme_areas`, `donor`, `recipient`, and `sources`) plus backend `search`, `profile_status`, `sort`, `page`, and `page_size`. It returns typed source identity, observed activity, amount policy, evidence sources, and explicit zero/one/many profile-link status. `GET /api/charities/grants/funders/{source_funder_key}` supports `detail_level=summary|full`; the primary UI loads full grant/recipient/evidence sections only when opened. + +For a selected country, multi-country grants count once for activity and recency, but their full amount is excluded from the country-attributable funding total. In `currency=auto`, only stored EUR values with `native_eur`, `ecb_award_date`, or `ecb_previous_business_day` conversion status are monetary eligible. In an explicit currency mode, only original grants in that selected currency are considered. A verified Directory profile is linked only where it already exists; source-only funders remain source-only and must never be converted into invented profiles. + +The current active 200,000-grant database is a reproducible **sampled ingestion cap**, not a complete 360Giving dataset. The import record is `src/data/processed/360giving_registry_publisher_pilot_import_200000.json`. The optimized source-funder list uses normalized beneficiary/programme indexes and a versioned narrow fact table; supported loaders invalidate it after data changes. See `docs/donor_directory_performance_report.md` for measured GB/US repository timings and the explicit HTTP/browser validation limits. + +The map CTA writes `funder_country` plus canonical `grant_*` filters to the URL. Search, status, sorting, page, sources, and selected donor also survive refresh/back/forward and copied URLs. Desktop uses a compact list plus right-side detail panel; tablet/mobile use a full-width/full-screen sheet. Real-browser acceptance is still outstanding and the legacy donor view remains available; see `docs/donor_directory_responsive_validation.md`. Directory profiles without a cached raw Charity Commission detail object still expose a schema-valid partial detail view assembled from normalized organization fields. Their registration status is reported as `UNKNOWN`, unavailable contact/financial sections remain empty, and the API does not invent missing source values or fail the entire profile request. An optional connection layer draws up to the 36 strongest registered-funder-location-to-beneficiary-country associations. The origin uses an explicit 360Giving funding-organization address country where present and otherwise the matched organization directory's registered headquarters country. These arrows are labelled in both settings and the map as illustrative associations, not verified financial routes; headquarters never substitutes for beneficiary geography or proves where a payment originated. -Grant-count mode counts a grant once in each explicitly associated country; the UI therefore labels the metric as grant-country associations when multi-country records are present. The current cache contains 39 such grants. Funding mode includes only non-negative, single-country amounts in one selected currency. It never repeats or invents allocations for multi-country awards: the current GBP map reports £59,831,607.31 as excluded multi-country amount. The map's country totals must not be added to that excluded amount or interpreted as complete 360Giving/global-market coverage. +Grant-count mode counts a grant once in each explicitly associated country; the UI therefore labels the metric as grant-country associations when multi-country records are present. The current cache contains 58 such grants. Funding mode includes only non-negative, single-country amounts in one selected currency. It never repeats or invents allocations for multi-country awards. The map's country totals must not be added to excluded multi-country amounts or interpreted as complete 360Giving/global-market coverage. ## Grant overview aggregations @@ -148,7 +160,7 @@ The programme chart defaults to the largest substantive categories, groups the r ## Explainable relevance score -No client-approved score definition, notes, target variable, or weights were found. The included `example-relevance-v1` configuration is therefore explicitly `experimental` and measures only relevance to a selected target profile. It is not a probability, recommendation, financial forecast, or prediction of donation behavior. +No client-approved score definition, notes, target variable, or weights were found. The included `example-relevance-v2` configuration is therefore explicitly `experimental` and measures only relevance to a selected target profile. It is not a probability, recommendation, financial forecast, or prediction of donation behavior. Default example weights are thematic fit 0.35, geographic fit 0.25, funding-capacity fit 0.15, historical grant-size fit 0.15, and organization-type fit 0.10. Component calculations expose their inputs, method, confidence, and missing reason. Overall confidence and data completeness are returned separately from the score. Missing components are excluded by renormalizing the available weights; they are never silently scored as zero. Financial/grant comparisons occur only when currencies match. @@ -326,6 +338,8 @@ All `/api/*` routes require the session cookie returned by `POST /api/auth/login | `GET /api/charities/{id}/grants?role=all|funder|recipient` | Observed transactions and coverage status | | `GET /api/charities/grants/summary` | Currency-separated network totals and rankings | | `GET /api/charities/grants/map?currency=GBP` | Filterable beneficiary-country associations, currency-safe funding totals, disclosed HQ-to-beneficiary connection groups, country explorer rankings, and coverage/exclusion metadata | +| `GET /api/charities/grants/funders?beneficiary_country=US` | SQL-filtered/paginated observed donor ranking with backend `search`, `profile_status`, sort, and canonical grant-scope filters | +| `GET /api/charities/grants/funders/{source_funder_key}?beneficiary_country=US&detail_level=summary` | Selected source-funder detail; summary-first with lazy full recipients, grants, and typed evidence, never a synthetic organization profile | | `GET /api/charities/grants/trends?currency=GBP&months=24` | Award-date monthly grant totals with unknown-coverage months and exclusions | | `GET /api/charities/grants/themes?currency=GBP` | Minor-unit-preserving programme allocations and classification coverage | | `GET /api/charities/{id}/sankey` | Auto EUR-converted observed donor-to-recipient flow; a concrete currency parameter retains source-currency-only behaviour | @@ -356,6 +370,8 @@ PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src \ # Frontend reproducible install and production build cd frontend npm ci +npm run test +npm run lint npm run build ``` @@ -364,7 +380,7 @@ GitHub Actions runs separate Python 3.12 backend and Node.js 22 frontend jobs. T ## Presentation flow 1. Start on Overview and show the full-width Global Grant Distribution map. Switch between grant-country associations and GBP funding, select a country to open its slim explorer, then show Grant Awards Over Time and Grant Allocation by Programme Area in the balanced row below it. -2. Open Organization Directory and demonstrate server-side Charity Commission name/number search, registration and financial filters, the Enriched profile / Observed grant data flags, and cursor-based “Load more”. +2. Select a map country and open Donor Directory. Demonstrate observed/linked status, backend search/sort/page, the right-side donor detail, ranked recipients, and explicit source evidence. Open Organization Research or Advanced Charity Commission Search only as secondary research tools. 3. Open Charity Projects (`326568`, whose source grant records use the funder name Comic Relief) to show Charity Commission identity/provenance, source versus inferred classifications, evidence/review state, observed 360Giving grants, and the donor-to-recipient Sankey. 4. Show the map's 62.86% known-country disclosure and 39 multi-country exclusions; explain why headquarters is not substituted for missing transaction geography and why multi-country amounts are not divided or duplicated. 5. Open Women Win (`-24788`) to show Philea organization type/source and the explicit `organization_level_only` transaction status. @@ -380,6 +396,6 @@ GitHub Actions runs separate Python 3.12 backend and Node.js 22 frontend jobs. T - When the BFF is offline, KPI/cards/detail/news/admin simulation use clearly labelled local mock content; grant charts, flows, map values, and scores remain unavailable rather than fabricated. - The news route depends on current external pages, Google News decoding, and Claude-compatible credentials, so it is not deterministic. - Demo authentication defaults and frontend-visible credentials are suitable only for local presentation use. -- The production frontend build currently emits a bundle-size warning above Vite's 500 kB advisory threshold; code splitting is future work. +- The production frontend build still emits a bundle-size warning above Vite's 500 kB advisory threshold. Registry and legacy donor views are now split, while remaining legacy profile/chart code still requires further extraction. - Live pipeline modes depend on upstream availability and can take time. Use cached consolidation for reproducibility. - Hinchilla code and historical preprocessed artifacts remain in the repository for compatibility/reference, but Hinchilla is not part of the active presentation rebuild. diff --git a/config/scoring.example.json b/config/scoring.example.json index 1b52391..20a7922 100644 --- a/config/scoring.example.json +++ b/config/scoring.example.json @@ -1,8 +1,8 @@ { - "score_version": "example-relevance-v1", + "score_version": "example-relevance-v2", "configuration_status": "experimental", "score_target": "relevance of an organization to a user-selected target profile", - "missing_data_behavior": "renormalize_available_components", + "missing_data_behavior": "zero_for_missing_components", "review_confidence_threshold": 0.6, "weights": { "thematic_fit": 0.35, @@ -22,7 +22,7 @@ "assumptions": [ "This is an example configuration and is not client-approved.", "The score supports exploration of target-profile relevance; it is not a donation probability or prediction.", - "Missing components are disclosed and excluded from relevance arithmetic rather than treated as zero relevance.", + "The final score is the weighted sum of all criteria; missing criteria contribute zero rather than inflating a partial match.", "Financial and grant-size comparisons are made only within the requested source currency." ] } diff --git a/docs/current_ui_and_donor_integration_audit.md b/docs/current_ui_and_donor_integration_audit.md new file mode 100644 index 0000000..19dd95a --- /dev/null +++ b/docs/current_ui_and_donor_integration_audit.md @@ -0,0 +1,766 @@ +# Current UI and donor-integration audit + +**Audit date:** 24 July 2026 +**Branch / commit audited:** \`9-fr-09-display-ranked-list-of-main-donors\` / \`9c46f275dcf5140d97bc45d56099d036bf602d00\` +**Scope:** current-state audit only. No application route, component, API, schema, or dataset was changed for this report. + +## 1. Executive Summary + +The application currently contains three different entity directories under one visual area: + +1. **Main Donors** is an observed-grant view. A row is a run-time grouping of 360Giving grant records by source-funder identity, not necessarily an organisation profile. +2. **Organization Profiles** is an enriched-directory view. A row is one of 364 \`charities\` profile records, mostly Philea records; it may have no observed grant relationship. +3. **Full Charity Commission Register** is an official-registry search. A row is one of 397,469 current Charity Commission records, not automatically a donor. + +This distinction is technically real and must survive a UI simplification. The desired two-screen journey is feasible if the new Donor Directory is based on observed source funders, and profile/registry data is attached only as explicitly evidenced enrichment. It is *not* safe to merge all three row types into one undifferentiated “organization” object. + +The map hand-off to Main Donors is already substantially implemented: selecting a beneficiary country can open a filtered observed-funder list and carries most grant scope in the query string. Its biggest state gap is that the global data-source selection is not reconstructed by the Overview when returning to the map. The current directory pages are separate modes inside a single React SPA rather than stable, independently routed pages. + +The primary user-facing slow path is Main Donors for a large country such as GB: it currently takes about 13.7 seconds in an in-process repository measurement because the backend fetches and parses many grant rows, then groups them in Python. The world-map overview is cache-backed and fast after warming, but its first computation is approximately 3.3 seconds and a cold yearly trend request approximately 6.7 seconds. The profile list and registry page are comparatively fast in the measured local database. + +The existing “Observed Funding Flows” renderer has real data for linked profiles such as Oxfam, but its Sankey presentation is visually unreliable for many-to-one relationships: labels clip at the left edge and links converge into an unreadable point. This is a visualisation problem, not proof that the data is empty. + +## 2. Audit Scope and Limitations + +### What was verified + +- Branch, commit, dirty worktree, local runtime health, local Vite frontend availability, database-backed API implementation, data counts, API/query paths, production build, lint, tests, and selected repository timing measurements. +- BFF health endpoint returned HTTP 200 at \`http://127.0.0.1:8000/health\`. +- A Vite application instance returned HTTP 200 at \`http://127.0.0.1:5175/\`. The documented default is normally port 5173; ports 5173 and 5174 were already occupied. +- \`PYTHONPATH=src ./venv/bin/python -m pytest -q\` passed: **242 passed** in 12.70s, with 39 existing deprecation warnings. +- \`frontend/npm run build\` and \`frontend/npm run lint\` completed successfully. Lint reports five existing exhaustive-dependency warnings in \`frontend/src/App.tsx\`. + +### Important limits + +- No browser automation, installed Chromium/Chrome, Playwright package, or screenshot-capable browser was available in this environment. The target-viewport section is therefore **code-inferred**, not rendered-device verified. Existing user-provided screenshots were considered only as corroborating evidence, not newly captured test artifacts. +- The performance timings are local, in-process repository timings serialised to JSON. They exclude HTTP, authentication, browser layout/paint, and network latency. +- The repository was already dirty. The listed changes belong to the active worktree and were not altered. This audit adds only this Markdown report. +- News fetching and any external AI/news service behaviour were inspected from code paths but not exercised against external services. + +### Runtime assumptions and setup + +The browser application uses the BFF at \`VITE_API_BASE_URL\` when supplied, otherwise \`//:8000\`; see \`frontend/src/App.tsx:72-80\`. The BFF uses the local SQLite database \`src/data/charities.db\` through \`src/bff/repositories.py:4721-4738\`. Authentication is a local demo-cookie/JWT flow; all charity endpoints are protected in \`src/bff/charity.py:24-28\`. + +Documented local commands are: + + ./start_backend.sh + cd frontend && npm run dev + cd frontend && npm run build + PYTHONPATH=src ./venv/bin/python -m pytest -q + +\`start_backend.sh\` launches Uvicorn on 127.0.0.1:8000 with reload; Docker Compose supplies the BFF-only alternative. No migration or missing-environment-variable issue blocked the locally running BFF during this audit. + +## 3. Verified Runtime and Architecture + +| Layer | Current implementation | Evidence | +|---|---|---| +| Frontend | React 19.2.7, TypeScript, Vite | \`frontend/package.json\` | +| Navigation | Manual SPA state plus \`window.location.pathname\`, \`history.pushState\`, and \`popstate\`; no React Router | \`frontend/src/App.tsx:410-484\` | +| Styling | Project CSS in \`frontend/src/index.css\`, utility-like class strings and substantial inline component styles | \`frontend/src/index.css\`, \`frontend/src/App.tsx\` | +| Client state | Local React state/effects; URL query parameters for portions of grant state; no Redux/Zustand/query cache | \`frontend/src/App.tsx:485-894\`, \`frontend/src/components/OverviewDashboard.tsx:145-257\` | +| Maps/charts | \`@svg-maps/world\`, Recharts, Leaflet / react-leaflet, Lucide icons | \`frontend/package.json\`, \`GrantWorldMap.tsx\` | +| Backend/BFF | FastAPI, repository layer | \`src/bff/main.py:51-91\`, \`src/bff/repositories.py\` | +| Database | SQLite \`src/data/charities.db\` | \`src/bff/repositories.py:4721-4738\` | +| Aggregate cache | \`grant_overview_cache\` table, startup warming for three selected sources | \`src/data/db_loader.py:229-249\`, \`src/bff/main.py:51-68\` | + +The BFF startup event is deprecated FastAPI syntax, which accounts for part of the test warning set; it is not an immediate user-facing failure. The frontend auto-logins against \`/api/auth/login\` before protected data calls; see \`frontend/src/App.tsx:87-108\`. + +The production build emitted one significant bundle signal: the main JavaScript output is **1,952.10 kB / 607.87 kB gzip**, above Vite’s 500 kB warning threshold. The map and chart libraries are currently included in the initial application bundle rather than clearly lazy-loaded by view. + +## 4. Current Navigation and Route Inventory + +There is one practical browser path, normally \`/\`, rather than three independently routed directory pages. \`App\` selects \`activeTab\` (\`overview\`, \`directory\`, \`admin\`) and \`directoryMode\` (\`profiles\`, \`registry\`, \`funders\`). The only special path detection is the presence of a \`funder_country\` query parameter; it opens the funder mode on initial load/back navigation. See \`frontend/src/App.tsx:410-484\`. + +| User-facing area | Current path / state | Primary component(s) | Fetch/API | Loading / empty / error | Reachability and duplication | +|---|---|---|---|---|---| +| Funding Landscape / Overview | \`/\`, \`activeTab=overview\` in memory; grant query params | \`App\`, \`OverviewDashboard\`, \`GrantWorldMap\` | \`GET /api/charities/grants/overview\`, optionally \`/overview/trends\` | Map/dashboard loading and fallback states; BFF unavailable message | Primary sidebar destination. A legacy overview exists only when \`SHOW_LEGACY_OVERVIEW\` is enabled; \`App.tsx:1589+\`. | +| World map country action | Same view; selected ISO country in query and component state | \`GrantWorldMap\` | Aggregate overview payload | Map supplies selection/loading/help affordance | “Explore active funders” invokes hand-off; map also has optional illustrative connections. | +| Main Donors | \`/?funder_country=GB&...\`; \`activeTab=directory,directoryMode=funders\` | \`SourceFunderResults\` | \`GET /api/charities/grants/funders\`; detail \`/funders/{key}\` | Explicit loading, empty and error states | Reachable from map and Directory tab. New, currently untracked worktree component: \`frontend/src/components/SourceFunderResults.tsx\`. | +| Organization Profiles | \`/\`; \`activeTab=directory,directoryMode=profiles\` | \`App\` profile list and inline detail modal | \`GET /api/charities\`, \`/{id}\`, \`/{id}/grants\`, \`/{id}/sankey\`, \`POST /{id}/score\` | List loading/empty/error; profile modal data states | Reachable tab. Overlaps Main Donors visually but uses a different row entity. | +| Full Charity Commission Register | \`/\`; \`activeTab=directory,directoryMode=registry\` | \`RegistryDirectory\` | \`GET /api/charities/directory/organizations\`, item detail endpoint | Loading, error, no-result, cursor load-more | Reachable tab; an advanced official-register utility, not a donor list. | +| Profile details | In-memory \`selectedCharity\`; no shareable detail URL | Inline modal in \`App\` | Four concurrent calls after selection; news on demand | Modal conditions and API errors | Selecting a profile card opens it; close returns to mounted list/filter state. | +| Source-funder details | In-memory selected item; no shareable detail URL | \`App\` source-funder modal / Recharts Sankey | \`GET /api/charities/grants/funders/{source_funder_key}\` | Source list/modal error and loading conditions | From Main Donors row/detail control. Current modal is visually distinct from profile detail. | +| Registry record details | In-memory \`selectedOrganization\` | \`RegistryDirectory\` dialog | Detail request by registry identifier | Modal loading/error | A linked enriched profile can be opened from this dialog. | +| Data sources | Header toggle / global selection | \`App\` header control | Passed to overview/profile calls; source strings in URL only on donor handoff | No separate request | It is visually global but does not consistently affect every directory mode. | +| News | Profile detail only, user initiated | \`App\` | Profile news endpoint | On-demand state | Not a directory-list concern. | + +The sidebar and header are rendered at \`frontend/src/App.tsx:1442-1564\`. Directory tabs are mounted at \`App.tsx:1790-1861\`; the profile list runs through \`App.tsx:1863-2147\`; profile detail begins around \`App.tsx:2365\`. + +## 5. Current Main Donors View + +### Purpose and entity + +The title and controls communicate “source funders by beneficiary geography.” This is the appropriate answer to “which organisations have been observed awarding grants associated with this country?” It is **not** an organization-profile directory. The endpoint is \`GET /api/charities/grants/funders\` in \`src/bff/charity.py:303-345\`, backed by \`SQLiteCharityRepository._source_funder_scope\` at \`src/bff/repositories.py:1716-1979\`. + +A source-funder identity is computed from: + +- grant source + raw funding-organisation source identifier when available; otherwise +- grant source + a normalised donor name. + +This produces 292 distinct observed source-funder identities in the present 200,000-grant 360Giving dataset. It is a query-time group; **there is no \`source_funders\` database table**. Name fallback may split one real organisation across spelling variants or merge ambiguously similar names. It must therefore retain identity method and source provenance. + +### Inclusion, exclusion, aggregation and labels + +The UI reads the following URL parameters in \`SourceFunderResults.tsx:141-210\`: + +\`funder_country\`, \`grant_currency\`, \`grant_from\`, \`grant_to\`, \`grant_geo\`, \`grant_programme\`, \`grant_donor\`, \`grant_recipient\`, \`grant_sources\`, \`funder_sort\`, and \`funder_page\`. + +The request sends beneficiary country, currency, sort, page/page_size=25, date range, beneficiary geographies, programme areas, donor, recipient, and sources. It groups source grant rows after those scopes are applied. Single-country grants contribute to country funding totals; multi-country grants contribute to observed activity but their full value is excluded from country-attributable money. The UI identifies this exception. In automatic currency mode it uses historical EUR conversion; a selected source currency restricts rather than converts the scope. Conversion is based on original grant date and stored ECB rates where available. + +Displayed information: + +| Default visible field | Meaning / source | Classification | Recommendation | +|---|---|---|---| +| Rank and funder name | Query-time source identity | Primary decision information | Keep | +| Identity method | Source ID versus normalised-name fallback | Data-quality/provenance | Move behind a compact “Observed source” status or details | +| Grant count, distinct recipients, latest activity | Aggregated observed source grants | Primary/supporting | Keep compactly | +| Country-attributable observed funding | Single-country amount, automatic EUR or selected original currency | Primary decision information | Keep; retain multi-country caveat in details | +| Leading programme areas | Derived grant programme categories | Supporting | Keep at most two; all in details | +| Data source and link status | 360Giving provenance / linked profile status | Provenance | Keep short text/badge; never colour-only | +| “Source-only” | No direct linked profile is available | Data-quality/provenance | Keep, with clear explanation | +| “Open verified profile” action | Only when exactly one direct linked profile is returned | Primary action where available | Keep, but call it “Open linked profile” unless a stronger reviewed-verification contract is established | +| Six summary tiles | Scope-level identity/grant/recipient/funding/link counts | Supporting | Keep only the 2–3 decision-critical totals by default; remainder under scope details | +| Eight permanent grant controls | Scope controls | Supporting / administrative | Put advanced controls in a drawer; preserve state and API support | + +### Linkage and interaction + +The source-funder scope records a linked directory profile only when a grant \`funding_charity_id\` joins an actual \`charities.charity_id\` and has a compatible linked funder name; see \`repositories.py:1886-1889\`. It reports a link only if precisely one directory ID is associated with the source identity; it intentionally does not choose arbitrarily when several IDs exist (\`repositories.py:1982-2020\`). This is safer than auto-enrichment but should be described as “direct linked profile” rather than silently equivalent to identity verification. + +Clicking a source funder fetches a source-funder detail and opens an in-memory modal. That state has no URL, so it cannot be deep-linked and browser Back cannot reproduce it. Pagination/sort/filter state are URL-backed and preserved. The detail presents one donor-to-many-recipient flow, capped to 15 links; see \`repositories.py:2023-2151\` and \`App.tsx:2739-2842\`. + +### Source evidence, 360Giving organisation links and website links + +This is feasible with the data already stored, but it is a **multi-hop evidence chain**, not one generic \`source_url\`. + +#### Verified current chain + +The raw 360Giving payload stored in \`grants.raw_grant_data\` contains organisation API links and, when the publisher supplied it, the organisation website. Both are different from the publisher URL already exposed by the API: + + Grant raw JSON + ├─ recipients[].self → 360Giving recipient organisation JSON + │ └─ grants_received → grant-record JSON, including recipientOrganization[].url when supplied + ├─ funders[].self → 360Giving funder organisation JSON + │ └─ grants_made → grant-record JSON, including fundingOrganization[].url when supplied + ├─ data.recipientOrganization[].url → recipient website supplied in the observed grant record + ├─ data.fundingOrganization[].url → funder website supplied in the observed grant record + └─ data.dataSource → publisher's own website / open-grants data + +For example, the stored Oxfam grant \`360G-BMT-2022-2-OXF\` contains: + + recipients[0].self = https://api.threesixtygiving.org/api/v1/org/GB-CHC-202918/ + funders[0].self = https://api.threesixtygiving.org/api/v1/org/GB-CHC-1076925/ + data.dataSource = https://www.brianmercertrust.org/ + +The recipient/funder \`self\` URL resolves to the public 360Giving organisation JSON. The live Oxfam recipient response verified during this audit gives its canonical ID, name, grant aggregates, linked organisation IDs, and a \`grants_received\` URL. Opening that second JSON response returns individual grant records. Some Oxfam records contain \`data.recipientOrganization[0].url = http://www.oxfam.org.uk\`. This is the exact navigation chain described in the request. + +The organisation endpoint itself does **not** contain a website field. The individual grant record’s recipient/funder \`.url\` is supplied by the grant publisher. Separately, the linked enriched/Charity Commission profile \`charities.charity_id=202918\` has \`website=www.oxfam.org.uk\` and an official Charity Commission \`source_url\`. These can agree, but remain distinct provenance sources. + +That means the honest chain is: + + observed grant + → 360Giving recipient/funder organisation JSON + → grants_received / grants_made JSON + → optional recipient/funder website embedded in an observed grant + → optional direct/accepted local profile or registry resolution + → separately sourced organisation website from profile/official-record metadata + +Each final website must carry its own origin label: either “website supplied in this 360Giving grant record” or “website from linked Charity Commission/profile data.” + +#### Current coverage and implementation state + +- Every current grant has a non-empty, valid HTTP(S) publisher \`source_url\` and a \`source_record_id\`. They are stored in \`grants\` by \`src/data/db_loader.py:195-200, 484-522\`, selected at \`src/bff/repositories.py:1037-1055, 4387-4438\`, and exposed as \`GrantDetail.source_url\` in \`src/bff/schemas.py:297-322\`. +- **183,947** raw grants expose \`recipients[0].self\`; all **200,000** expose \`funders[0].self\`. Neither is currently normalised into a dedicated database column or emitted by the grant/funder endpoints; they remain inside \`raw_grant_data\`. +- **72,906** grants contain \`data.recipientOrganization[0].url\`, representing 35,841 distinct recipient website values. **3,594** grants contain \`data.fundingOrganization[0].url\`, representing five distinct funder website values. These are the publisher-provided organisation website fields found by following the 360Giving organisation-record path; they are also retained locally in \`raw_grant_data\`. +- The API already returns a representative publisher \`source_url\` for a grouped source-funder row (\`SourceFunderItem.representative_source_url\`, \`schemas.py:370-385\`; calculated at \`repositories.py:1914-1916, 2019\`). Source-funder detail returns up to 50 individual grant samples with that publisher URL (\`repositories.py:2266-2305\`). +- The 200,000 publisher URLs have only **309 distinct values**, so they are often publisher/API/open-data URLs shared by many awards. They must not be labelled “recipient website” or “individual grant page.” +- 361 of 364 enriched profile rows have a profile \`source_url\`; those rows also carry an optional \`website\` field in \`charities\`. Profile evidence is separate from observed grant evidence. +- The current header **Data sources** disclosure exposes source-selection buttons only (\`frontend/src/App.tsx:1539-1561\`). Main Donors does not render any publisher/evidence link, and the profile grant table does not render an outbound organisation-record link (\`frontend/src/components/SourceFunderResults.tsx:12-45, 430-452\`; \`App.tsx:2666-2726\`). + +The future detail should add a final, collapsible **Source evidence** section, not a generic external-link icon scattered across every list row: + +| Evidence item | Recommended user-facing label | When to show | Required safety / provenance rule | +|---|---|---|---| +| \`recipients[].self\` / \`funders[].self\` | “View 360Giving recipient record” / “View 360Giving funder record” | Individual observed grant relationship | Extract and expose the raw value explicitly; open in a new tab with \`noopener noreferrer\`; it is an organisation-data JSON link | +| \`recipientOrganization[].url\` / \`fundingOrganization[].url\` | “Visit recipient website” / “Visit funder website” | Individual observed grant relationship, when this raw field exists | State “Website supplied in this grant record”; it is publisher-provided and should not be silently merged with a profile website | +| \`data.dataSource\` / current \`source_url\` | “View publisher’s grant data” | Individual grant or grouped funder detail | State that it can cover multiple awards and may be a publisher/API page | +| Resolved profile \`website\` | “Visit organisation website” | Only if a direct/accepted linked profile supplies it | Label its provenance, for example “Website from Charity Commission profile”; never derive it from name alone | +| \`source_record_id\` | “Source record ID” | Copyable detail/methodology field | Preserve source namespace and ID; no name-only reconstruction | +| Profile \`source_url\` / \`source_records\` | “Profile data source” | Linked-profile evidence section | Keep visibly separate from observed grant evidence | +| Data-source selector | “Included sources” | Header / active-filter chips | Continue to filter scope; it is not a proof link | + +No new crawler or live 360Giving fetch is required. An endpoint extension is required to expose the recipient/funder \`self\` URLs and publisher-provided recipient/funder website values without making the browser parse arbitrary raw JSON. A later ingestion enhancement can normalise these into \`funder_source_record_url\`, \`recipient_source_record_url\`, \`funder_website_observed\` and \`recipient_website_observed\`, with a validated \`link_kind\` (360Giving organisation JSON, observed organisation website, publisher data source, profile website, unknown). + +## 6. Current Organization Profiles View + +### Purpose and population + +The profile directory answers “which enriched organisations are in our directory?” It uses \`GET /api/charities\` at \`src/bff/charity.py:30-72\`, whose repository implementation is \`get_all\` at \`src/bff/repositories.py:3220-3419\`. Rows are \`charities\` records, not source-funder identities. + +Current population: + +- 364 profile records; +- 299 sourced from Philea and 65 from the Charity Commission; +- 299 are currently organisation-level only and 65 are classified as unknown coverage; +- only 17 profiles have any direct matched grant on either donor/recipient side. + +Therefore a profile may be valuable background research but must not be presented as an observed donor merely because it has a name, headquarters, or financial data. + +### Filters and their technical meaning + +The profile list uses a 250 ms debounced search and 50-row pagination (\`App.tsx:585-606\`, \`801-894\`). It sends \`search\`, \`reg_status\`, \`tag\`, \`region\`, \`size\`, \`tags\`, \`foundation_regions\`, \`funding_regions\`, \`sources\`, \`min_annual_giving\`, \`min_avg_grant_size\`, \`skip\`, and \`limit\`. + +The left sidebar fields in \`App.tsx:1863-2071\` are always visible: name search/suggestions, thematic sector, foundation location, beneficiary geography, annual giving, average grant and reset. “Beneficiary Geography” is technically ambiguous here: its backend criterion is grant rows for which the linked directory organisation is the **funder** (\`funding_charity_id\`), followed by parsed beneficiary locations; it is not generic geographic focus nor recipient activity. This causes empty results even when a similarly named source funder appears on the map. + +### Visible fields and information density + +| Default field | Meaning / source | Classification | Recommendation | +|---|---|---|---| +| Profile ID | Philea source record / charity identifier | Technical | Move to details, except official charity number in registry mode | +| Organisation name | \`charities\` profile name | Primary decision information | Keep | +| Source badge | Profile source | Provenance | Keep one compact label | +| Organisation type | Profile metadata | Supporting | Keep only where reliably populated; otherwise details | +| HQ location | Profile headquarters | Supporting; not beneficiary geography | Keep only if relevant to comparison | +| Up to two programme tags | Source or inferred classification | Supporting / provenance | Keep one primary tag, show source/inferred basis in details | +| “Review suggested” / organisation-level-only | Coverage status | Data quality | Keep as an accessible text status, not as a competing primary badge | +| Latest income | Profile financial field | Supporting | Keep where present; label source/year and move other financial values to detail | +| “Select Details” | Card action | Primary action | Keep but use a semantic button/card pattern | + +The rows can duplicate the same real organisation across Philea, Charity Commission and source-funder identity representations. That is acceptable only when the interface makes the record type and relationship explicit. + +### Interaction issues + +Typing suggestions are fetched with \`?search=&limit=6&sources=\`; see \`App.tsx:632-673\`. The suggestion UI has listbox/options but lacks full combobox state/keyboard behaviour. Profile cards are clickable \`div\` elements without keyboard semantics (\`App.tsx:2082-2117\`). Selecting one launches four requests in parallel: profile, grants, Sankey data and scoring (\`App.tsx:675-692\`). This produces detail work even before a user requests the deeper sections. + +## 7. Current Charity Commission Register View + +### Purpose and entity + +The registry is a large, official-record lookup. It uses \`RegistryDirectory\` and \`GET /api/charities/directory/organizations\` in \`src/bff/charity.py:87-146\`. Its table is \`charity_registry_organizations\`, not \`charities\` and not observed grant funders. The current registry contains 397,469 source-current records and matching FTS records. + +The query implementation at \`src/bff/repositories.py:2918-3137\` uses FTS5/prefix search where applicable, \`is_current_source_record\`, finance/status fields and stable cursor pagination. It is the correct advanced utility for questions such as “does the Commission register contain this charity?” rather than “who funds this beneficiary country?” + +### UI, filters and linkage + +\`frontend/src/components/RegistryDirectory.tsx:81-168\` maintains its own states and fetches 50 records at a time with a 300 ms search debounce. Its sidebar has eleven persistent controls (\`RegistryDirectory.tsx:217-270\`): organisation name, exact number, status, income/expenditure ranges, country/region, observed beneficiary geography, profile layer, observed grants, and sort. It has cursor “Load 50” pagination. + +Cards show charity number, registered name, status, enriched/observed/Philea badges, city/region, and GBP income (\`RegistryDirectory.tsx:273-297\`). Its modal displays registry attributes, activities, observed-data message, and an accepted enriched-profile link when present (\`RegistryDirectory.tsx:302-321\`). + +The accepted registry link table is \`organization_registry_links\` (\`src/data/registry.py:115-128\`). Current data has 343 accepted links, all exact-identifier method and confidence 1.0; 64 distinct enriched profiles are linked, while 9 profiles link to multiple registry records. Only 121 accepted registry links have any direct observed grant existence. A registry record must never inherit observed-donor status merely because it has a profile link. + +| Default field / feature | Classification | Recommendation | +|---|---|---| +| Registered name and charity number | Primary for registry search | Keep in advanced registry mode | +| Registration status and registered office | Supporting official information | Keep compactly | +| Income/expenditure and activities | Supporting / research | Keep in record detail; summary only when a user sorts by it | +| Enriched/observed/profile badges | Linkage / provenance | Keep as text-led evidence, do not imply equivalence | +| Registry-specific range/status filters | Advanced search | Keep in a drawer/advanced mode | +| Beneficiary geography filter | Data relationship feature | Keep only with explicit label: “observed grants linked to an accepted profile” | + +## 8. Current Donor/Organization Detail Behaviour + +There are three substantially different detail experiences: + +1. **Profile modal** — \`App.tsx:2365-2731\`: contact/address, website, program/geography metadata, profile classification and coverage, experimental relevance score, ranked funding relationships, financial totals/trends, lazy AI news, and observed grant table. +2. **Observed source-funder modal** — \`App.tsx:2739-2842\`: selected source identity, aggregate grant details, donor-to-recipient Sankey, currency/multi-country notes and source data. +3. **Registry dialog** — \`RegistryDirectory.tsx:302-321\`: official registry record and optional accepted enriched-profile link. + +Oxfam is a useful concrete test case. Its directory profile has 40 direct recipient-linked grant records. The repository returns Sankey/relationship data as available with 9 nodes and 8 links. A prior apparent “empty” or bad funding-flow view is explained by the visualisation/label layout, rather than absence of the 40 stored relationships. + +The profile flow data uses only direct \`funding_charity_id = ? OR recipient_charity_id = ?\` matching in \`get_grants_for_charity\` and \`get_sankey_data\` (\`repositories.py:4372-4478\`, \`4480+\`). This is deliberately narrower than donor-name matching and should remain so until stronger linkage evidence exists. + +Common information hierarchy for a future shared detail is feasible: + +| Section | Observed-only source funder | Direct linked profile | Registry record | +|---|---|---|---| +| Identity / observed status | Always | Always, plus link evidence | Only when opened in advanced registry mode | +| Grant activity / recipients / amounts | Always within the selected grant scope | When direct grant links exist | Only explicitly linked observed evidence | +| Profile contact/HQ/website/programmes | Never fabricate | Show with source and coverage | Registry fields, not profile fields | +| Registry registration data | Never fabricate | Show only through accepted registry link | Native content | +| Methodology and match evidence | Available last | Available last | Available last | + +The existing profile and source modals should not be forced into one renderer without a discriminated result type. A common layout can share shell, heading, amount, tabs/sections and loading behaviour, but fields must remain conditional on entity/evidence type. + +## 9. Data Model and Entity Relationships + +### Entity diagram + + grants (grant_id) + ├─ funding_name + funding_org_source_id ──> source-funder identity (query-time, 1 grant : 1 identity) + ├─ funding_charity_id ──> charities.charity_id (optional direct join; not universally valid) + ├─ recipient_name + recipient_org_source_id ──> source-recipient identity (query-time) + ├─ recipient_charity_id ──> charities.charity_id (optional direct join; not universally valid) + ├─ 0..n grant_beneficiary_terms / grant_beneficiary_countries + ├─ 0..n grant_programme_categories + └─ original amount/currency/date + optional amount_eur/exchange-rate basis + + charities (charity_id, enriched directory profile) + └─ 0..n organization_registry_links ──> charity_registry_organizations + + charity_registry_organizations (registry_id, official current record) + └─ 0..n organization_registry_links (match status/method/confidence/review metadata) + + profile enrichment / financial metadata / news + └─ belongs to charities, and must not be assumed for a source-funder identity + +### Relationships and evidence + +| Relationship | Cardinality / join | Mandatory? | Evidence level / duplicate risk | +|---|---|---|---| +| Grant → source funder | 1:1 query-time identity from source ID or normalised name | Conceptually required from grant donor fields; ID may be absent | Observed source data; name fallback can split/merge | +| Grant → directory profile (funder/recipient) | 0..1 stored ID per side; join to \`charities\` | Optional | Direct join only when ID resolves; current database has unresolved values | +| Source funder → directory profile | 0..n derived through direct grant joins | Optional | Endpoint exposes an actionable profile only if exactly one ID; no persistent source-funder-link table | +| Directory profile → registry record | 0..n through \`organization_registry_links\` | Optional | Current importer accepts exact identifier links; match metadata retained | +| Grant → beneficiary country | 0..n normalised derived rows / JSON fallback | Optional | Beneficiary location, not donor HQ | +| Grant → programme area | 0..n derived category rows | Optional but currently populated for all grants | Source/inferred classification must remain traceable | +| Grant → EUR amount | 0..1 stored historical conversion | Optional | 199,579 / 200,000 currently have valid conversion | + +### Integrity findings + +The present database has 2,294 foreign-key-check violations relating to source entity IDs stored in \`funding_charity_id\` / \`recipient_charity_id\`: 464 funding-side rows across 65 distinct unresolved IDs, and 1,830 recipient-side rows across 1,260 unresolved IDs. A non-null value in either field is therefore **not by itself proof of a valid directory link**. Current joined code is safer than naïvely using non-null IDs, but any future unification must use successful joins plus evidence metadata. + +Current direct join coverage is: + +- 2,440 grants have non-null funding IDs; 1,976 join current profile rows; +- 2,907 grants have non-null recipient IDs; 1,077 join current profile rows; +- only 17 profile rows participate in at least one direct observed grant link; +- three profiles occur as direct funders and fifteen as direct recipients (one or more can do both). + +The conceptual chain requested by the product is supported only as: + + grant → source funder [observed] + → optional directory profile [derived direct ID join] + → optional registry record [accepted exact-identifier link] + +It is not a universal mandatory chain and cannot be made one by normalized-name matching without misleading users. + +## 10. End-to-End Data Flow + +### Funding Landscape + +1. \`OverviewDashboard\` reads grant filters from URL in \`frontend/src/components/OverviewDashboard.tsx:145-160\`. +2. It requests the aggregate overview in \`OverviewDashboard.tsx:223-257\`. +3. The repository uses cache-backed overview data; country/programme scoped source rows are then processed to build map/KPIs/trends. +4. \`GrantWorldMap\` renders beneficiary-country choropleth geometry. Map selection is ISO country selection, and is not a donor-headquarters selector. Keyboard/click selection can scroll to the selected-country content (\`GrantWorldMap.tsx:353-360\`). +5. “Explore active funders” calls \`openSourceFundersFromMap\` at \`App.tsx:1047-1069\`. + +### Map-to-donor hand-off + +The hand-off writes \`funder_country\` and \`funder_sort\`, removes \`funder_page\`, then carries \`grant_currency\`, \`grant_from\`, \`grant_to\`, \`grant_geo\`, \`grant_programme\`, \`grant_donor\`, \`grant_recipient\`, \`grant_granularity\`, and \`grant_sources\`. This is a strong basis for preserving country/grant scope. On return, \`openSourceFundersFromMap\` only removes funder country/sort/page (\`App.tsx:1072-1080\`) so most grant URL state remains. + +**Gap:** \`OverviewDashboard\` serialises its map query keys in \`OverviewDashboard.tsx:187-200\`, but it does not reconstruct the global source selection from \`grant_sources\` on reload/back. The donor page receives source scope initially; the overview can later show an unselected global source control despite the query string. This is a state-contract inconsistency. + +### Directory profile flow + +The profile list calls \`/api/charities\`; open profile fires the four detail calls described in section 6. Profile grant details and source-funder detail intentionally take different back-end paths. Do not silently substitute one for the other. + +### Registry flow + +Registry search uses its own cursor/query state and calls the registry endpoint. It can open a linked enriched profile, which preserves registry mode because the component remains mounted. The registry does not inherit map grant filters, which is semantically safer than applying them as if registry rows were donor results. + +## 11. Filter and URL-State Audit + +| Filter / state | UI and frontend state | URL / API | Backend effect and meaning | Navigation survival / issues | +|---|---|---|---|---| +| Beneficiary country | Map selection / Main Donors country control | \`funder_country\`; \`beneficiary_country\` API | Normalised beneficiary-country scope | Transfers map → Main Donors; profile label means something narrower | +| Beneficiary geography | Map multi-scope / donor advanced control / profile and registry controls | \`grant_geo\`; \`beneficiary_geographies\` | Grant beneficiary geography; profile version only linked profile-as-funder grants; registry version only accepted-link observed grants | Same label masks three different semantics | +| Programme area | Overview / donor control | \`grant_programme\`; \`programme_areas\` | Derived programme categories | Transfers map → donor | +| Time range | Overview / donor control | \`grant_from\`, \`grant_to\`; date params | Grant award-date scope | Transfers map → donor | +| Currency | Overview / donor control | \`grant_currency\`; \`currency\` | Auto EUR conversion or original-currency filter | Transfers map → donor | +| Donor text | Overview / donor control | \`grant_donor\`; \`donor\` | Grant donor text scope | Transfers map → donor | +| Recipient text | Overview / donor control | \`grant_recipient\`; \`recipient\` | Grant recipient text scope | Transfers map → donor | +| Source | Header global selector / donor advanced control | \`grant_sources\`; \`sources\` | Source scope | Donor retains it; Overview fails to rehydrate global selection | +| Granularity | Overview | \`grant_granularity\` | Trend/map aggregation UI state | Not relevant to donor identity list | +| Main-donor sort/page | Main Donors | \`funder_sort\`, \`funder_page\` | Backend sort/cursor-like page | Preserved in URL | +| Profile name/status/tags/HQ/amount filters | Profile sidebar | API params listed in section 6; no equivalent canonical URL contract | Profile table filters | State is component-local; not shareable/back-restorable | +| Registry number/status/finance/country/region/links/sort/cursor | Registry sidebar | Registry request params in section 7; component-local | Official record lookup | Separate correctly; not map scope | + +The current world-map-to-donor contract is the safest existing foundation. A future route design should first introduce a **canonical serialisable grant-scope object** (including sources) and test round trips, rather than immediately renaming every path. Once a conventional router is introduced, equivalent paths such as \`/funding-landscape?... \` and \`/donors?... \` can use that same contract. Current code does not yet establish those paths as a stable public convention. + +Filters most in need of restructuring are the permanently visible profile/registry sidebars, which expose many advanced controls before the user has selected a research task. Move them into an explicit drawer while retaining every current server parameter and clear active-filter chips. + +## 12. Visual Hierarchy and Information-Density Audit + +### General + +The UI frequently gives decision data, provenance, technical metadata, and administrative filters the same visual weight. This is clearest in the profile-card tag stack and permanent filter rail. It makes the answer (“who should I investigate?”) compete with source badges, types, HQ, inferred classifications, review states, and multiple monetary fields. + +### Main Donors + +The row hierarchy is closest to a practical donor list because observed funding and recipients are visible. The six KPI tiles plus eight control fields consume significant vertical space before results. The best default comparison set is: source donor name, observed/linked state, grant count, recipient count, country-attributable amount, and one leading programme. Identity method, all programme tags, multi-country calculation detail, and raw source metadata belong behind details/methodology. + +### Profiles + +The profile grid uses visually large cards with multiple small tags and often unavailable or weakly comparable fields. The default card shows source, type, HQ and topical tags at similar emphasis to name/income. Technical IDs and coverage/review labels should not outrank a clear answer about record type and evidence. + +### Registry + +The register needs more factual fields than donor discovery, but those are appropriate in a separate advanced research mode. Its eleven visible filters form a heavy permanent administrative panel. The default should lead with name/number, current registration status and location; financial ranges and data-link controls can be advanced. + +### Funding-flow visual + +The observed funding-flow canvas uses a pale beige field, wide empty canvas and multiple left labels whose available width is too small. Curves converge at one destination, producing an imprecise funnel rather than a readable comparison. This is especially poor in the supplied Oxfam screenshot. It needs a ranked relationship list/table as the default, with a compact Sankey only when there are few balanced links and labels can be placed without clipping. + +## 13. Responsive Behaviour + +This section is code-inferred because a rendered browser was not available. Validate it manually at the following exact sizes before shipping a redesign. + +| Target viewport | Inferred current behaviour | Risk / required validation | +|---|---|---| +| 1440 × 900 | Full 260 px sidebar; desktop map uses \`clamp(320px, 100svh - 410px, 580px)\`, about 490 px map height; source donor summary can use six columns | Strong desktop information density, but map/chart and large cards consume initial viewport | +| 1280 × 800 | At exactly 1280, desktop sidebar rule remains rather than the \`<=1279px\` compact treatment; map tends around 390 px | Abrupt breakpoint discontinuity one pixel below; check title/header and four KPI cards | +| 1024 × 768 | Sidebar has compact behaviour at max 1279; KPI transition begins at \`<=1023px\`, so exact 1024 still attempts four KPI columns; map minimum about 380 px | Likely cramped KPI row and a brittle 1023/1024 transition | +| 834 × 1194 | Portrait/tablet drawer rules are present; KPI grid is two columns; map clamp about 400 px | Validate drawer focus, scroll locking and map touch targets | +| 390 × 844 | Mobile drawer; two KPI columns; map clamp about 304 px; donor table switches to cards only at \`<=680px\` | Validate tab wrapping, source-funder modal and full-screen detail treatment | + +Relevant responsive rules are concentrated in \`frontend/src/index.css:868-888\` and later override rules at approximately \`3007-3069\`. The CSS contains overlapping historical and new media queries, which makes changes difficult to reason about. The source-funder result table has a 1040 px desktop minimum and becomes cards only at 680 px; between 681 and 1050 px it may horizontally scroll. Its KPI grid changes from six to two to one columns at 1050/680. Registry has more deliberate one-column/reduced-detail rules at 900/520, while profile and source-funder modal inline styles have less intentional mobile restructuring. + +The future detail model is feasible as: + +- **Desktop:** list plus an accessible right-side detail panel, with wide table/chart only in the panel’s dedicated section. +- **Tablet:** list plus drawer/full-width sheet; do not retain the desktop three-column information density. +- **Mobile:** list then full-screen detail route/sheet; do not attempt desktop modal-within-list geometry. + +The overview filter drawer already has useful focus trapping and Escape behaviour. The future donor filter drawer can reuse that interaction pattern; the existing profile/sidebar implementation cannot simply be shrunk. + +## 14. Performance Findings + +### Measured repository timings + +Measurements ran against the current local SQLite database on 24 July 2026 and include JSON serialisation; they do not include HTTP or browser rendering. + +| Operation | Result | Interpretation | +|---|---:|---| +| Overview, cold, all sources | 3.281 s, 88,746 bytes | Acceptable only with loading feedback/cache; too slow for repeated interactive re-scoping | +| Overview, hot cache | 0.004 s, 88,746 bytes | Cache works very well | +| Yearly trends, cold | 6.737 s, 5,929 bytes | Small result but expensive computation; it should be cached or pre-aggregated | +| Profile page (51 rows) | 0.002 s, 52,006 bytes | List paging itself is fast | +| Registry page (50 rows) | 0.013 s, 18,078 bytes | Cursor/FTS approach is effective | +| Registry search “oxfam” | 0.006 s, 1,449 bytes | Effective | +| Main Donors, US | 3.369 s, 22,669 bytes, 22 results | Work remains high despite a smaller country scope | +| Main Donors, GB | 13.734 s, 26,579 bytes, 25 results | Current primary donor journey is too slow | + +The database is approximately 1.33 GB. It contains 200,000 360Giving grants, 71,286 mapped beneficiary-country grant rows (35.64%), all 200,000 categorised by programme, 199,579 valid EUR converted amounts, 421 grants without a valid conversion, and 37 overview-cache entries. + +### Main bottlenecks + +| Priority | Layer | Finding | Evidence / impact | +|---|---|---|---| +| P0 | Backend/query/CPU | \`_source_funder_scope\` obtains scoped grant rows then parses JSON and groups all candidate rows in Python | \`repositories.py:1716-1979\`; GB scans a large mapped scope and costs 13.7 s | +| P0 | Backend/API | Source-funder detail recomputes full source scope instead of retaining/caching grouped work | \`repositories.py:2023-2151\`; detail adds repeat latency | +| P1 | Backend/query | Cold overview/trend aggregation is expensive despite a small returned payload | Cache helps overview; trends need a compatible cache/materialised aggregation | +| P1 | Frontend/API | Opening one profile starts four requests, including score and grant/relationship data before requested | \`App.tsx:675-692\`; lazy-load secondary tabs | +| P1 | Frontend bundle | One 607.87 kB gzip initial JS chunk includes map/chart dependencies | Build output; defer directory/registry/Sankey/chart code | +| P2 | Backend/query | Profile \`funding_regions\` filtering can fetch and JSON-parse every non-null funder-linked grant; broad \`LIKE %query%\` search is also expensive at scale | \`repositories.py:3220-3419\` | +| P2 | Frontend rendering | Sankey attempts many labels/paths in a wide canvas and has poor readable density | Current source detail renderer \`App.tsx:2739-2842\` | + +No evidence of an N+1 pattern was found in the paged registry or initial profile-list request. The greater risk is broad Python post-processing and repeated aggregate queries. Use staged SQL aggregation/derived table/cache before attempting client-side optimisation. + +## 15. Accessibility Findings + +### Strengths + +- Overview filter drawer has dialog semantics, focus trapping, Escape support and focus restoration: \`OverviewDashboard.tsx:310-368\`. +- Interactive map countries expose button role, keyboard Enter/Space operation, tab stops and country-specific accessible labels: \`GrantWorldMap.tsx:448-513\`. +- Map loading has live/progress communication; map legend/help includes textual explanation in addition to colours. +- Main Donors desktop table has table headers, and mobile cards expose data labels. +- Registry record dialog uses \`role="dialog"\` and \`aria-modal\`; form labels are generally present. + +### Gaps + +| Priority | Issue | Evidence / remedy | +|---|---|---| +| P0 | Profile cards are clickable non-semantic divs | \`App.tsx:2082-2117\`; use a button/link or complete keyboard handling | +| P0 | Profile and observed-source modals lack dialog role, focus trap, Escape/return-focus lifecycle | \`App.tsx:2365+\`, \`2739+\`; use the overview drawer pattern | +| P1 | Registry dialog has role but not demonstrated focus trap, Escape or focus restoration | \`RegistryDirectory.tsx:302-321\` | +| P1 | Tabs lack complete \`aria-controls\` / \`tabpanel\` relationship and arrow-key tab behaviour | Directory tab implementation around \`App.tsx:1790-1830\` | +| P1 | Search suggestions lack complete combobox semantics and keyboard active-descendant selection | \`App.tsx:632-673\` | +| P1 | Sankey labels clip, which is both visual and assistive-information loss | Use an accessible ranked table and make visual chart optional | +| P2 | \`.world-map-country:focus\` removes standard outline and relies on SVG stroke visual | \`frontend/src/index.css\`; validate strong visible focus at high contrast | + +Verified/linked/source-only states must always include text, not just purple/grey/badge colour. Automated contrast, screen-reader, keyboard, touch-target and responsive tests remain required because no browser/AT tooling was available here. + +## 16. Data-Preservation Matrix + +“Move” means relocate in the interface; it does **not** authorise deletion from API or database. + +| Current page | Current field / feature | Current data source | Component / API | Future location | Default? | Details? | Advanced? | Retained? | Required change | Risk | +|---|---|---|---|---|---|---|---|---|---|---| +| Main Donors | Source funder name / ID method | Grant donor fields; query identity | \`SourceFunderResults\` / \`/grants/funders\` | Donor Directory row | Yes | Yes | Yes | Yes | Shared discriminated donor result | High | +| Main Donors | Source provenance | \`grants.source\` | Same | Row status + methodology | Compact | Yes | Yes | Yes | Preserve source / identity method | High | +| Main Donors | 360Giving recipient/funder JSON link, observed recipient/funder website, publisher URL, source record ID | \`raw_grant_data.recipients[].self\`, \`funders[].self\`, \`recipientOrganization[].url\`, \`fundingOrganization[].url\`, \`grants.source_url\`, \`source_record_id\` | Grant/funder APIs expose publisher URL/ID; 360Giving JSON/observed websites require a typed response extension | Collapsible Source evidence; individual relationship/transaction actions | No | Yes | Yes | Yes | Extract typed values server-side; distinguish observed organisation website, publisher data, 360Giving JSON and profile website links | High | +| Main Donors | Grant count / recipients / latest date | Aggregated grants | Same | Donor row/detail | Yes | Yes | Yes | Yes | Reuse endpoint or aggregate layer | Low | +| Main Donors | Country-attributable funding | EUR/original grant amounts, single-country rule | Same | Donor row/detail | Yes | Yes | Yes | Yes | Preserve multi-country policy and label | High | +| Main Donors | Original currency / EUR conversion | \`grants\`, \`exchange_rates\` | Same | Details/methodology; selected display | No | Yes | Yes | Yes | Canonical currency/scope contract | High | +| Main Donors | Leading programmes | Derived programme categories | Same | One row tag / all details | One | Yes | Yes | Yes | Keep classification provenance | Medium | +| Main Donors | Source-only / linked profile state | Direct joined profile IDs | Same | Row status | Yes | Yes | Yes | Yes | Do not equate null/non-null ID with link | High | +| Main Donors | Recipient relationship / Sankey | Stored grant transactions | Source detail API | Details, ranked default | No | Yes | No | Yes | Replace fragile chart default, retain data | Medium | +| Main Donors | Country/time/program/source/donor/recipient filters | URL + API scope | \`SourceFunderResults\` | Compact chips + drawer | Chips | N/A | Yes | Yes | Canonical URL object | High | +| Profiles | Profile name | \`charities\` | \`App\` / \`/charities\` | Search result / linked donor enrichment | Conditional | Yes | Yes | Yes | Separate from source-funder row | High | +| Profiles | Profile source/type/HQ/tags/coverage | Enriched profile metadata | \`App\` | Linked-profile section | Minimal | Yes | Yes | Yes | Move secondary tags from row | Medium | +| Profiles | Website/email/phone/address | Enriched profile metadata | Profile detail endpoint | Linked-profile detail | No | Yes | No | Yes | Conditional evidence-only rendering | High | +| Profiles | Latest income/expenditure / trends | Profile financial fields | Profile detail | Financial detail | Optional | Yes | Sort/filter | Yes | Label currency/year/source | Medium | +| Profiles | Relevance score | Score endpoint | \`POST /{id}/score\` | Secondary detail | No | Yes | No | Yes | Lazy load; clarify experimental status | Medium | +| Profiles | AI news | News enrichment | Profile news endpoint | Secondary detail | No | Yes | No | Yes | On-demand loading/progress/date/source | Low | +| Profiles | Profile grant table / relationships | Direct profile-linked grants | \`/{id}/grants\`, \`/{id}/sankey\` | Details | No | Yes | No | Yes | Preserve direct-link scope distinction | High | +| Profiles | Profile search and all filters | Profile query params | \`/api/charities\` | Separate profile/advanced directory filter set | No | N/A | Yes | Yes | Keep out of observed donor default unless status is explicit | High | +| Registry | Registered name / charity number | \`charity_registry_organizations\` | \`RegistryDirectory\` / registry API | Advanced Registry Search | Yes in registry | Yes | Yes | Yes | Retain route/mode initially | High | +| Registry | Registration status / address / activities | Official record | Registry detail | Registry detail | Compact | Yes | Yes | Yes | Registry-specific layout | Low | +| Registry | Income/expenditure / sorting | Official registry finance | Registry list/API | Advanced filters/sort | No | Yes | Yes | Yes | Drawer/filter preservation | Low | +| Registry | Accepted enriched link / method / confidence | \`organization_registry_links\` | Registry API/detail | Evidence section | Status | Yes | Yes | Yes | Never convert registry row into donor automatically | High | +| Registry | Observed grant indicator/geography | Direct accepted link then grants | Registry filter | Advanced registry evidence | No | Yes | Yes | Yes | Explicit semantics in label | High | + +## 17. Feasibility of the Two-Screen Structure + +The proposed structure is technically and semantically feasible **with a staged integration layer**, not a merge of current tables. + +### Screen 1 — Funding Landscape + +Retain \`OverviewDashboard\` and \`GrantWorldMap\`. The map already uses beneficiary geography and can hand a canonical grant scope to observed funders. Keep country, programme, time and compact aggregate summary. The primary action should remain “Explore observed funders” or a similarly precise term; calling them merely “organisations” obscures the entity distinction. + +### Screen 2 — Donor Directory + +Make the current Main Donors source-funder aggregation the default list. It should contain a discriminated item such as: + + kind: source_funder + identity: source + source-id-or-normalised-name + observed_metrics: counts, recipient count, currency-aware totals, programmes + profile_link: zero / one / many-without-auto-selection, with evidence + registry_link: absent or separately traceable accepted link + +Suggested statuses are: + +- **All observed funders** — all source-funder identities in map scope; +- **Linked profiles** — only rows with a successful, explicitly described profile linkage; +- **Observed only** — source-funder identities without that linkage. + +Avoid “Verified profile” until product/data owners confirm whether today’s direct Funder-ID join meets their definition of verification. The current record has good evidence of a direct database relationship but does not store a separately reviewed source-funder-profile match record. + +### Details + +Use one detail shell but render discriminated sections. Observed transactions remain source-data facts. Enriched profile information appears only under a linked-profile evidence section; it must never fill an observed-only donor with inferred website/HQ/type. The registry remains a linked official-record section or separate advanced record dialog. + +The shared shell should end with a collapsed **Source evidence** section. Reuse the existing grant-level publisher \`source_url\`, \`source_record_id\`, raw recipient/funder website values, profile \`source_url\`/website and group-level \`representative_source_url\`; expose raw 360Giving recipient/funder \`self\` links and observed website values through a typed server response rather than parsing raw JSON in the browser. Keep publisher data, 360Giving organisation JSON, website supplied in an observed grant, and profile website links visibly separate. + +### Registry + +Keep \`RegistryDirectory\` as an advanced registry-search mode. Do not apply map filters to it by default. A user can explicitly filter “registry records with observed grant evidence through accepted linked profiles,” but that label must remain precise. + +### Routes and redirects + +No stable individual routes currently exist beyond \`/\` and query state, so no current route should be removed or redirected in the first migration. Retain the existing modes while a conventional route contract is introduced and parity-tested. Only after the new Donor Directory reaches data/interaction parity should legacy tabs redirect to it. The future exact paths remain a product decision; the existing query names provide a compatible starting point. + +## 18. Reusable Components + +| Component / capability | Reuse assessment | Conditions | +|---|---|---| +| \`OverviewDashboard\` | Reuse | Extract/standardise URL grant-scope parsing, including sources | +| \`GrantWorldMap\` | Reuse | Preserve beneficiary-country semantics; leave connection arrows explicitly illustrative | +| \`SourceFunderResults\` | Reuse as donor-result data/row foundation | Reduce KPI/control density, improve state and detail renderer | +| Source-funder API and identity grouping | Reuse conceptually | Optimise query/cache before making it default for heavy countries | +| Registry cursor/FTS API | Reuse | Keep as advanced registry utility | +| \`RegistryDirectory\` record detail | Reuse | Add focus lifecycle and simplify default filter exposure | +| Overview filter drawer | Reuse interaction pattern | Adopt semantics/focus handling for donor filters | +| Profile financial/news/grant sections | Reuse conditionally | Lazy load and render only for actual linked profile | +| Existing relationship data | Reuse | Prefer ranked table/list first; only show Sankey under readable thresholds | + +## 19. Components Requiring Refactoring + +| Area | Required refactor | +|---|---| +| \`App.tsx\` | Split page composition, directory mode coordination and all three modals into focused components; retain behaviour during transition | +| Routing/state | Introduce one typed parse/serialise grant-scope contract; route/back tests before path changes | +| Profile cards | Semantic controls, less tag density, explicit profile/coverage state | +| Source funder detail | URL-addressable selection or controlled sheet state, accessible dialog lifecycle, ranked relationships | +| Directory filters | Convert permanent desktop sidebars to active chips plus advanced drawer; do not remove any API filter | +| Modal system | Shared accessible sheet/modal primitive with role, focus trap, Escape, return focus and responsive mode | +| CSS | Consolidate overlapping responsive media rules into component-owned breakpoints/tokens after visual parity is protected | +| Charts | Code-split Recharts/Sankey and use a data-table fallback | + +## 20. API and Query Changes Required + +### APIs that can remain initially + +- \`GET /api/charities/grants/overview\` +- \`GET /api/charities/grants/overview/trends\` +- \`GET /api/charities/grants/funders\` +- \`GET /api/charities/grants/funders/{source_funder_key}\` +- \`GET /api/charities\` and profile-detail endpoints +- Registry page/detail endpoints + +### Changes required for a fast, safe unified experience + +1. Define a typed donor-result response or versioned extension that explicitly includes \`record_type\`, identity method, source provenance, observed metrics, and **zero/one/many** profile-link evidence. Do not flatten profile fields onto source-only rows. +2. Push source-funder aggregation down to indexed SQL/derived aggregation where feasible, or precompute/cache by canonical scope. Avoid parsing all GB-country JSON rows on each request. +3. Cache or materialise cold trend aggregates by the same filter dimensions that the overview supports. Invalidate based on data-load/version, not a vague timeout alone. +4. Let source-funder detail query one source identity within the already-scoped aggregate rather than recomputing every result row. Add a small detail cache keyed by scope and source-funder key if measurements justify it. +5. Return explicit amount-policy metadata: automatic EUR conversion coverage, selected original-currency scope, and excluded multi-country amount. This avoids a UI having to infer totals. +6. Bound/lazy-load profile grant tables, relationships, scores and news. The four initial profile requests should become summary first, then user-requested sections. +7. Preserve registry endpoint independence. If an endpoint offers registry observed evidence, return the link method/status/confidence so it cannot be mistaken for a direct donor record. +8. Existing funder list/detail and profile-grant responses already expose publisher \`source_url\` values and source record IDs. Add a small typed response extension for raw 360Giving \`recipients[].self\` / \`funders[].self\` links and \`recipientOrganization[].url\` / \`fundingOrganization[].url\` values; the browser must not parse arbitrary raw JSON. A later structured \`link_kind\`/validation field would prevent publisher data, 360Giving JSON, an observed organisation website and a profile website being confused. + +No database migration is required merely to introduce UI changes, but a durable source-funder identity/link table or materialised aggregation may be justified for performance and auditable linkage. It must include source, source identifier/name fallback, data-load version and evidence—not only a normalised display name. + +## 21. Risks and Possible Regressions + +### Data integrity / meaning + +1. **False donor profiles:** merging source-funder rows with enrichment by name or non-null foreign key can show profile data for an entity that is not proven to be the observed funder. +2. **Registry misrepresentation:** treating a Charity Commission row or an accepted profile link as observed-donor evidence changes the meaning of the official register. +3. **Geography confusion:** donor HQ, registered office, stated geographic focus and grant beneficiary country are different attributes. The current profile “Beneficiary Geography” label is already ambiguous. +4. **Currency-total drift:** changing automatic-EUR, selected-original-currency or multi-country treatment without a shared policy breaks comparability. +5. **Identity collapse/duplication:** normalised-name fallback can split or merge source funders; multiple directory/registry profiles cannot be silently resolved for UI convenience. + +### Product and functional regressions + +- A new donor list that drops source-only rows would hide the majority of observed grant evidence. +- A new profile-only list would lose recipient counts, country-attributable grant totals and map continuity. +- Moving filters into drawers can lose advanced research capability unless every currently supported server parameter is preserved and tested. +- Replacing modals with a panel/page can break list/filter back state unless the selection and filters are URL-backed. +- Retiring registry navigation before parity would remove the only complete official-register search. + +### Performance regressions + +- Making slow GB source-funder aggregation the default landing query without backend optimisation will produce a visibly slow primary product flow. +- Querying all profile detail sections on every selection compounds API/render work. +- Loading map/chart libraries into the initial bundle slows first interaction, especially mobile. + +## 22. Open Product Decisions + +These decisions should be made before implementation, because each changes wording, states and test assertions: + +1. Does “verified profile” mean a successful direct \`charities\` join, a manually reviewed funder-profile linkage, or only a registry exact-identifier record? Recommended current label: **Linked profile**. +2. Is “Donor Directory” intended to mean observed source funders only (recommended), or should it also support an explicitly separate “directory profiles” research mode? +3. What is the canonical user-facing amount policy: automatic historical EUR by default, selected original-currency-only scope, and explicit multi-country exclusion (the current implementation), or another policy? +4. Which of the 364 enriched profiles should be eligible for a donor-directory link if there is no direct observed grant link? Recommended: none until evidence is recorded. +5. Should a donor-detail selection be a shareable URL state? Recommended: yes after the base grant-scope contract is stable. +6. Should the map’s illustrative headquarters-to-beneficiary arrows remain? Recommended: only as an explicitly labelled optional view; never as a confirmed money route. +7. Is the AI news feature a core discovery surface or a secondary profile-research capability? The answer determines whether it belongs in details or a separate module. +8. Should the product surface every available evidence link or only show it in donor details? Recommended: details only, with “View 360Giving recipient/funder record” for the stored raw organisation JSON URL, “Visit recipient/funder website” when supplied in the observed grant, “View publisher’s grant data” for shared dataset/API links, and a separately labelled profile website when one is linked. + +## 23. Recommended Implementation Sequence + +This sequence fits the actual repository and preserves the existing modes until parity is demonstrated. + +### Phase 1 — Terminology, evidence and state contract + +- Define source funder, linked profile, source-only and registry-record terminology. +- Define the typed canonical grant-scope parse/serialise contract, including data sources. +- Add count and semantic regression tests for current dataset/linkage behaviour. +- Fix the current source-selection round-trip gap without redesigning the UI. + +### Phase 2 — Unified observed Donor Directory alongside current modes + +- Build on \`SourceFunderResults\` data/API semantics. +- Introduce status filtering based on explicit link evidence. +- Use compact default rows and an advanced filter drawer. +- Keep Organization Profiles and Registry modes unchanged/reachable while testing parity. + +### Phase 3 — Shared evidence-aware donor details + +- Introduce accessible desktop panel/tablet sheet/mobile full-screen treatment. +- Lazy-load relationships, profile facts, score, financial history and news. +- Use ranked relationship list first; retain Sankey as optional progressive enhancement when readable. + +### Phase 4 — Map connection and navigation + +- Make map-to-donor scope/back navigation fully URL-round-trippable. +- Add country/programme/time/source transfer tests and browser history tests. + +### Phase 5 — Position the registry correctly + +- Reframe Full Charity Commission Register as Advanced Registry Search. +- Retain its complete record and filters; add explicit relationship labels. + +### Phase 6 — Consolidate only after parity and optimise + +- Redirect/retire duplicated navigation only after acceptance criteria pass. +- Optimise donor aggregation/trend cache and code-split heavy visualisations. +- Consolidate CSS breakpoints and complete manual responsive/accessibility validation. + +## 24. Acceptance Criteria for the Future Redesign + +The redesign should not be accepted until all are true: + +1. A donor-list row declares whether it is an observed source funder, a linked profile, or a source-only record. +2. Source-only donors never display fabricated profile contact/HQ/type/financial data. +3. Profile and registry linkage display record type, source, match method/status/confidence where applicable. +4. Every outbound data-evidence link has an accurate label: 360Giving recipient/funder record, observed recipient/funder website, publisher grant data, or linked-profile website/data source. No publisher/API URL is presented as a recipient website or unique grant page without evidence. +5. Map country/programme/time/currency/donor/recipient/source state survives Explore donors, browser Back, refresh and shared URL. +6. Beneficiary geography is never substituted with HQ, registered office or stated focus. +7. Original amount, historical EUR conversion coverage and multi-country treatment are consistent between map, donor list and details. +8. All current Main Donors, profile and registry fields remain either visible by default, available in details, or available in advanced search as documented in the preservation matrix. +9. Registry search remains complete, cursor-paginated and visibly distinct from observed donors. +10. Donor aggregation meets an agreed latency budget for large countries before it becomes the primary navigation destination. Suggested target: a cached/typical result under 1 second and a cold result with useful progress under 2 seconds, measured end-to-end. +11. Detail loads initial identity/summary first and lazy-loads secondary data; no unbounded relationship/grant payload is fetched by default. +12. Keyboard users can operate tabs, map, search suggestions, rows, drawers and all detail views with correct focus restoration. +13. Desktop, 1280 laptop, 1024 landscape tablet, 834 portrait tablet and 390 mobile are manually tested for overflow, labels, touch targets, sheets/panels and loading state. +14. Automated tests cover canonical identity, unresolved foreign IDs, zero/one/many links, registry-link evidence, currency/multi-country policy, source-link labels, pagination, empty states and URL round trips. + +## 25. Evidence Appendix + +### Repository state + +At audit time the worktree was not clean. Existing modified/untracked files included \`README.md\`, \`frontend/src/App.tsx\`, \`frontend/src/components/GrantWorldMap.tsx\`, \`frontend/src/components/OverviewDashboard.tsx\`, \`frontend/src/index.css\`, multiple BFF/data/test files, and the untracked \`frontend/src/components/SourceFunderResults.tsx\`. This report does not attribute those changes to a particular implementation decision beyond describing their current observable code. + +### Key source locations + +| Area | Source location | +|---|---| +| App navigation, map hand-off, modes, list/modals | \`frontend/src/App.tsx:410-484, 585-894, 1047-1114, 1442-1564, 1790-2147, 2365-2842\` | +| Overview URL/requests | \`frontend/src/components/OverviewDashboard.tsx:145-257\` | +| World-map behaviour/accessibility | \`frontend/src/components/GrantWorldMap.tsx:116-124, 250-295, 353-360, 448-513, 598-653\` | +| Observed donor list | \`frontend/src/components/SourceFunderResults.tsx:12-452\` | +| Registry view | \`frontend/src/components/RegistryDirectory.tsx:81-321\` | +| Responsive styling | \`frontend/src/index.css:868-888, 3007-3069\` | +| BFF routes | \`src/bff/charity.py:24-146, 303-470\` | +| Overview warm-up | \`src/bff/main.py:51-68\` | +| Source-funder aggregation/details | \`src/bff/repositories.py:1716-2151\` | +| Registry query | \`src/bff/repositories.py:2918-3137\` | +| Directory profile list query | \`src/bff/repositories.py:3220-3419\` | +| Direct profile grant / relationships | \`src/bff/repositories.py:4372+\` | +| Core/derived schemas | \`src/data/db_loader.py:67-113, 229-259\` | +| Registry/link schemas | \`src/data/registry.py:82-128\` | + +### Current local data counts used in this audit + +| Measure | Count / value | +|---|---:| +| Grants | 200,000 | +| Grant source | 360Giving only | +| Source-funder identities | 292 | +| Source-recipient identities | 75,760 | +| Enriched directory profiles | 364 | +| Current Charity Commission registry records | 397,469 | +| Accepted registry links | 343 | +| Distinct profiles with accepted registry link | 64 | +| Profiles with direct observed grant link | 17 | +| Grants with usable country geography | 71,286 (35.64%) | +| Grants with programme category | 200,000 | +| Grants with valid EUR conversion | 199,579 | +| Grants without valid conversion | 421 | +| Grants with valid stored HTTP(S) source URL | 200,000 | +| Distinct grant source URLs | 309 | +| Raw grants with 360Giving recipient JSON URL | 183,947 | +| Raw grants with 360Giving funder JSON URL | 200,000 | +| Raw grants with publisher-provided recipient website | 72,906 | +| Distinct publisher-provided recipient websites | 35,841 | +| Raw grants with publisher-provided funder website | 3,594 | +| Profile records with a stored source URL | 361 | +| Overview cache rows | 37 | + +### Documentation drift + +\`README.md\` still describes an earlier 100,000-grant / 39,018-mapped state in several locations (including lines approximately 12, 71-72, 127, 141 and 149). The current database is 200,000 grants and 71,286 mapped beneficiary-country grants. Any user-facing or operational documentation should be refreshed as a separate, deliberate change so dataset claims do not drift again. + +### Test/build record + + PYTHONPATH=src ./venv/bin/python -m pytest -q + 242 passed, 39 warnings in 12.70s + + cd frontend && npm run build + Build succeeded; Vite emitted a >500 kB bundle warning. + + cd frontend && npm run lint + Exit 0; five existing react-hooks/exhaustive-deps warnings. + +No new screenshots were captured because browser automation was not installed in this environment. diff --git a/docs/donor_directory_data_preservation.md b/docs/donor_directory_data_preservation.md new file mode 100644 index 0000000..7b9a7a5 --- /dev/null +++ b/docs/donor_directory_data_preservation.md @@ -0,0 +1,93 @@ +# Donor Directory data-preservation report + +Date: 2026-07-25 + +## Preserved entity layers + +The implementation keeps these layers separate: + +1. **Observed source funder** — stable source namespace plus source ID, or a namespaced normalized-name fallback. +2. **Enriched organization profile** — the existing `charities` record. +3. **Official registry organization** — the existing Charity Commission registry record. +4. **Link evidence** — explicit source-to-profile grant linkage and accepted profile-to-registry linkage. + +No name-only presentation merge was added. A source funder may have zero, one, or multiple supported profile IDs. Multiple IDs remain unresolved and expose no selected profile. + +## Before/after source counts + +| Population | Before | After | +|---|---:|---:| +| grants | 200,000 | 200,000 | +| enriched profiles | 364 | 364 | +| current registry records | 397,469 | 397,469 | +| source records rewritten | 0 | 0 | +| unresolved grant foreign keys deleted | 0 | 0 | +| new derived facts | 0 | 71,403 | + +The derived rows cover 71,286 distinct grants with usable beneficiary-country and funder identity evidence. A multi-country grant has one fact per associated country, which explains the row/grant difference. + +## Integrity checks + +Before applying the derived schema to the current database: + +- `PRAGMA integrity_check`: `ok` +- `PRAGMA foreign_key_check` rows: 2,294 + +After the derived schema/index existed: + +- `PRAGMA integrity_check`: `ok` +- `PRAGMA foreign_key_check` rows: 2,294 + +The 2,294 foreign-key findings pre-date this redesign and are unresolved stored grant profile IDs. The redesign neither deletes nor silently repairs them. A fact row stores `linked_profile_id` only when that ID exists in `charities`; the original unresolved value remains untouched in `grants`. + +## Derived-table provenance + +`grant_source_funder_facts` is generated exclusively from existing grant columns and normalized derived indexes. It stores no copied raw grant JSON. Every row includes `data_revision`, source namespace, source identity evidence, source record ID, and publisher record URL. + +The current derived coverage is: + +- 71,403 grant-country facts; +- 71,286 distinct grants; +- 133 source-funder identities; +- 113 beneficiary countries. + +Facts with full multi-country awards remain activity evidence but their full amount is excluded from country-attributable funding totals. + +## Evidence inventory + +The current audit found: + +- all 200,000 grants retain a source URL and source record ID; +- 183,947 raw grants contain `recipients[0].self`; +- all 200,000 raw grants contain a funder `self` reference; +- 72,906 raw grants contain a recipient-organization website value; +- 3,594 raw grants contain a funder-organization website value; +- 361 of 364 enriched profiles contain a source URL. + +The new BFF exposes these stored values as typed safe links. It does not call the linked 360Giving organization JSON or any website. That preserves source provenance without turning read-time presentation into an external crawler. + +## Loader and invalidation safety + +- Schema creation is idempotent. +- `insert_grants`, JSONL load, and ECB backfill paths remove `grant_overview_index_revision` and cached Overview payloads. +- Atomic staging loads cannot preserve a stale revision after replacing grants. +- Rebuilds delete and regenerate only derived beneficiary, programme, source-funder, and cache rows. +- No generated database or cache artifact is added to Git. + +## Backup and regeneration + +For this local implementation audit, a full database copy was made at `/private/tmp/fip-donor-redesign-baseline.db` before testing the migration. `/private/tmp` is not a deployment backup and may be removed by the OS. + +The durable recovery mechanism is the existing atomic database loader plus stored preprocessed/source inputs. A production rollout should take its own managed backup before first migration. + +To regenerate the facts after a load, start the application or call `rebuild_grant_overview_indexes` through the existing repository/loader workflow. To roll back, drop only the derived fact table and metadata keys; source grants, profiles, registry rows, and links remain sufficient to regenerate prior indexes. + +## Preservation gate + +- Observed source funders remain searchable: passed. +- Unlinked enriched profiles remain available in Organization Research: passed. +- Complete registry remains available in Advanced Charity Commission Search: passed. +- Multiple links are not auto-selected: passed by automated test. +- Existing source counts preserved: passed. +- No new foreign-key violations: passed by before/after count. +- Legacy donor UI removed: **no**; retained until browser parity is validated. diff --git a/docs/donor_directory_performance_report.md b/docs/donor_directory_performance_report.md new file mode 100644 index 0000000..59c03e7 --- /dev/null +++ b/docs/donor_directory_performance_report.md @@ -0,0 +1,91 @@ +# Donor Directory performance report + +Date: 2026-07-25 + +## Environment + +- Hardware: MacBook Air (Apple M1, 8 cores, 8 GB RAM, arm64) +- OS/kernel: Darwin 25.4.0 (local macOS environment) +- Python: 3.12.3 +- Node: 22.20.0 +- npm: 10.9.3 +- SQLite: 3.51.0 +- Database: `src/data/charities.db` +- Starting database size: 1,331,175,424 bytes +- Final database size: 1,389,834,240 bytes +- Grants: 200,000 +- Enriched profiles: 364 +- Current registry rows: 397,469 +- Derived facts: 71,403 rows, 71,286 distinct grants, 133 source-funder identities, 113 countries +- Data revision: `[200000,"v360G-virginmoneyfdn-3050","2026-07-24T15:29:23.308211Z","2026-07-10",199579,15652485145.3]` +- Code baseline: `9c46f275dcf5140d97bc45d56099d036bf602d00` + +## Timing definitions + +- **Repository** means a direct in-process call to `SQLiteCharityRepository`; JSON payload byte counts use local serialization where recorded. +- **HTTP** means a completed request through a listening BFF socket. +- **Browser** means interaction/route start to usable rendered content in a real browser. + +Repository measurements do not prove HTTP or browser targets. + +## Before and after + +The pre-change baseline used a fresh copied database with the persisted Overview payload cache cleared. The old list path loaded broad grant rows and performed source identity grouping in Python. Final timings use the reproducible derived facts and current local database. + +| Operation | Before repository | Final repository | State / notes | +|---|---:|---:|---| +| GB donor list, 25 rows | 15.754 s | 0.772 s first call; 0.296 s repeat | 131 funders / 68,430 grants; final list is SQL filtered/grouped/sorted/paged | +| US donor list | 0.088 s | 0.234 s | 22 funders / 585 grants; richer typed response adds fixed joins; still bounded | +| NG donor list | 0.096 s | 0.022 s | Post-index test-copy measurement; 11 funders / 308 grants | +| GB linked large-donor detail | 16.293 s | 0.324 s summary | 1,965 grants; summary returns no grant sample/evidence | +| Same full lazy detail | 16.293 s | 0.071 s warm; 0.620 s isolated first full call | 50 grant samples, 50 ranked recipients, 40 evidence links capped | +| GB search + observed-only status | not available | 0.292 s | Backend search/status, 36 matching funders for `trust` in the measured scope | + +The GB list is approximately 20× faster on the first measured final repository call and approximately 53× faster on the repeated call. The large detail no longer recomputes all GB funders and is approximately 26× faster even using the isolated 0.620-second full measurement. + +## Query plan + +SQLite confirms the two critical accesses use the new indexes: + +```text +SEARCH grant_source_funder_facts USING COVERING INDEX + idx_source_funder_facts_country_key (country_code=? AND source_namespace=?) + +SEARCH fact USING INDEX idx_source_funder_facts_key_country + (source_funder_key=? AND country_code=?) +``` + +The list response reads no `raw_grant_data`. Only a selected full detail reads up to 50 raw stored grant records for typed evidence extraction. + +## Build comparison + +| Asset | Baseline | Final | +|---|---:|---:| +| Main JS | 1,952.10 kB / 607.87 kB gzip | 1,942.73 kB / 604.89 kB gzip | +| CSS | 63.34 kB / 11.19 kB gzip | 79.29 kB / 13.43 kB gzip | +| Lazy registry JS | in main bundle | 13.64 kB / 3.72 kB gzip | +| Lazy legacy donor JS | in main bundle | 15.25 kB / 4.25 kB gzip | + +The primary bundle remains above Vite’s 500 kB warning threshold, but is 9.37 kB smaller (2.98 kB smaller gzip) than baseline despite including the new primary directory/detail. The registry and legacy donor page are split out, and the clipped legacy source-funder Sankey was replaced by an accessible ranked list. Profile detail charts and other legacy Recharts code still dominate the main bundle and are a documented remaining optimization. + +## Index-build and cache cost + +The derived facts are rebuilt only when the schema or grant data revision is invalidated. A test run that combined first-time schema/fact creation and a cold full Overview aggregation took 60.420 seconds. That is a maintenance/cold-aggregation cost, not a normal Donor Directory request. It increased this local database by 58,658,816 bytes (about 4.4%). + +The BFF no longer blocks application readiness on the full Overview warmup. Supported loaders and ECB conversion backfills invalidate the revision explicitly. The next request performs one reproducible rebuild rather than using an indefinitely stale materialization. + +## HTTP and browser status + +Socket-level HTTP measurement was attempted with an isolated BFF on `127.0.0.1:8011`. The sandbox rejected local port binding. The escalation needed to bind the test process was unavailable because of the current tool-approval limit. Therefore: + +- no socket-level HTTP timing is claimed; +- no cold/warm HTTP target is claimed as passed; +- TestClient results are treated as tests, not production HTTP performance; +- no browser transition target is claimed as passed. + +## Remaining risks + +- First rebuild after a new data snapshot remains expensive and should be run as a deployment/load step for larger datasets. +- The wide legacy Overview aggregation is still slow on an uncached new query. +- The main JS bundle remains large. +- HTTP and browser timings require a locally permitted server and real browser run. diff --git a/docs/donor_directory_redesign_implementation.md b/docs/donor_directory_redesign_implementation.md new file mode 100644 index 0000000..26199a3 --- /dev/null +++ b/docs/donor_directory_redesign_implementation.md @@ -0,0 +1,179 @@ +# Donor Directory redesign implementation + +Date: 2026-07-25 +Branch: `9-fr-09-display-ranked-list-of-main-donors` +Starting commit: `9c46f275dcf5140d97bc45d56099d036bf602d00` + +## Outcome + +The application now distinguishes three record populations instead of presenting them as one organization directory: + +1. **Funding Landscape** — observed grant activity grouped by beneficiary geography. +2. **Donor Directory** — source-namespaced funder identities observed in the current grant scope. +3. **Organization Research / Advanced Charity Commission Search** — secondary enriched-profile and official-registry research. + +The default Donor Directory never inserts an enriched profile or registry row merely because its name or registered location resembles a source funder. A profile appears only as an explicit zero/one/many linkage attached to the same observed source identity. + +## Shared application shell + +`frontend/src/components/AppHeader.tsx` is the single global header implementation. It always renders: + +`Foundation Intelligence Platform` → `Filters` → `Reset` → `Data sources` + +The active page supplies the filter count and contextual actions: + +- Funding Landscape opens and resets grant-scope filters. +- Donor Directory opens and resets donor and grant-scope filters. +- Organization Research targets its existing profile filters. +- Advanced Charity Commission Search targets and resets registry filters. +- Pipeline Monitor keeps contextual actions visible but disabled. + +Data-source selection is global, synchronized, and persisted in `grant_sources`. An explicit empty value means no sources selected; a missing parameter means the page may use its supplied defaults. + +## Canonical GrantScope + +`frontend/src/lib/grantScope.ts` owns: + +- URL parsing and validation; +- normalization and serialization; +- API query construction; +- active chips and filter counts; +- contextual value removal; +- scope equality; +- Donor Directory search/status/sort/page/detail route state. + +The existing query names remain compatible: + +| Meaning | URL parameter | API parameter | +|---|---|---| +| beneficiary country | `funder_country` | `beneficiary_country` | +| beneficiary terms | `grant_geo` | `beneficiary_geographies` | +| programme areas | `grant_programme` | `programme_areas` | +| date range | `grant_from`, `grant_to` | `date_from`, `date_to` | +| currency | `grant_currency` | `currency` | +| donor / recipient text | `grant_donor`, `grant_recipient` | `donor`, `recipient` | +| source selection | `grant_sources` | `sources` | +| directory search | `donor_search` | `search` | +| profile-link status | `donor_status` | `profile_status` | +| sort / page | `funder_sort`, `funder_page` | `sort`, `page` | +| selected detail | `donor` | path parameter | + +The small `view` parameter identifies primary/secondary application state (`donors`, `research`, `registry`, `legacy-donors`, or `pipeline`); Funding Landscape remains the default when it is absent. + +Browser Back, Forward, refresh, direct URLs, copied URLs, detail opening/closing, search, sorting, pagination, and source selection use this contract. The pure contract is tested with the Node test runner. + +## Backend design + +The broad list no longer selects and parses the wide `grants.raw_grant_data` column for an entire country. The idempotent schema migration creates `grant_source_funder_facts`, a reproducible narrow table with one row per grant-country association and these semantics: + +- source namespace; +- stable source-funder key; +- source organization ID where present; +- normalized-name fallback where required; +- identity method and display name; +- recipient identity and label; +- award date and original/EUR monetary status; +- multi-country count; +- supported linked profile ID; +- publisher record reference; +- data revision. + +The list query filters before aggregation, groups in SQL, sorts aggregated results, and pages in SQL. Programme classifications use `grant_programme_categories`; geography uses the normalized beneficiary indexes. Raw JSON is not read by list requests. + +Supported writers delete `grant_overview_index_revision` after changing grant or exchange-rate facts. The next request rebuilds all derived tables transactionally. `load_jsonl_to_db` now also invalidates preserved staging revisions so an atomic reload cannot publish stale or empty derived facts. + +The BFF startup no longer awaits a full Overview aggregation. Repository initialization is immediate; persisted caches are reused and expensive aggregation is request-driven. This prevents the health endpoint from appearing crashed during a cold 1.3 GB scan. + +## Typed result and linkage status + +List responses preserve legacy fields while adding: + +- `kind: source_funder`; +- typed `identity`; +- `evidence_sources`; +- `observed_activity`; +- `amount_policy`; +- explicit `profile_link` with `none`, `single`, or `multiple` status; +- accepted registry link information only through a single supported profile. + +Status filters are backend operations: + +- `all` — every observed source identity in scope; +- `linked` — exactly one supported profile link; +- `observed_only` — zero or multiple supported profile links. + +Multiple candidates are never auto-selected and expose no profile fields in the donor row. + +## Shared donor detail + +The primary Donor Directory opens a right-side detail panel on desktop, a full-width sheet on tablet, and a full-screen sheet on mobile. The list and selected row remain present on desktop. Detail state is stored in the URL and Escape/Back close it. + +Initial requests use `detail_level=summary` and return only summary facts plus a relationship availability summary. Opening grant activity or source evidence makes the explicit `detail_level=full` request. The full query is restricted to the selected source-funder key and country; it does not rebuild the country directory. + +The default relationship representation is a ranked, keyboard-readable recipient list with amount, grant count, and latest date. The previously clipped source-funder Sankey in the legacy overlay was also replaced by a ranked textual equivalent. A Sankey is optional, not loaded by the primary donor workflow, and may be reintroduced later only as a lazy readable enhancement with the list retained. + +Observed-only details state that no linked profile exists. Linked details show the profile as a separate section and route to Organization Research. Observed, enriched-profile, and registry facts are not merged. + +## Evidence links + +The BFF parses already stored values into these typed kinds: + +- `360giving_funder_record`; +- `360giving_recipient_record`; +- `observed_funder_website`; +- `observed_recipient_website`; +- `publisher_grant_data`; +- `profile_website`; +- `profile_source`. + +Only syntactically valid HTTP(S) URLs without embedded credentials are returned. The BFF does not fetch, proxy, preflight, follow, or validate external destinations. The frontend opens a link only after an explicit click with `target="_blank"` and `rel="noopener noreferrer"`. Source organization identifiers are visible and copyable in methodology. + +## Navigation and legacy access + +Primary navigation is now: + +1. Funding Landscape +2. Donor Directory +3. Pipeline Monitor + +Organization Research and Advanced Charity Commission Search are secondary actions inside the Donor Directory. Their data and filters remain intact. The former Main Donors component is lazy-loaded behind **Legacy donor view** because real-browser responsive acceptance could not be completed in this environment. + +## Phase checkpoints + +### Phase 1 — contract and performance foundation + +- Shared header, GrantScope, derived source-funder facts, optimized list/detail, source invalidation, tests, and measurements implemented. +- Backend focused suite: 20 passed. +- Frontend contract tests: 5 passed. +- Safe to continue: yes. + +### Phase 2 — directory and map hand-off + +- Compact observed donor rows, backend search/status/sort/page, active chips, filter drawer, reset, and History API state implemented. +- Source selection round trip covered by tests. +- Legacy data remains accessible. +- Safe to continue: yes. + +### Phase 3 — donor detail and evidence + +- Responsive detail shell, summary-first fetch, lazy full activity, ranked recipients, conditional linkage, and typed evidence implemented. +- No fabricated profile data for observed-only/multiple-link results is covered by backend tests. +- Safe to continue: yes. + +### Phase 4 — secondary research and validation + +- Navigation hierarchy, shared header, lazy registry/legacy routes, and secondary page introductions implemented. +- Real-browser validation and socket-level HTTP measurement remain incomplete because local process binding/escalation was unavailable. +- Legacy view retained as required by the removal gate. +- Final removal gate: **not passed**. + +## Rollback + +The source `grants`, `charities`, registry, and linkage tables are unchanged. To roll back the derived structure: + +1. deploy the prior application revision; +2. drop only `grant_source_funder_facts` and its indexes if desired; +3. remove `grant_overview_schema_version` and `grant_overview_index_revision` metadata keys; +4. allow the prior code to rebuild its existing beneficiary/programme indexes. + +Dropping the derived table loses no source record. Do not delete or rewrite unresolved grant IDs to perform a rollback. diff --git a/docs/donor_directory_responsive_validation.md b/docs/donor_directory_responsive_validation.md new file mode 100644 index 0000000..746ed8b --- /dev/null +++ b/docs/donor_directory_responsive_validation.md @@ -0,0 +1,83 @@ +# Donor Directory responsive-validation report + +Date: 2026-07-25 + +## Acceptance status + +**Incomplete — not claimed as passed.** + +The implementation includes responsive rules and accessibility behavior, but this environment did not permit the required real-browser run. Chrome, Edge, and Firefox applications are installed locally; however, the sandbox rejected binding the temporary local BFF port and the required escalation was unavailable under the current tool-approval limit. No screenshots were captured. + +Per the removal gate, the legacy donor view remains lazy-loaded and reachable from Organization Research. + +## Implemented responsive behavior + +### Shared header + +- One `AppHeader` is used for primary and secondary views. +- Desktop/laptop retain text labels and fixed action order. +- At narrow widths actions become icon controls while accessible names remain in the DOM. +- Reset remains present and uses the native disabled attribute. +- Data sources uses the same persistent disclosure across pages. +- Focus-visible styling is retained. + +### Donor Directory + +- Desktop uses compact semantic list rows and a sticky right detail panel. +- Tablet switches detail to a full-width sheet below the global header. +- Mobile uses the full screen below the header; donor rows become stacked rather than a desktop-width table. +- Search/status controls are keyboard-operable. +- Selected rows use text/status plus colour and `aria-pressed`. +- Detail selection, search, status, sort, page, and scope are URL state. +- Escape and browser Back close details. +- The filter drawer traps focus, supports Escape, and returns focus to the global filter action. +- Loading and error states use live/status semantics. + +### Secondary research + +- Organization Research and registry search retain their data and existing responsive behavior. +- Registry result cards remain semantic buttons. +- Registry and legacy donor modules are lazy-loaded. + +## Required viewport matrix + +The following real-browser checks remain outstanding: + +| Viewport | Source inspection | Real browser | Screenshot | Acceptance | +|---|---|---|---|---| +| 1440 × 900 | completed | not run | none | incomplete | +| 1280 × 800 | completed | not run | none | incomplete | +| 1024 × 768 | completed | not run | none | incomplete | +| 834 × 1194 | completed | not run | none | incomplete | +| 390 × 844 | completed | not run | none | incomplete | + +## Manual/browser checklist + +At every viewport, verify: + +- global title and Filters → Reset → Data sources order; +- no header or page horizontal overflow; +- disabled Reset does not shift layout; +- Data sources disclosure remains on screen and all choices are reachable; +- navigation drawer open/close and focus behavior; +- Funding Landscape map height and country keyboard behavior; +- Donor Directory search, status pills, row truncation, pagination, and empty/error/loading states; +- desktop list width while detail is open; +- tablet/mobile sheet dimensions and scrolling; +- selected row restoration and scroll position after closing detail; +- filter-drawer focus trap and trigger focus restoration; +- external evidence labels and explicit-click-only opening; +- Organization Research filters and profile detail; +- registry filters, cards, cursor loading, and detail close behavior; +- keyboard focus visibility throughout. + +## Automated validation completed + +- TypeScript production build: passed. +- Oxlint: passed with five pre-existing `App.tsx` hook-dependency warnings; no warnings remain in newly added Donor Directory, GrantScope, AppHeader, or modified Overview code. +- GrantScope Node tests: 5 passed. +- Focused backend donor/overview tests: 20 passed. + +## Remaining corrective gate + +Do not remove the legacy donor route or claim responsive acceptance until a real browser run at all five target sizes produces durable screenshots (or an equivalent recorded artifact) and verifies overflow, focus lifecycle, loading/error states, and Back behavior. diff --git a/frontend/.gitignore b/frontend/.gitignore index a547bf3..a2c5b94 100644 --- a/frontend/.gitignore +++ b/frontend/.gitignore @@ -10,6 +10,7 @@ lerna-debug.log* node_modules dist dist-ssr +.test-dist *.local # Editor directories and files diff --git a/frontend/package.json b/frontend/package.json index 227f878..be465ad 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -7,6 +7,7 @@ "dev": "vite", "build": "tsc -b && vite build", "lint": "oxlint", + "test": "tsc -p tsconfig.tests.json && node --test .test-dist/tests/*.test.js", "preview": "vite preview" }, "dependencies": { diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index 323859f..338291c 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -1,21 +1,21 @@ -import { useState, useEffect, useMemo, useRef } from "react"; +import { lazy, Suspense, useState, useEffect, useMemo, useRef } from "react"; import { Building2, + ArrowLeft, TrendingUp, Activity, Search, - SlidersHorizontal, Terminal, ArrowRight, TrendingDown, DollarSign, - Database, Play, Newspaper, - Menu, + LoaderCircle, PanelLeftClose, PanelLeftOpen, - RotateCcw, + Star, + X, } from "lucide-react"; import { BarChart, @@ -27,17 +27,25 @@ import { ResponsiveContainer, AreaChart, Area, - Sankey } from "recharts"; import GrantWorldMap from "./components/GrantWorldMap"; -import OverviewDashboard from "./components/OverviewDashboard"; -import RegistryDirectory from "./components/RegistryDirectory"; +import OverviewDashboard, { type OverviewFilters } from "./components/OverviewDashboard"; +import AppHeader from "./components/AppHeader"; +import DonorDirectoryPage, { type FavoriteDonorPayload, type HeaderContextState } from "./components/DonorDirectoryPage"; +import { + applyGrantScopeToParams, + grantScopeFromUrl, + type GrantScope, +} from "./lib/grantScope"; import amplifyLogo from "./assets/amplify-logo.svg"; import type { GrantMapFilters, GrantMapResponse, + SourceFunderCountrySelection, } from "./components/GrantWorldMap"; +const RegistryDirectory = lazy(() => import("./components/RegistryDirectory")); + // Configuration for API requests const API_BASE = import.meta.env.VITE_API_BASE_URL || `${window.location.protocol}//${window.location.hostname}:8000`; @@ -71,6 +79,128 @@ interface Charity { source_record_id?: string | null; source_url?: string | null; transaction_coverage?: string; + relevance_score?: number | null; + score_confidence?: number | null; + score_completeness?: number | null; + score_target?: string | null; + score_version?: string | null; + score_configuration_status?: string | null; +} + +type FavoriteProfile = { + key: string; + profile: Charity; + savedAt: number; +}; + +type FavoriteResearchView = { + key: string; + label: string; + filters: { + searchTerm: string; + selectedTags: string[]; + selectedFoundationRegions: string[]; + selectedRecipientRegions: string[]; + annualGivingIndex: number; + maxAnnualGivingInput: string; + avgGrantSizeIndex: number; + maxAvgGrantSizeInput: string; + profileSort: "score_desc" | "income_desc" | "name_asc"; + }; + savedAt: number; +}; + +type FavoriteLandscapeView = { + key: string; + label: string; + route: string; + savedAt: number; +}; + +type FavoritesState = { + profiles: FavoriteProfile[]; + donors: FavoriteDonorPayload[]; + researchViews: FavoriteResearchView[]; + landscapeViews: FavoriteLandscapeView[]; +}; + +type NewsSourceItem = { + title: string; + link: string; + source: string; + published: string; + note?: string; +}; + +type NewsSummaryPayload = { + foundation: string; + summary: string; + sources: NewsSourceItem[]; + searched_weeks: number; + generated_at: string; +}; + +type NewsProgressStep = "discovering" | "reading" | "summarizing"; + +type SavedNewsRun = NewsSummaryPayload & { + organizationKey: string; + savedAt: string; + mode: "live" | "illustrative"; +}; + +const FAVORITES_STORAGE_KEY = "foundation-intelligence-favorites-v1"; +const NEWS_RUN_STORAGE_KEY = "foundation-intelligence-news-runs-v1"; +const EMPTY_FAVORITES: FavoritesState = { profiles: [], donors: [], researchViews: [], landscapeViews: [] }; +const NEWS_PROGRESS_STEPS: Array<{ key: NewsProgressStep; label: string; detail: string }> = [ + { key: "discovering", label: "Find recent coverage", detail: "Search current Google News coverage for this organization." }, + { key: "reading", label: "Read source articles", detail: "Resolve publisher links and extract article evidence." }, + { key: "summarizing", label: "Create AI briefing", detail: "Generate a cited briefing from the collected evidence." }, +]; + +function loadFavorites(): FavoritesState { + try { + const stored = window.localStorage.getItem(FAVORITES_STORAGE_KEY); + if (!stored) return EMPTY_FAVORITES; + const parsed = JSON.parse(stored) as Partial; + return { + profiles: Array.isArray(parsed.profiles) ? parsed.profiles : [], + donors: Array.isArray(parsed.donors) ? parsed.donors : [], + researchViews: Array.isArray(parsed.researchViews) ? parsed.researchViews : [], + landscapeViews: Array.isArray(parsed.landscapeViews) ? parsed.landscapeViews : [], + }; + } catch { + return EMPTY_FAVORITES; + } +} + +function loadNewsRuns(): SavedNewsRun[] { + try { + const stored = window.localStorage.getItem(NEWS_RUN_STORAGE_KEY); + if (!stored) return []; + const parsed = JSON.parse(stored); + if (!Array.isArray(parsed)) return []; + return parsed.filter((run): run is SavedNewsRun => ( + run + && typeof run.organizationKey === "string" + && typeof run.foundation === "string" + && typeof run.summary === "string" + && Array.isArray(run.sources) + && typeof run.savedAt === "string" + )); + } catch { + return []; + } +} + +function newsOrganizationKey(charity: Charity): string { + return `organization:${charity.registered_charity_number}:${charity.source_record_id || ""}`; +} + +function formatNewsDate(value: string | null | undefined): string { + if (!value) return "Date unavailable"; + const date = new Date(value); + if (Number.isNaN(date.getTime())) return value; + return new Intl.DateTimeFormat("en-GB", { day: "numeric", month: "short", year: "numeric" }).format(date); } interface KPIStats { @@ -168,13 +298,17 @@ interface GrantDetail { } interface SankeyNode { + id?: string; name: string; + role?: "donor" | "recipient"; + depth?: number; } interface SankeyLink { source: number; target: number; value: number; + grantCount: number; } interface SankeyData { @@ -368,6 +502,12 @@ const ANNUAL_GIVING_LABELS = ["£0", "£100k", "£500k", "£1M", "£5M", "£10M" const AVG_GRANT_SIZE_STEPS = [0, 1000, 5000, 10000, 50000, 100000, 250000, 500000, 1000000]; const AVG_GRANT_SIZE_LABELS = ["£0", "£1k", "£5k", "£10k", "£50k", "£100k", "£250k", "£500k", "£1M"]; +const optionalFilterAmount = (value: string): number | null => { + if (!value.trim()) return null; + const amount = Number(value); + return Number.isFinite(amount) && amount >= 0 ? amount : null; +}; + const SECTORS = [ { value: "Socio-economic Development, Poverty", label: "Poverty & Economic Development" }, { value: "Environment/Climate", label: "Climate & Environment" }, @@ -398,10 +538,25 @@ const EMPTY_GRANT_MAP_FILTERS: GrantMapFilters = { }; export default function App() { - const [activeTab, setActiveTab] = useState<"overview" | "directory" | "admin">("overview"); - const [directoryMode, setDirectoryMode] = useState<"profiles" | "registry">("profiles"); + const hasFunderRoute = () => Boolean(new URLSearchParams(window.location.search).get("funder_country")); + const initialView = () => new URLSearchParams(window.location.search).get("view"); + const [activeTab, setActiveTab] = useState<"overview" | "directory" | "favorites" | "admin">(() => initialView() === "pipeline" ? "admin" : initialView() === "favorites" ? "favorites" : (hasFunderRoute() || ["donors", "research", "registry"].includes(initialView() || "")) ? "directory" : "overview"); + const [directoryMode, setDirectoryMode] = useState<"donors" | "profiles" | "registry">(() => initialView() === "research" ? "profiles" : initialView() === "registry" ? "registry" : "donors"); const [overviewFilterCount, setOverviewFilterCount] = useState(0); const [overviewFiltersOpen, setOverviewFiltersOpen] = useState(false); + const [donorHeaderState, setDonorHeaderState] = useState({ + filterCount: 0, + resetDisabled: true, + filtersExpanded: false, + }); + const [registryHeaderState, setRegistryHeaderState] = useState({ + filterCount: 0, + resetDisabled: true, + filtersExpanded: false, + }); + const [profileFiltersOpen, setProfileFiltersOpen] = useState(false); + const [favorites, setFavorites] = useState(loadFavorites); + const [donorReturnToFavorites, setDonorReturnToFavorites] = useState(false); const [sidebarCollapsed, setSidebarCollapsed] = useState(() => { const saved = window.localStorage.getItem("sidebar-collapsed"); return saved === null ? window.innerWidth < 1280 : saved === "true"; @@ -410,11 +565,20 @@ export default function App() { // Data states const [stats, setStats] = useState(MOCK_STATS); - const [dataSourceSelections, setDataSourceSelections] = useState>({}); + const [dataSourceSelections, setDataSourceSelections] = useState>(() => { + const params = new URLSearchParams(window.location.search); + if (!params.has("grant_sources")) return {}; + const selected = new Set((params.get("grant_sources") || "").split(",").filter(Boolean)); + return Object.fromEntries(DEFAULT_DATA_SOURCES.map(source => [source, selected.has(source)])); + }); const [charities, setCharities] = useState(MOCK_CHARITIES); + const [profileSort, setProfileSort] = useState<"score_desc" | "income_desc" | "name_asc">("score_desc"); const [profileOffset, setProfileOffset] = useState(0); const [profilesHaveMore, setProfilesHaveMore] = useState(false); const [loadingMoreProfiles, setLoadingMoreProfiles] = useState(false); + const [directoryInitialLoaded, setDirectoryInitialLoaded] = useState(false); + const directoryRequestRef = useRef(null); + const directoryRequestSequenceRef = useRef(0); const [mapData, setMapData] = useState(EMPTY_MAP); const [mapLoading, setMapLoading] = useState(false); const [mapError, setMapError] = useState(null); @@ -433,8 +597,11 @@ export default function App() { // News summarizer states const [newsLoading, setNewsLoading] = useState(false); - const [newsSummary, setNewsSummary] = useState(null); + const [newsSummary, setNewsSummary] = useState(null); const [newsError, setNewsError] = useState(null); + const [newsProgressStep, setNewsProgressStep] = useState(null); + const [newsRunMode, setNewsRunMode] = useState<"live" | "illustrative" | null>(null); + const [newsRuns, setNewsRuns] = useState(loadNewsRuns); // Filter states const [searchTerm, setSearchTerm] = useState(""); @@ -447,12 +614,43 @@ export default function App() { const [beneficiaryLocationOptions, setBeneficiaryLocationOptions] = useState(BENEFICIARY_GEOGRAPHIES); const [directoryHandoff, setDirectoryHandoff] = useState({ version: 0, query: "", beneficiaryGeography: "" }); const [annualGivingIndex, setAnnualGivingIndex] = useState(0); + const [maxAnnualGivingInput, setMaxAnnualGivingInput] = useState(""); const [avgGrantSizeIndex, setAvgGrantSizeIndex] = useState(0); + const [maxAvgGrantSizeInput, setMaxAvgGrantSizeInput] = useState(""); const [loading, setLoading] = useState(false); const [isBffOnline, setIsBffOnline] = useState(false); const [initialLoading, setInitialLoading] = useState(true); const [apiError, setApiError] = useState(null); const [authError, setAuthError] = useState(null); + + useEffect(() => { + if (activeTab !== "directory" || directoryMode !== "profiles") setProfileFiltersOpen(false); + }, [activeTab, directoryMode]); + + useEffect(() => { + const syncRouteMode = () => { + const view = initialView(); + if (view === "pipeline") { + setActiveTab("admin"); + } else if (view === "favorites") { + setActiveTab("favorites"); + } else if (view === "research") { + setDirectoryMode("profiles"); + setActiveTab("directory"); + } else if (view === "registry") { + setDirectoryMode("registry"); + setActiveTab("directory"); + } else if (view === "donors" || hasFunderRoute()) { + setDirectoryMode("donors"); + setActiveTab("directory"); + } else { + setActiveTab("overview"); + } + }; + window.addEventListener("popstate", syncRouteMode); + return () => window.removeEventListener("popstate", syncRouteMode); + }, []); + // The stats endpoint returns a fresh array on every response. Keep the // source list referentially stable while its values are unchanged so it // cannot retrigger directory and overview requests indefinitely. @@ -461,12 +659,52 @@ export default function App() { () => dataSourceNamesKey.split("\u001f").filter(Boolean), [dataSourceNamesKey], ); - const isDataSourceSelected = (source: string) => dataSourceSelections[source] !== false; const selectedDataSources = useMemo( () => dataSourceNames.filter(source => dataSourceSelections[source] !== false), [dataSourceNames, dataSourceSelections], ); + const toggleDataSource = (source: string) => { + setDataSourceSelections(current => ({ + ...current, + [source]: current[source] === false, + })); + }; + + useEffect(() => { + const currentScope = grantScopeFromUrl(window.location.search, selectedDataSources); + const params = applyGrantScopeToParams( + new URLSearchParams(window.location.search), + { ...currentScope, sources: selectedDataSources }, + { persistEmptySources: true }, + ); + const suffix = params.toString(); + window.history.replaceState( + window.history.state, + "", + `${window.location.pathname}${suffix ? `?${suffix}` : ""}`, + ); + }, [selectedDataSources]); + + useEffect(() => { + const syncSourcesFromRoute = () => { + const params = new URLSearchParams(window.location.search); + if (!params.has("grant_sources")) return; + const selected = new Set((params.get("grant_sources") || "").split(",").filter(Boolean)); + setDataSourceSelections(Object.fromEntries(dataSourceNames.map(source => [source, selected.has(source)]))); + }; + window.addEventListener("popstate", syncSourcesFromRoute); + return () => window.removeEventListener("popstate", syncSourcesFromRoute); + }, [dataSourceNames]); const selectedCharityId = selectedCharity?.registered_charity_number ?? null; + const savedNewsRun = useMemo(() => { + if (!selectedCharity) return null; + return newsRuns.find(run => run.organizationKey === newsOrganizationKey(selectedCharity)) || null; + }, [newsRuns, selectedCharity]); + const savedNewsRunIsOpen = Boolean( + savedNewsRun + && newsSummary + && newsSummary.generated_at === savedNewsRun.generated_at, + ); const financialHistoryData = useMemo(() => { const history = selectedCharityDetail?.financial_history; if (!Array.isArray(history)) return []; @@ -483,8 +721,27 @@ export default function App() { Expenditure: item.expenditure || 0, })); }, [selectedCharityDetail?.financial_history]); - - + const selectedCharityFlowRows = useMemo(() => { + if (!sankeyData || selectedCharityId === null) return []; + const selectedNodeId = `organization:${selectedCharityId}`; + return sankeyData.links + .map(link => { + const source = sankeyData.nodes[link.source]; + const target = sankeyData.nodes[link.target]; + if (!source || !target) return null; + if (source.id === selectedNodeId) { + return { counterparty: target.name, direction: "Awarded to", amount: link.value, grantCount: link.grantCount }; + } + if (target.id === selectedNodeId) { + return { counterparty: source.name, direction: "Received from", amount: link.value, grantCount: link.grantCount }; + } + return null; + }) + .filter((row): row is { counterparty: string; direction: "Awarded to" | "Received from"; amount: number; grantCount: number } => row !== null) + .sort((left, right) => right.amount - left.amount); + }, [sankeyData, selectedCharityId]); + const selectedCharityFlowMaximum = selectedCharityFlowRows[0]?.amount || 0; + const selectedCharityFlowTotal = selectedCharityFlowRows.reduce((total, row) => total + row.amount, 0); // Admin & pipeline states const [pipelineStatus, setPipelineStatus] = useState({ status: "idle", @@ -513,15 +770,30 @@ export default function App() { }, [isBffOnline]); useEffect(() => { - fetchCharities(); - }, [isBffOnline, selectedTags, selectedFoundationRegions, selectedRecipientRegions, annualGivingIndex, avgGrantSizeIndex, selectedDataSources]); - - useEffect(() => { + if (activeTab !== "directory" || directoryMode !== "profiles") return; + const delay = searchTerm.trim() || maxAnnualGivingInput || maxAvgGrantSizeInput ? 250 : 0; const debounce = window.setTimeout(() => { fetchCharities(); - }, 350); - return () => window.clearTimeout(debounce); - }, [searchTerm]); + }, delay); + return () => { + window.clearTimeout(debounce); + directoryRequestRef.current?.abort(); + }; + }, [ + activeTab, + directoryMode, + isBffOnline, + searchTerm, + selectedTags, + selectedFoundationRegions, + selectedRecipientRegions, + annualGivingIndex, + maxAnnualGivingInput, + avgGrantSizeIndex, + maxAvgGrantSizeInput, + profileSort, + selectedDataSources, + ]); useEffect(() => { if (!isBffOnline) return; @@ -596,6 +868,8 @@ export default function App() { setNewsSummary(null); setNewsError(null); setNewsLoading(false); + setNewsProgressStep(null); + setNewsRunMode(null); fetchCharityDetail(selectedCharityId); fetchCharityGrants(selectedCharityId); fetchSankeyData(selectedCharityId); @@ -605,6 +879,8 @@ export default function App() { setNewsSummary(null); setNewsError(null); setNewsLoading(false); + setNewsProgressStep(null); + setNewsRunMode(null); setScoreData(null); } }, [selectedCharityId]); @@ -638,6 +914,14 @@ export default function App() { window.localStorage.setItem("sidebar-collapsed", String(sidebarCollapsed)); }, [sidebarCollapsed]); + useEffect(() => { + window.localStorage.setItem(FAVORITES_STORAGE_KEY, JSON.stringify(favorites)); + }, [favorites]); + + useEffect(() => { + window.localStorage.setItem(NEWS_RUN_STORAGE_KEY, JSON.stringify(newsRuns)); + }, [newsRuns]); + useEffect(() => { const receiveFilterCount = (event: Event) => setOverviewFilterCount(Number((event as CustomEvent).detail) || 0); window.addEventListener("overview-filter-count", receiveFilterCount); @@ -650,6 +934,15 @@ export default function App() { return () => window.removeEventListener("overview-filter-drawer-state", receiveFilterDrawerState); }, []); + useEffect(() => { + const receiveRegistryHeaderState = (event: Event) => { + const detail = (event as CustomEvent).detail; + if (detail) setRegistryHeaderState(detail); + }; + window.addEventListener("registry-header-state", receiveRegistryHeaderState); + return () => window.removeEventListener("registry-header-state", receiveRegistryHeaderState); + }, []); + const autoLogin = async () => { try { const resp = await fetch(`${API_BASE}/api/auth/login`, { @@ -717,9 +1010,14 @@ export default function App() { }; const fetchCharities = async (forceOnline?: boolean, append = false) => { + if (activeTab !== "directory" || directoryMode !== "profiles") return; const isOnline = forceOnline !== undefined ? forceOnline : isBffOnline; const pageSize = 50; const offset = append ? profileOffset : 0; + directoryRequestRef.current?.abort(); + const controller = new AbortController(); + directoryRequestRef.current = controller; + const requestSequence = ++directoryRequestSequenceRef.current; if (append) { setLoadingMoreProfiles(true); } else { @@ -736,20 +1034,39 @@ export default function App() { if (minGiving > 0) { filtered = filtered.filter(c => (c.latest_expenditure || 0) >= minGiving); } + const maxGiving = optionalFilterAmount(maxAnnualGivingInput); + if (maxGiving !== null) { + filtered = filtered.filter(c => (c.latest_expenditure || 0) <= maxGiving); + } const minAvgGrant = AVG_GRANT_SIZE_STEPS[avgGrantSizeIndex]; if (minAvgGrant > 0) { filtered = filtered.filter(c => ((c.latest_expenditure || 0) / 10) >= minAvgGrant); } + const maxAvgGrant = optionalFilterAmount(maxAvgGrantSizeInput); + if (maxAvgGrant !== null) { + filtered = filtered.filter(c => ((c.latest_expenditure || 0) / 10) <= maxAvgGrant); + } + if (profileSort === "score_desc") { + filtered.sort((left, right) => (right.relevance_score || 0) - (left.relevance_score || 0) || left.charity_name.localeCompare(right.charity_name)); + } else if (profileSort === "income_desc") { + filtered.sort((left, right) => (right.latest_income || 0) - (left.latest_income || 0) || left.charity_name.localeCompare(right.charity_name)); + } else { + filtered.sort((left, right) => left.charity_name.localeCompare(right.charity_name)); + } const nextPage = filtered.slice(offset, offset + pageSize); setCharities(current => append ? [...current, ...nextPage] : nextPage); setProfileOffset(offset + nextPage.length); setProfilesHaveMore(offset + nextPage.length < filtered.length); - setLoading(false); - setLoadingMoreProfiles(false); + if (requestSequence === directoryRequestSequenceRef.current) { + setDirectoryInitialLoaded(true); + setLoading(false); + setLoadingMoreProfiles(false); + directoryRequestRef.current = null; + } return; } try { - let url = `${API_BASE}/api/charities?limit=${pageSize + 1}&skip=${offset}`; + let url = `${API_BASE}/api/charities?limit=${pageSize + 1}&skip=${offset}&include_score=true&sort=${profileSort}`; if (searchTerm) url += `&search=${encodeURIComponent(searchTerm)}`; url += `&sources=${encodeURIComponent(selectedDataSources.join(","))}`; if (selectedTags.length > 0) { @@ -765,14 +1082,24 @@ export default function App() { if (minGiving > 0) { url += `&min_annual_giving=${minGiving}`; } + const maxGiving = optionalFilterAmount(maxAnnualGivingInput); + if (maxGiving !== null) { + url += `&max_annual_giving=${maxGiving}`; + } const minAvgGrant = AVG_GRANT_SIZE_STEPS[avgGrantSizeIndex]; if (minAvgGrant > 0) { url += `&min_avg_grant_size=${minAvgGrant}`; } + const maxAvgGrant = optionalFilterAmount(maxAvgGrantSizeInput); + if (maxAvgGrant !== null) { + url += `&max_avg_grant_size=${maxAvgGrant}`; + } - const resp = await fetch(url, { credentials: "include" }); + const resp = await fetch(url, { credentials: "include", signal: controller.signal }); + if (requestSequence !== directoryRequestSequenceRef.current) return; if (resp.ok) { const data: Charity[] = await resp.json(); + if (requestSequence !== directoryRequestSequenceRef.current) return; const nextPage = data.slice(0, pageSize); setCharities(current => append ? [...current, ...nextPage] : nextPage); setProfileOffset(offset + nextPage.length); @@ -782,13 +1109,20 @@ export default function App() { setApiError(`Directory request failed (${resp.status}).`); } } catch (e) { + if ((e as Error).name === "AbortError") return; console.error("Failed to fetch charities", e); - setApiError("The organization directory is temporarily unavailable."); + if (requestSequence === directoryRequestSequenceRef.current) { + setApiError("The organization directory is temporarily unavailable."); + } } finally { - if (append) { - setLoadingMoreProfiles(false); - } else { - setLoading(false); + if (requestSequence === directoryRequestSequenceRef.current) { + setDirectoryInitialLoaded(true); + directoryRequestRef.current = null; + if (append) { + setLoadingMoreProfiles(false); + } else { + setLoading(false); + } } } }; @@ -836,41 +1170,137 @@ export default function App() { } }; - const fetchFoundationNews = async (name: string) => { + const saveNewsRun = ( + organization: Charity, + briefing: NewsSummaryPayload, + mode: "live" | "illustrative", + ) => { + const savedAt = new Date().toISOString(); + const run: SavedNewsRun = { + ...briefing, + organizationKey: newsOrganizationKey(organization), + savedAt, + mode, + }; + setNewsRuns(current => [ + run, + ...current.filter(item => item.organizationKey !== run.organizationKey), + ].slice(0, 24)); + }; + + const openSavedNewsRun = (run: SavedNewsRun) => { + setNewsSummary({ + foundation: run.foundation, + summary: run.summary, + sources: run.sources, + searched_weeks: run.searched_weeks, + generated_at: run.generated_at, + }); + setNewsRunMode(run.mode); + setNewsError(null); + setNewsProgressStep(null); + }; + + const toggleSavedNewsRun = (run: SavedNewsRun) => { + if (newsSummary?.generated_at === run.generated_at) { + setNewsSummary(null); + setNewsRunMode(null); + setNewsError(null); + return; + } + openSavedNewsRun(run); + }; + + const fetchFoundationNews = async (organization: Charity) => { + const name = organization.charity_name; setNewsLoading(true); setNewsError(null); setNewsSummary(null); + setNewsProgressStep("discovering"); + setNewsRunMode(null); if (!isBffOnline) { - // Mock news fallback - setTimeout(() => { - setNewsSummary({ + // Keep illustrative mode legible while preserving the same visible stages. + await new Promise(resolve => window.setTimeout(resolve, 350)); + setNewsProgressStep("reading"); + await new Promise(resolve => window.setTimeout(resolve, 500)); + setNewsProgressStep("summarizing"); + await new Promise(resolve => window.setTimeout(resolve, 550)); + const briefing: NewsSummaryPayload = { foundation: name, summary: `Here is a mock summary of recent news for "${name}". The foundation has been actively expanding its socio-economic support programs in the UK. They announced a new partnership with local food banks to address food insecurity. Furthermore, they are investing in digital transformation initiatives to streamline grant-making processes for small charities.`, sources: [ { title: "Netlight News: Insecurity Partnership", link: "https://example.com/news1", source: "Netlight Post", published: "Mon, 20 Jul 2026 10:00:00 GMT" }, { title: "Charity Digital: Streamlining Grants", link: "https://example.com/news2", source: "Charity Daily", published: "Sun, 19 Jul 2026 14:30:00 GMT" } - ] - }); - setNewsLoading(false); - }, 1500); + ], + searched_weeks: 4, + generated_at: new Date().toISOString(), + }; + setNewsSummary(briefing); + setNewsRunMode("illustrative"); + saveNewsRun(organization, briefing, "illustrative"); + setNewsProgressStep(null); + setNewsLoading(false); return; } try { - const resp = await fetch(`${API_BASE}/api/news/${encodeURIComponent(name)}/summary`, { credentials: "include" }); - if (resp.ok) { - const data = await resp.json(); - setNewsSummary(data); - } else { - const errorDetail = await resp.json(); - setNewsError(errorDetail.detail || "Failed to load news summary from backend."); + const resp = await fetch(`${API_BASE}/api/news/${encodeURIComponent(name)}/summary/stream`, { + credentials: "include", + headers: { Accept: "text/event-stream" }, + }); + if (!resp.ok || !resp.body) { + const errorDetail = await resp.json().catch(() => null); + throw new Error(errorDetail?.detail || `News research request failed (${resp.status}).`); } + const reader = resp.body.getReader(); + const decoder = new TextDecoder(); + let buffer = ""; + let completed = false; + const handleEvent = (message: string) => { + const event = message.match(/^event:\s*(.+)$/m)?.[1]?.trim(); + const serialized = message.match(/^data:\s*(.+)$/m)?.[1]; + if (!event || !serialized) return; + const payload = JSON.parse(serialized) as Record; + if (event === "progress") { + const step = payload.step; + if (step === "discovering" || step === "reading" || step === "summarizing") setNewsProgressStep(step); + return; + } + if (event === "error") throw new Error(typeof payload.detail === "string" ? payload.detail : "News research could not be completed."); + if (event === "complete") { + if (typeof payload.foundation !== "string" || typeof payload.summary !== "string" || !Array.isArray(payload.sources)) { + throw new Error("News research returned an incomplete briefing."); + } + const briefing: NewsSummaryPayload = { + foundation: payload.foundation, + summary: payload.summary, + sources: payload.sources as NewsSourceItem[], + searched_weeks: typeof payload.searched_weeks === "number" ? payload.searched_weeks : 4, + generated_at: typeof payload.generated_at === "string" ? payload.generated_at : new Date().toISOString(), + }; + setNewsSummary(briefing); + setNewsRunMode("live"); + saveNewsRun(organization, briefing, "live"); + completed = true; + } + }; + while (true) { + const { value, done } = await reader.read(); + buffer += decoder.decode(value || new Uint8Array(), { stream: !done }); + const messages = buffer.split("\n\n"); + buffer = messages.pop() || ""; + messages.forEach(handleEvent); + if (done) break; + } + if (buffer.trim()) handleEvent(buffer); + if (!completed) throw new Error("News research ended before a briefing was returned."); } catch (e: any) { console.error("Failed to fetch news summary", e); - setNewsError("An error occurred while connecting to the news service."); + setNewsError(e?.message || "An error occurred while connecting to the news service."); } finally { setNewsLoading(false); + setNewsProgressStep(null); } }; @@ -926,6 +1356,34 @@ export default function App() { } }; + const navigateApplication = ( + view: "overview" | "donors" | "research" | "registry" | "favorites" | "pipeline", + mode: "push" | "replace" = "push", + ) => { + setDonorReturnToFavorites(false); + const query = new URLSearchParams(window.location.search); + if (view === "overview") { + query.delete("view"); + ["funder_country", "funder_sort", "funder_page", "donor_search", "donor_status", "donor"].forEach(key => query.delete(key)); + setActiveTab("overview"); + } else { + query.set("view", view); + if (view === "pipeline") { + setActiveTab("admin"); + } else if (view === "favorites") { + setActiveTab("favorites"); + } else { + setActiveTab("directory"); + setDirectoryMode(view === "research" ? "profiles" : view === "registry" ? "registry" : "donors"); + } + } + const suffix = query.toString(); + window.history[mode === "push" ? "pushState" : "replaceState"]( + {}, "", `${window.location.pathname}${suffix ? `?${suffix}` : ""}`, + ); + setMobileNavigationOpen(false); + }; + const openOrganizationDirectoryFromMap = (filters: GrantMapFilters) => { setSearchTerm(filters.search); setSelectedTags(filters.tags); @@ -939,11 +1397,67 @@ export default function App() { beneficiaryGeography: filters.fundingRegions.length === 1 ? filters.fundingRegions[0] : "", })); setSelectedCharity(null); - setDirectoryMode("profiles"); + navigateApplication("research"); + window.scrollTo({ top: 0, behavior: "smooth" }); + }; + + const openSourceFundersFromMap = ( + selection: SourceFunderCountrySelection, + grantFilters: OverviewFilters, + ) => { + const query = applyGrantScopeToParams( + new URLSearchParams(window.location.search), + { + beneficiaryCountry: selection.countryCode, + currency: grantFilters.currency || undefined, + dateFrom: grantFilters.dateFrom || undefined, + dateTo: grantFilters.dateTo || undefined, + beneficiaryGeographies: grantFilters.beneficiaryGeographies, + programmeAreas: grantFilters.programmeAreas, + donor: grantFilters.donor, + recipient: grantFilters.recipient, + sources: selectedDataSources, + }, + { persistEmptySources: true }, + ); + query.delete("funder_sort"); + query.delete("funder_page"); + query.delete("donor_search"); + query.delete("donor_status"); + query.delete("donor"); + if (grantFilters.granularity === "auto") query.delete("grant_granularity"); + else query.set("grant_granularity", grantFilters.granularity); + query.set("view", "donors"); + window.history.pushState({}, "", `${window.location.pathname}?${query.toString()}`); + setSelectedCharity(null); + setDirectoryMode("donors"); setActiveTab("directory"); window.scrollTo({ top: 0, behavior: "smooth" }); }; + const openLinkedDirectoryProfile = (profile: { charity_id: number; name: string | null }) => { + setSelectedCharity({ + registered_charity_number: profile.charity_id, + suffix: 0, + link: "", + charity_name: profile.name || "Linked organization", + reg_status: "UNKNOWN", + reporting_status: "", + removal_reason: null, + latest_income: null, + latest_expenditure: null, + programme_areas_source: [], + programme_areas_inferred: [], + geographic_focus_source: [], + geographic_focus_inferred: [], + organization_type: "unknown", + source_names: [], + transaction_coverage: "unknown", + }); + navigateApplication("research"); + window.scrollTo({ top: 0, behavior: "smooth" }); + }; + const resetDirectoryFilters = () => { setSearchTerm(""); setOrganizationSuggestions([]); @@ -952,11 +1466,156 @@ export default function App() { setSelectedFoundationRegions([]); setSelectedRecipientRegions([]); setAnnualGivingIndex(0); + setMaxAnnualGivingInput(""); setAvgGrantSizeIndex(0); + setMaxAvgGrantSizeInput(""); setProfileOffset(0); setProfilesHaveMore(false); }; + const profileFavoriteKey = (profile: Charity) => + `profile:${profile.registered_charity_number}:${profile.source_record_id || ""}`; + + const toggleFavoriteProfile = (profile: Charity) => { + const key = profileFavoriteKey(profile); + setFavorites(current => current.profiles.some(item => item.key === key) + ? { ...current, profiles: current.profiles.filter(item => item.key !== key) } + : { ...current, profiles: [{ key, profile, savedAt: Date.now() }, ...current.profiles] }); + }; + + const toggleFavoriteDonor = (donor: FavoriteDonorPayload) => { + setFavorites(current => current.donors.some(item => item.key === donor.key) + ? { ...current, donors: current.donors.filter(item => item.key !== donor.key) } + : { ...current, donors: [{ ...donor, savedAt: Date.now() }, ...current.donors] }); + }; + + const currentResearchFavorite = (): FavoriteResearchView => { + const filters: FavoriteResearchView["filters"] = { + searchTerm, + selectedTags, + selectedFoundationRegions, + selectedRecipientRegions, + annualGivingIndex, + maxAnnualGivingInput, + avgGrantSizeIndex, + maxAvgGrantSizeInput, + profileSort, + }; + const activeTerms = [ + searchTerm.trim(), + selectedTags[0], + selectedFoundationRegions[0], + selectedRecipientRegions[0], + ].filter(Boolean); + return { + key: `research:${JSON.stringify(filters)}`, + label: activeTerms.length ? `Organization Research · ${activeTerms.join(" · ")}` : "Organization Research", + filters, + savedAt: Date.now(), + }; + }; + + const researchFavorite = currentResearchFavorite(); + const researchViewIsFavorite = favorites.researchViews.some(view => view.key === researchFavorite.key); + + const toggleCurrentResearchFavorite = () => { + const view = currentResearchFavorite(); + setFavorites(current => current.researchViews.some(item => item.key === view.key) + ? { ...current, researchViews: current.researchViews.filter(item => item.key !== view.key) } + : { ...current, researchViews: [{ ...view, savedAt: Date.now() }, ...current.researchViews] }); + }; + + const removeFavoriteResearch = (key: string) => { + setFavorites(current => ({ ...current, researchViews: current.researchViews.filter(item => item.key !== key) })); + }; + + const landscapeFavorite = (filters: OverviewFilters): FavoriteLandscapeView => { + const query = applyGrantScopeToParams( + new URLSearchParams(), + { + currency: filters.currency || undefined, + dateFrom: filters.dateFrom || undefined, + dateTo: filters.dateTo || undefined, + beneficiaryGeographies: filters.beneficiaryGeographies, + programmeAreas: filters.programmeAreas, + donor: filters.donor || undefined, + recipient: filters.recipient || undefined, + sources: selectedDataSources, + }, + { persistEmptySources: true }, + ); + if (filters.granularity !== "auto") query.set("grant_granularity", filters.granularity); + const route = query.toString() ? `?${query.toString()}` : ""; + const activeTerms = [filters.donor, filters.recipient, filters.beneficiaryGeographies[0], filters.programmeAreas[0]].filter(Boolean); + return { + key: `landscape:${route}`, + label: activeTerms.length ? `Funding Landscape · ${activeTerms.join(" · ")}` : "Funding Landscape", + route, + savedAt: Date.now(), + }; + }; + + const isFavoriteLandscape = (filters: OverviewFilters) => + favorites.landscapeViews.some(view => view.key === landscapeFavorite(filters).key); + + const toggleFavoriteLandscape = (filters: OverviewFilters) => { + const view = landscapeFavorite(filters); + setFavorites(current => current.landscapeViews.some(item => item.key === view.key) + ? { ...current, landscapeViews: current.landscapeViews.filter(item => item.key !== view.key) } + : { ...current, landscapeViews: [{ ...view, savedAt: Date.now() }, ...current.landscapeViews] }); + }; + + const openFavoriteProfile = (favorite: FavoriteProfile) => { + setSelectedCharity(favorite.profile); + }; + + const restoreSourcesFromFavoriteRoute = (route: string) => { + const params = new URLSearchParams(route.startsWith("?") ? route.slice(1) : route); + if (!params.has("grant_sources")) return; + const selected = new Set((params.get("grant_sources") || "").split(",").filter(Boolean)); + setDataSourceSelections(Object.fromEntries(dataSourceNames.map(source => [source, selected.has(source)]))); + }; + + const openFavoriteDonor = (favorite: FavoriteDonorPayload) => { + restoreSourcesFromFavoriteRoute(favorite.route); + const query = new URLSearchParams(favorite.route.startsWith("?") ? favorite.route.slice(1) : favorite.route); + query.set("view", "donors"); + query.set("donor", favorite.key); + window.history.pushState({}, "", `${window.location.pathname}?${query.toString()}`); + setSelectedCharity(null); + setDonorReturnToFavorites(true); + setDirectoryMode("donors"); + setActiveTab("directory"); + setMobileNavigationOpen(false); + window.scrollTo({ top: 0, behavior: "smooth" }); + }; + + const openFavoriteResearch = (favorite: FavoriteResearchView) => { + const { filters } = favorite; + setSearchTerm(filters.searchTerm); + setSelectedTags(filters.selectedTags); + setSelectedFoundationRegions(filters.selectedFoundationRegions); + setSelectedRecipientRegions(filters.selectedRecipientRegions); + setAnnualGivingIndex(filters.annualGivingIndex); + setMaxAnnualGivingInput(filters.maxAnnualGivingInput); + setAvgGrantSizeIndex(filters.avgGrantSizeIndex); + setMaxAvgGrantSizeInput(filters.maxAvgGrantSizeInput); + setProfileSort(filters.profileSort); + setProfileOffset(0); + setSelectedCharity(null); + navigateApplication("research"); + window.scrollTo({ top: 0, behavior: "smooth" }); + }; + + const openFavoriteLandscape = (favorite: FavoriteLandscapeView) => { + restoreSourcesFromFavoriteRoute(favorite.route); + window.history.pushState({}, "", `${window.location.pathname}${favorite.route}`); + setSelectedCharity(null); + setActiveTab("overview"); + setMobileNavigationOpen(false); + window.scrollTo({ top: 0, behavior: "smooth" }); + }; + const fetchGrantAnalytics = async (forceOnline?: boolean, currencyOverride?: string) => { const isOnline = forceOnline !== undefined ? forceOnline : isBffOnline; if (!isOnline) { @@ -1049,12 +1708,13 @@ export default function App() { const formattedLinks = data.links.map((l: any) => ({ source: nodeMap.get(l.source), target: nodeMap.get(l.target), - value: l.value + value: l.value, + grantCount: l.grant_count, })); setSankeyData({ status: data.status, - nodes: data.nodes.map((n: any) => ({ name: n.label })), + nodes: data.nodes.map((n: any) => ({ id: n.id, name: n.label, role: n.role })), links: formattedLinks, currency: data.metadata?.selected_currency || null, excludedCount: data.metadata?.excluded_grant_count || 0, @@ -1199,6 +1859,56 @@ export default function App() { latestAnalyticsRefresh && Date.now() - Date.parse(latestAnalyticsRefresh) > 30 * 24 * 60 * 60 * 1000 ); + const profileFilterCount = Number(Boolean(searchTerm.trim())) + + selectedTags.length + + selectedFoundationRegions.length + + selectedRecipientRegions.length + + Number(annualGivingIndex > 0) + + Number(Boolean(maxAnnualGivingInput.trim())) + + Number(avgGrantSizeIndex > 0) + + Number(Boolean(maxAvgGrantSizeInput.trim())); + const activeHeaderState: HeaderContextState = activeTab === "overview" + ? { + filterCount: overviewFilterCount, + resetDisabled: overviewFilterCount === 0, + filtersExpanded: overviewFiltersOpen, + } + : activeTab === "directory" && directoryMode === "donors" + ? donorHeaderState + : activeTab === "directory" && directoryMode === "registry" + ? registryHeaderState + : activeTab === "directory" && directoryMode === "profiles" + ? { + filterCount: profileFilterCount, + resetDisabled: profileFilterCount === 0, + filtersExpanded: profileFiltersOpen, + } + : { filterCount: 0, resetDisabled: true, filtersExpanded: false }; + + const openContextFilters = () => { + if (activeTab === "overview") { + window.dispatchEvent(new Event("overview-open-filters")); + } else if (activeTab === "directory" && directoryMode === "donors") { + window.dispatchEvent(new Event("donor-directory-open-filters")); + } else if (activeTab === "directory" && directoryMode === "profiles") { + setProfileFiltersOpen(true); + } else if (activeTab === "directory" && directoryMode === "registry") { + window.dispatchEvent(new Event("registry-open-filters")); + } + }; + + const resetContext = () => { + if (activeTab === "overview") { + window.dispatchEvent(new Event("overview-reset-filters")); + } else if (activeTab === "directory" && directoryMode === "donors") { + window.dispatchEvent(new Event("donor-directory-reset")); + } else if (activeTab === "directory" && directoryMode === "profiles") { + resetDirectoryFilters(); + setSelectedCharity(null); + } else if (activeTab === "directory" && directoryMode === "registry") { + window.dispatchEvent(new Event("registry-reset-filters")); + } + }; if (initialLoading) { return ( @@ -1290,7 +2000,7 @@ export default function App() {