Lobbying explorer frontend#2182
Draft
nesanders wants to merge 18 commits into
Draft
Conversation
Scrapes the MA Secretary of State lobbying portal (sec.state.ma.us/LobbyistPublicSearch)
and writes structured data to Firestore for joining with MAPLE bill data.
New collections:
- /lobbyingRegistrants — one doc per (registrant, year), regType Lobbyist|Employer
- /lobbyingFilings — one doc per (registrant, client, bill, court), with billId
null for Executive/Other chambers so the join guard is type-level
Key design points:
- billId is constructed as {chamberPrefix}{integer} (e.g. H1234, SD56) to match
Bill.id in the existing bills collection; raw integer + chamber stored separately
- Entity name normalization pipeline ported from reference implementation (10 steps:
d/b/a stripping, legal entity words, punctuation, THE, ampersand, typo fix, etc.)
- Both raw and *Norm name fields stored for provenance and grouping
- Live Cloud Function scrapes current+prior year on a 24h schedule with a
summaryDiscCache to avoid re-fetching summary pages in steady state
- Backfill admin script handles full 2005-present history with a Firestore
subcollection cursor (/scrapers/lobbyingBackfill/processedUrls) that scales
to ~50k URLs and is safely resumable
Files:
- functions/src/lobbying/{types,normalize,portal,scrapeLobbying,index}.ts
- scripts/firebase-admin/backfillLobbying.ts
- firestore.rules + firestore.indexes.json updated
- docs/lobbying-disclosure-ingestion.md: full plan, test plan, future work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The MA SoS portal is protected by Imperva WAF, which uses TLS fingerprinting to classify HTTP clients before examining headers. Python's requests library produces a fingerprint that Imperva allows through; Node.js does not. A standalone Cloud Run container (Python 3.12) is therefore used for the scheduled ingestion instead of a Cloud Function. lobbying-scraper/ — Cloud Run container (3 pip deps: requests, beautifulsoup4, google-cloud-firestore): - scrape.py: entry point with --mode weekly (incremental, fast exit if nothing new) and --mode backfill (full 2005-present history, resumable subcollection cursor). Weekly mode caches summary URL→disc URL mappings so prior-year registrants with no new filings require zero additional HTTP requests. - portal.py: HTTP session management + HTML parsing for all three portal page levels (search POST, summary GET, disclosure GET). Handles both modern (>=2013) and legacy (<2013) disclosure formats. - normalize.py: port of functions/src/lobbying/normalize.ts — 10-step entity name normalization pipeline, must match the TypeScript version exactly. - writer.py: Firestore document construction and batch writes. Schema matches types.ts (lobbyingRegistrants, lobbyingFilings collections). scripts/firebase-admin/backfillLobbying.ts — simplified to spawn scrape.py as a subprocess; all HTTP and Firestore logic moved to Python. functions/src/lobbying/http/ — thin Python HTTP helper kept for reference; not used in the current architecture. Note: server-side IP reputation behavior with Imperva untested. Build and run the container on Cloud Run with --dry-run to validate before full deploy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Per code review feedback: the TypeScript Firebase Function and backfill script added no value — the portal's TLS fingerprinting requirements mean Node.js cannot reach it, so the TS HTTP layer was non-functional and the backfill script was just a thin subprocess wrapper with no benefit over calling scrape.py directly. Removed: - functions/src/lobbying/scrapeLobbying.ts (broken Cloud Function) - functions/src/lobbying/portal.ts (non-functional TS HTTP layer) - functions/src/lobbying/http/ (unused Python fetch helper) - scripts/firebase-admin/backfillLobbying.ts (shell wrapper, no value) - scrapeLobbying export from functions/src/index.ts Kept: - functions/src/lobbying/types.ts — Firestore schema; imported by frontend - functions/src/lobbying/normalize.ts — normalization pipeline - lobbying-scraper/ — the working Cloud Run container (unchanged) The historical backfill is now run directly: python3 lobbying-scraper/scrape.py --mode backfill Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ucture
Portal parser (portal.py):
- Hybrid era (2014-2018): per-client compensation from Panel1 divs — was
silently $0 due to missing code path (e.g. Murphy Donoghue 2016: $990k)
- Legacy era (2009-2013): per-client totals from 'Compensation received'
column, with dedup of (client, amount) pairs before summing — was silently
$0 per client (e.g. ML Strategies 2011: $641k across 23 clients)
- Legacy bill semicolon separator: 'H73; Title' now parsed correctly
- 'Total amount' summary row excluded from compensation pairs
- HTTP retry on 429/500/502/503/504 (was aborting on first transient error)
- parse_summary() and parse_disclosure_detail() split out as pure functions
(no I/O) so the offline reparse driver can call them without a session
GCS archiving (archive.py):
- Write-only cold storage: every fetched Summary/CompleteDisclosure page
saved as gs://{project}-lobbying-archive/raw_html/{sha1(url)}.html
- Enabled by ARCHIVE_RAW=1 env var; no-op otherwise
- Failures are logged but never interrupt the live scrape path
Offline reparse driver (reparse_archive.py):
- Lists CompleteDisclosure blobs from GCS, resolves registrant meta from
Firestore via disclosureUrls array_contains, re-runs pure parsers,
writes back via writer.py; resumable via /scrapers/lobbyingReparse cursor
Pytest suite (tests/test_portal_parser.py, 26 tests):
- All 4 eras verified against committed gzipped fixture pages
- Compensation totals, client counts, bill counts, era detection asserted
- Specific bug-fix regressions: Total-amount artifact, H73 semicolon,
hybrid Panel1 compensation, 2007 _total_salary_ fallback
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- writer.py: move firestore import out of TYPE_CHECKING block so
firestore.ArrayUnion() is available at runtime (was NameError)
- writer.py/scrape.py/reparse_archive.py: strip leading slash from
Firestore path constants (SCRAPER_DOC, BACKFILL_DOC, REPARSE_DOC) —
db.document('/scrapers/x') raises ValueError: odd path element count
- scrape.py: add os import; pass GOOGLE_CLOUD_PROJECT to firestore.Client()
so local runs target the correct project rather than the ADC default
Verified: 3 registrants / 6 disclosures written to digital-testimony-dev;
re-run writes 0 (cursor working).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… 4 HTML eras - Replace TypeScript/ts-node test plan steps with Python equivalents - Document all 4 CompleteDisclosure HTML format eras (2005-2008, 2009-2013, 2014-2018, 2019+) including parser behavior and known quirks - Update Historical Backfill section with GOOGLE_CLOUD_PROJECT and ARCHIVE_RAW=1; remove local environment references - Add partial backfill pattern (--limit 50 across all years) as large-scale dev validation step - Remove stale Function Export section referencing deleted scrapeLobbying function - Update Step 8 (deploy) from Firebase Function commands to Cloud Run job commands - Fix cursor path prose to remove leading slashes; fix jsdom → beautifulsoup4 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Embed actual registrant and filing documents from dev Firestore to illustrate the data model concretely for reviewers - Correct registrantId description from "slugified" to SHA-256 hash (matches the actual implementation in writer.py) - filingId description likewise updated to SHA-256 hash Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extend LobbyingFiling type with registrantId; add LobbyingStats, LobbyingClientSummary, LobbyingPositionCounts aggregate types using Dictionary for string-keyed maps (runtypes v5 compat) - Add lobbyingMeta/stats singleton collection constants; add CLIENTS_COLLECTION - Add Firestore composite indexes for registrantId+year and clientNameNorm+year (DESCENDING) on lobbyingFilings to support firm and client detail queries - Add data hooks (useLobbyingFilingsForBill, useLobbyingRegistrant, etc.) in components/db/lobbying.ts following existing useAsync pattern - Add lobbying.json locale namespace; add navigation.lobbying to common.json - Add NavbarLinkLobbying component; wire into Navbar behind lobbyingTable flag - Add LobbyingPositionChip (support/oppose/neutral/none pill), LobbyingFilingsTable (configurable columns), LobbyingBillCard (sidebar card with position stacked bar, truncated table, explorer link) - Replace dummy LobbyingTable in BillDetails with LobbyingBillCard - Enable lobbyingTable feature flag in development env - Install recharts (v3); update chartTheme.tsx for v3 TooltipContentProps API Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add /lobbying/bills browse page: session dropdown, position filter, bill ID search, table with per-bill filing counts and mini position bar - Add stub pages for /lobbying, /lobbying/clients, /lobbying/firms and dynamic routes for client/firm detail - Add useLobbyingFilingsForCourt hook for court-level data loading - Fix LobbyingBillCard: surface errors in dev, handle not-requested state, inline collection constants to avoid pulling firebase-admin into browser bundle (was causing net module build error on bill detail page) - Add lobbying namespace to bill detail page getServerSideProps so LobbyingBillCard translations load correctly - Add titles.lobbying key to common.json for page title Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements four new routes for the lobbying explorer: - /lobbying/firms — browse all registrant firms, filter by type/name - /lobbying/firms/[registrantId] — firm detail with filings table, client list, disclosure links - /lobbying/clients — browse unique clients derived from registrant docs - /lobbying/clients/[clientSlug] — client detail with filings table, linked firms, position summary Firm and client slugs use URL-encoded entityNameNorm/clientNameNorm. Both detail pages use getStaticPaths with fallback:blocking. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Skips clientNameNorm values that are empty or equal to _total_salary_, which is used for pre-2013 legacy filings where compensation is reported as a single total rather than per client. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Build out /lobbying overview: stat cards (bills, clients, sessions, spend), dual-axis spend+filings chart, entry cards linking to browse pages - Extract LobbyingAttribution component with correct SoS link - Fix broken attribution on bills page (was passing Trans interpolation syntax to t() which rendered literal <0> tags) - Stats and chart degrade gracefully when the pipeline stats doc is absent Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add LobbyingAttribution to firms browse/detail and clients browse/detail - Add seedLobbyingStats admin script: reads all filings and registrants, computes totals and per-year breakdowns, writes to lobbyingMeta/stats (run with: yarn firebase-admin run-script seedLobbyingStats --env dev) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add allow read: if true for lobbyingMeta collection so the stats doc is readable by the frontend (was silently returning undefined) - Move Lobbying nav item after Testimony in both mobile and desktop nav Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…il to lobbying explorer - Add LobbyingSubnav (Overview / Bills / Clients / Lobbying Firms) to all lobbying pages - Add usePagination hook and LobbyingPaginationBar component (50/page on all browse indexes) - Add sortable column headers (SortTh) to bills, clients, and firms index pages - Move bills.tsx → bills/index.tsx; add bills/[court]/[billId].tsx paginated filings detail - Filter _total_salary_ sentinel by both clientName and clientNameNorm in clients browse - Fix duplicate "View all lobbying activity" link in LobbyingBillCard - Fix undefined–undefined year range on firm detail when registrant query fails - Add composite Firestore indexes for entityNameNorm+year and clientNameNorm+year queries - Add showActivity prop to LobbyingFilingsTable Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nesanders
requested review from
Mephistic,
alexjball,
kiminkim724,
mertbagt,
mvictor55,
sashamaryl and
timblais
as code owners
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
nesanders
changed the title
…onsive column hiding - Wire t() through all lobbying components: LobbyingSubnav, LobbyingPaginationBar, LobbyingFilingsTable, and all browse/detail pages - Add missing translation keys: subnav, pagination, misc, registrantType, and new fields (bill, positions, activity, sessions, type, firms, total) - Add lobbying.module.css with media query to hide table columns on mobile - Bills index: hide Session and position count columns on mobile (<768px), leaving Bill | Filings | Positions mini-bar - Bill detail: hide Activity column on mobile to reduce table width - Client and firm detail pages: paginate filings table at 25/page (year desc) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a public lobbying explorer to MAPLE, surfacing MA Secretary of State lobbyist disclosure data with interactive browse and detail pages. This is a foundation of a frontend to be updated to align to future design work.
Depends on #2158
New pages
/lobbying— overview with aggregate stats and a filing-count-by-year chart/lobbying/bills— paginated bills index, filterable by session/position/search, sortable by filing count or position counts/lobbying/bills/[court]/[billId]— full paginated filings table for a specific bill (client, firm, activity, position, amount, year)/lobbying/clients— paginated, searchable clients index sortable by name, compensation, or firm count/lobbying/clients/[clientSlug]— client detail/lobbying/firms— paginated, searchable firms index sortable by name, clients, or sessions/lobbying/firms/[registrantId]— firm detailShared components
LobbyingSubnav— "LOBBYING EXPLORER" label + Overview / Bills / Clients / Lobbying Firms nav, active-link highlighted, spans all lobbying pagesusePaginationhook +LobbyingPaginationBar— 50 rows/page on all index pagesBug fixes
_total_salary_sentinel rows from clients browse (artifact of old SoS publication format)entityNameNorm+yearandclientNameNorm+yearqueriesChecklist
firestore.indexes.json.Screenshots
NOTE: not all data is backfilled yet from #2158, so data shown below is partial.
Lobbyin explorer landing page:
Individual index pages:
Mobile view of bill index page (hiding columns automatically)
Embed in main bill page:
Lobbying bill detail page:
Client detail page:
Lobbyist detail page:
Known issues
lobbyingi18n namespace; English-only for now.Steps to test/reproduce
/lobbying— overview stats and chart should load/lobbying/bills— change session, filter by position, search by bill ID, click column headers to sort/lobbying/bills/194/H4001— filings table with client/firm/activity/position/amount/year; position filter and sort work/lobbying/clients— search and sort/lobbying/firms— type filter, search, sort by clients/sessions