Election scraping

[delexagon]

Summary

Backend pipeline for #2130.

Known issues

I strongly recommend looking over the data definitions in functions/src/legislators/electionTypes.ts.
If there are more than 1200 elections in a search (in our case, more than 1200 general elections in a year), they will be clipped off at 1200. Currently, the scraper does not catch this case.
The id is a hash of the data. If we want to normalize the district data to a standard format, this would change all the ids.
We should probably consider how we are going to want to connect the candidate url used as a key here to the legislator elsewhere.
May want to discuss whether there are a possibility of additional 'types' of votes like no preference, which may be rare but existing.
Does not currently check if all the votes add up to the total, which might be desirable.

Steps to test/reproduce

1. Run curl 'http://localhost:5001/demo-dtp/us-central1/triggerPubsubFunction?scheduled=scrapeElections'. This should fetch all elections for this year (and last year if before July).
2. Run yarn firebase-admin run-script backfillElections --env local --startYear 2024. This should fetch every general election from 2024.
3. You may want to try fetchElectionsData in a variety of cases, for example when there are too many candidates (which requires a secondary page fetch), when there are no candidates, when there are 'No preference' votes, when there are write in candidates, etc. I've been using npx ts-node --compiler-options '{"module":"CommonJS"}' functions/src/legislators/scrapeElections.ts with

import { writeFileSync } from "fs"
(async () => {
  const data = await fetchElectionsData()
  writeFileSync("/tmp/election-results.json", JSON.stringify(data, null, 2))
})()

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
maple-dev	Ready	Preview, Comment	Jun 28, 2026 3:22am

[Mephistic]

Love this! Just a few minor nits.

As per your "Known Issues":

• Types look solid to me
• Oof... 1200 is such an unfortunate cutoff when there were ~1260 elections in 2024. I think it's fine for the first pass here to limit this to the elections we care most about: State House and State Senate general elections. I think we can view both non-state legislative races and primaries as "nice-to-have"s for some future PR. (If we could get only one in this PR, I'd say let's prefer to include primaries - that is at least directly relevant to specific legislators - but neither are required in the design for the Elections tab)
• At first glance, it seems that elections have unique ids on the specific election pages (e.g. https://electionstats.state.ma.us/elections/view/165440/ - 165440 looks like a unique id. Could we use that to be more resilient to data formatting changes?
• Connecting candidate and legislator is important, but I'm totally fine calling that a problem for a follow-up PR
• I think we're fine to just handle the obvious vote types in the first pass - "no preference" can wait
• Hmm... that might be interesting, but I think we can hold off on an explicit check here - IMO that seems more likely to get us mired in the details of those uncommon additional vote types than to give us robust security.

[Mephistic]

Would this cause errors in the scraper to be swallowed rather than propagated to the Error Reporting service? For context, functions.logger.error calls and uncaught exceptions will be propagated to Error Reporting. (The same note applies to all error logging in this file).

(Not sure how often we expect to see the "Could not parse CSS stylesheet" error - is that expected?)

[Mephistic]

This is some pretty detailed parsing - could we pull down an example of both the Elections page and the single Election page that we could use as fixtures for unit tests (to make sure we don't break this on our end if we need to make changes in the future?).