Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,9 +269,8 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve
- `ModelDeduper` — Deduplication logic. On a natural-key collision it now merges the losing row into the survivor (recursively, via itself) so the loser's own children move first — instead of destroying the loser and cascading them away or aborting on a protective `before_destroy` (e.g. a professional license with CE registrations). It finds reassignable children two ways: the model's own `has_many` declarations, and a scan of the DB's foreign keys. **A polymorphic child carries no foreign key**, so it is reachable only through an explicit `has_many … as:` — without one, `#unhandled_references` sees it and the merge is refused outright (this is why `Person`/`Organization` declare `has_many :invoices, as: :invoicee`). A config can also pass `reference_columns:` — column names that point at the model in any table without a DB foreign key (the users deduper names `user_id`/`created_by_id`/`updated_by_id`/`welcome_instructions_sent_by_id`). Self-references on the model's own table follow the keeper, except the keeper's own reference to the duplicate, which clears. The preview lists each moving record by its decorator's optional `#dedupe_label` (Affiliation, ContactMethod, Comment, Notification define one; text snippets cap at `SNIPPET_LENGTH`)
- `ContinuingEducationDeduper` — Collapses duplicate CE registrations left sharing one event registration + license after a merge, moving the losers' allocations (payments) and comments onto the survivor. Keeps the real cost (larger of the two) rather than inventing one to cover the combined payments — if they now exceed it, `ContinuingEducationRegistration#over_allocated?` flags it (`ContinuingEducationRegistrationDecorator#overpayment_badge` → "Check payments") for an admin. CE has no DB unique index, so `ModelDeduper` never compares CE rows and just moves them all onto the keeper; a unique index is deliberately avoided because it would force the engine to move CE via validated `update!`, which trips `ContinuingEducationRegistration#license_belongs_to_registrant` mid-merge. Run from the people deduper's `after_merge`
- `AddressNormalizer` — Comparison keys for address parts so spellings of one place match: street suffixes/directionals/ordinals (`Street`→`st`, `North`→`n`, `First`→`1st`), unit designators (`Suite`/`Apt`/`Unit`/`#` → `#`), `Saint`/`Mount`/`Fort` in cities, state names → USPS abbreviations, ZIP+4 → five digits, US country spellings (and a blank country in a US state) → `us`, a trailing "County", and an "unknown" type/locality → blank. `.signature(address)` is the whole-address key. Matching only — stored addresses keep their spelling. Used by `AddressDeduper`, `OrganizationServices::AddressMatcher`, `OrganizationServices::DuplicateFinder`'s shared-address signal, and `PersonServices::CaptureFromSubmission`'s mailing-address match
- `AddressTranslation` — Turns imported and legacy address values into what the address dropdowns store (ADR-0009): `#country(value, state:)` → a `GeographyHelper::COUNTRIES` name (aliases for our data's spellings, ISO codes, plain names, a country read from the state — a US state incl. "CA - California", a `REGIONS` province, or a country named as the state; placeholders like "-"/"n/a"/"unknown" → nil; "CA" next to state CA → US), `#state(value, country:)` → USPS code for a US state (other regions kept), `#address_type` (the archive's/forms' words via `CONTACT_TYPES`), `#locality` (`AddressLocality` on the translated parts), and `#address(attrs)` for a whole hash. Lookup tables only, never guessed (like `TagTranslation`): a country nothing answers to is cleared, so every stored country is a dropdown name or blank, and counted in `#unmatched`; an unreadable state is kept and counted. Used by `FmAddressAttributes` (every FileMaker import), `EventRegistrationImporter` (country/state answers + address/phone types; unmatched values listed on the import preview and counted in its notice), and `AddressServices::TranslateExisting`. No save hook — the dropdowns own new data
- `AddressTranslation` — Turns imported and legacy address values into what the address dropdowns store (ADR-0009): `#country(value, state:)` → a `GeographyHelper::COUNTRIES` name (aliases for our data's spellings, ISO codes, plain names, a country read from the state — a US state incl. "CA - California", a `REGIONS` province, or a country named as the state; placeholders like "-"/"n/a"/"unknown" → nil; "CA" next to state CA → US), `#state(value, country:)` → USPS code for a US state (other regions kept), `#address_type` (the archive's/forms' words via `CONTACT_TYPES`), `#locality` (`AddressLocality` on the translated parts), and `#address(attrs)` for a whole hash. Lookup tables only, never guessed (like `TagTranslation`): a country nothing answers to is cleared, so every stored country is a dropdown name or blank, and counted in `#unmatched`; an unreadable state is kept and counted. Used by `FmAddressAttributes` (every FileMaker import) and `EventRegistrationImporter` (country/state answers + address/phone types; unmatched values listed on the import preview and counted in its notice). No save hook — the dropdowns own new data
- `FmAddressAttributes` — Turns an `FmAddress` row into `Address` attributes for any FileMaker import (street from `StreetNo` + lines, ZIP or its `PostalCodeLookup`, county from the archive's ZIP table read once, everything clipped to 255), with state/country/type/locality through `AddressTranslation`; nil when the row lacks a street, city or state. An `address_type:` overrides the row's (an organization's addresses are work); `#place(city:, state:, county:)` builds a street-less address for a place the archive records by city alone (a project). `#unmatched` gathers what it couldn't translate. Used by `PersonServices::ImportFromRolodex` (listed in `fm:import_people`'s report) and the FileMaker organizations/affiliations imports (ADR-0008)
- `AddressServices::TranslateExisting` — Rewrites stored `Address` countries/states and the `mailing_*`/`organization_*` country/state `FormAnswer`s through `AddressTranslation` (each answer read beside its paired state/country answer), fills a locality only where it's Unknown, clears (address) or blanks (answer) a country nothing translates, and logs each changed record as one `autochange.address`/`autochange.form_answer` Ahoy event (all in one visit, `reason: "address_translation"`). Idempotent; `dry_run:` reports without writing. Drives the `data:translate_addresses` rake task
- `AddressLocality` — Works out which `Address::LOCALITIES` bucket an address falls in from its own city/state/country/county, for a writer that has an address but never asked the question (`PersonServices::ImportFromRolodex`, `PersonServices::CaptureFromSubmission`, `OrganizationServices::UpsertAddress`, `ProfileChangeRequests::Apply` — all on create only, so a locality a person picked is never overwritten). Reads a named non-US country as **Outside USA**, a non-CA US state as **Outside CA**, a Los Angeles mailing city as **LA City**, the rest of Los Angeles County as **LA County**, and Orange County as **Orange County**. Everything else stays **Unknown**: the Northern/Central/Southern line through California is AWBW's own and nothing records where it runs, and a state no US state answers to could as easily be a typo as a foreign region. Every comparison goes through `AddressNormalizer`. FileMaker records a `Locality` only on a **project**, so the organizations pass (ADR-0008) carries that across rather than deriving it; a person's own address has none
- `AddressDeduper` — Merges a person's or organization's addresses that are one place into a single address. `.matching_sets` groups by `AddressNormalizer.signature`, then merges a less complete copy (blank ZIP, street, country, …) into the single fuller address whose every part it shares — skipped when it fits more than one; works fullest-first so a chain of partial copies lands on the fullest. Each merge goes through `ModelDeduper#merge`, which repoints the merged address's inbound FKs — `contact_methods.address_id`, `invoices.bill_to_address_id`, `affiliations.organization_address_id` — onto the survivor. Within a set the survivor is the copy still in use — active before inactive, then the primary, then the oldest (`.keeper_order`) — since `Address.active` drives bill-to addresses, an org's program location and the geographic breakdowns; blank columns on the survivor (`CARRIED_COLUMNS` — the street/ZIP/country/county/district/type/locality, `phone` and the three `la_*` districts; "unknown" type/locality counts as blank) are filled from the merged copies. Then settles a single primary on the set holding the signature captured in the deduper's `merge_keeper` (the survivor's own pre-merge primary address, else the deleted record's), demoting the rest. Runs in the people and organizations dedupers' `after_merge`; `AddressDeduper.merge_duplicates(addressable)` runs it standalone (keeping the owner's current primary) for the **Dedupe addresses** admin page (`AddressesController#dedupe_index`/`#dedupe_perform`)
- `AddressServices::DuplicateFinder` — Lists the people and organizations holding addresses that `AddressDeduper.matching_sets` would merge, for the **Dedupe addresses** page. Compares only within one owner — two orgs at one street are separate records, not duplicates
Expand Down Expand Up @@ -595,7 +594,7 @@ RuboCop linting on PRs and pushes to main.

## Rake Tasks

Located in `lib/tasks/` (19 files):
Located in `lib/tasks/` (18 files):
- `dev.rake` — Development database seeding from XML/CSV
- `rhino_migrator.rake` — Rich text editor migration
- `attachment_report.rake` — Attachment reporting
Expand All @@ -608,7 +607,6 @@ Located in `lib/tasks/` (19 files):
- `migrate_workshop_logs.rake` — Workshop log migration
- `backfill_user_stamps.rake` — Backfill `created_by_id`/`updated_by_id` on legacy rows from the Ahoy trail (`data:backfill_user_stamps`)
- `backfill_notification_stamps.rake` — Backfill notifications' `created_by_id`/`updated_by_id` from `sender_id`, nil-only (`data:backfill_notification_stamps`)
- `translate_addresses.rake` — Rewrite stored address countries/states and their form answers as the dropdowns store them (`data:translate_addresses`, via `AddressServices::TranslateExisting`, idempotent, DRY_RUN, lists values nothing translates; every run, dry or not, saves its report — including every change as a CSV row (record, person/organization/submission ids, field, before, after; `VERBOSE=true` also prints them) — as an ActiveStorage blob and prints how to read it back; a real run credits its Ahoy events to `Setting.import_user` and aborts when none is chosen)
- `fm_import.rake` — Load the FileMaker CSV exports into the `fm_*` archive tables (`fm:import`, reads each CSV from the `FM Archive` resources, batch `insert_all`, skips rows whose key is already stored)
- `fm_import_people.rake` — Create people from the FileMaker Rolodex archive (`fm:import_people`, via `PersonServices::ImportFromRolodex`, links existing by name + email, idempotent, DRY_RUN; VERBOSE=true lists every row, FM_IDS=a,b re-runs just those rows past the idempotence guard; commits every 1,000 rows, prints progress, saves the full report as an ActiveStorage blob even on a crash, and aborts when no `TagAlias` rows exist or no `Setting.import_user` is chosen; logs every record it creates or fills as activity under a run name (`Current.source = "fm_import_people"`), which `fm:undo_import_people RUN=…` reverses via `PersonServices::UndoRolodexImport`)
- `resolve_resource_organizations.rake` — Link resources to organizations from the legacy `agency` free-text name (`data:resolve_resource_organizations`, match-or-create, idempotent, DRY_RUN)
Expand Down
4 changes: 1 addition & 3 deletions app/models/form_answer.rb
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# `submitted_answer` is stored verbatim — exactly what the applicant typed — so
# their words are never silently altered (e.g. "revenue < $5k" or "C++ <T>"
# survive intact). It is NOT HTML-sanitized on write. The one exception is the
# address country/state answers, which AddressServices::TranslateExisting rewrites
# as the dropdowns store them (ADR-0009), logging every change as an Ahoy event.
# survive intact). It is NOT HTML-sanitized on write.
#
# Safety is enforced at render time instead: every place that displays an answer
# uses plain `<%= %>`, which auto-escapes, so any markup shows as inert text.
Expand Down
105 changes: 0 additions & 105 deletions app/services/address_services/translate_existing.rb

This file was deleted.

Loading
Loading