diff --git a/AGENTS.md b/AGENTS.md index 8b60da2b2a..7099d2f963 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -49,7 +49,7 @@ This codebase (Rails 8.1) | Directory | Purpose | Count | |---|---|---| | `app/models/` | ActiveRecord models | ~146 files | -| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~150 files | +| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~155 files | | `app/jobs/` | SolidQueue background jobs | 6 files | | `app/models/concerns/` | Shared model modules | 23 concerns | @@ -266,6 +266,9 @@ or symbol. So write `authorize! @setting` in `show` (not `authorize! @setting, t - `ModelDeduper` — Deduplication logic. On a natural-key collision it now merges the losing row into the survivor (recursively, via itself) so the loser's own children move first — instead of destroying the loser and cascading them away or aborting on a protective `before_destroy` (e.g. a professional license with CE registrations). It finds reassignable children two ways: the model's own `has_many` declarations, and a scan of the DB's foreign keys. **A polymorphic child carries no foreign key**, so it is reachable only through an explicit `has_many … as:` — without one, `#unhandled_references` sees it and the merge is refused outright (this is why `Person`/`Organization` declare `has_many :invoices, as: :invoicee`). A config can also pass `reference_columns:` — column names that point at the model in any table without a DB foreign key (the users deduper names `user_id`/`created_by_id`/`updated_by_id`/`welcome_instructions_sent_by_id`). Self-references on the model's own table follow the keeper, except the keeper's own reference to the duplicate, which clears. The preview lists each moving record by its decorator's optional `#dedupe_label` (Affiliation, ContactMethod, Comment, Notification define one; text snippets cap at `SNIPPET_LENGTH`) - `ContinuingEducationDeduper` — Collapses duplicate CE registrations left sharing one event registration + license after a merge, moving the losers' allocations (payments) and comments onto the survivor. Keeps the real cost (larger of the two) rather than inventing one to cover the combined payments — if they now exceed it, `ContinuingEducationRegistration#over_allocated?` flags it (`ContinuingEducationRegistrationDecorator#overpayment_badge` → "Check payments") for an admin. CE has no DB unique index, so `ModelDeduper` never compares CE rows and just moves them all onto the keeper; a unique index is deliberately avoided because it would force the engine to move CE via validated `update!`, which trips `ContinuingEducationRegistration#license_belongs_to_registrant` mid-merge. Run from the people deduper's `after_merge` - `AddressNormalizer` — Comparison keys for address parts so spellings of one place match: street suffixes/directionals/ordinals (`Street`→`st`, `North`→`n`, `First`→`1st`), unit designators (`Suite`/`Apt`/`Unit`/`#` → `#`), `Saint`/`Mount`/`Fort` in cities, state names → USPS abbreviations, ZIP+4 → five digits, US country spellings (and a blank country in a US state) → `us`, a trailing "County", and an "unknown" type/locality → blank. `.signature(address)` is the whole-address key. Matching only — stored addresses keep their spelling. Used by `AddressDeduper`, `OrganizationServices::AddressMatcher`, `OrganizationServices::DuplicateFinder`'s shared-address signal, and `PersonServices::CaptureFromSubmission`'s mailing-address match +- `AddressTranslation` — Turns imported and legacy address values into what the address dropdowns store (ADR-0009): `#country(value, state:)` → a `GeographyHelper::COUNTRIES` name (aliases for our data's spellings, ISO codes, plain names, a country read from the state — a US state incl. "CA - California", a `REGIONS` province, or a country named as the state; placeholders like "-"/"n/a"/"unknown" → nil; "CA" next to state CA → US), `#state(value, country:)` → USPS code for a US state (other regions kept), `#address_type` (the archive's/forms' words via `CONTACT_TYPES`), `#locality` (`AddressLocality` on the translated parts), and `#address(attrs)` for a whole hash. Lookup tables only, never guessed (like `TagTranslation`): a country nothing answers to is cleared, so every stored country is a dropdown name or blank, and counted in `#unmatched`; an unreadable state is kept and counted. Used by `FmAddressAttributes` (every FileMaker import), `EventRegistrationImporter` (country/state answers + address/phone types; unmatched values listed on the import preview and counted in its notice), and `AddressServices::TranslateExisting`. No save hook — the dropdowns own new data +- `FmAddressAttributes` — Turns an `FmAddress` row into `Address` attributes for any FileMaker import (street from `StreetNo` + lines, ZIP or its `PostalCodeLookup`, county from the archive's ZIP table read once, everything clipped to 255), with state/country/type/locality through `AddressTranslation`; nil when the row lacks a street, city or state. An `address_type:` overrides the row's (an organization's addresses are work); `#place(city:, state:, county:)` builds a street-less address for a place the archive records by city alone (a project). `#unmatched` gathers what it couldn't translate. Used by `PersonServices::ImportFromRolodex` (listed in `fm:import_people`'s report) and the FileMaker organizations/affiliations imports (ADR-0008) +- `AddressServices::TranslateExisting` — Rewrites stored `Address` countries/states and the `mailing_*`/`organization_*` country/state `FormAnswer`s through `AddressTranslation` (each answer read beside its paired state/country answer), fills a locality only where it's Unknown, clears (address) or blanks (answer) a country nothing translates, and logs each changed record as one `autochange.address`/`autochange.form_answer` Ahoy event (all in one visit, `reason: "address_translation"`). Idempotent; `dry_run:` reports without writing. Drives the `data:translate_addresses` rake task - `AddressLocality` — Works out which `Address::LOCALITIES` bucket an address falls in from its own city/state/country/county, for a writer that has an address but never asked the question (`PersonServices::ImportFromRolodex`, `PersonServices::CaptureFromSubmission`, `OrganizationServices::UpsertAddress`, `ProfileChangeRequests::Apply` — all on create only, so a locality a person picked is never overwritten). Reads a named non-US country as **Outside USA**, a non-CA US state as **Outside CA**, a Los Angeles mailing city as **LA City**, the rest of Los Angeles County as **LA County**, and Orange County as **Orange County**. Everything else stays **Unknown**: the Northern/Central/Southern line through California is AWBW's own and nothing records where it runs, and a state no US state answers to could as easily be a typo as a foreign region. Every comparison goes through `AddressNormalizer`. FileMaker records a `Locality` only on a **project**, so the organizations pass (ADR-0008) carries that across rather than deriving it; a person's own address has none - `AddressDeduper` — Merges a person's or organization's addresses that are one place into a single address. `.matching_sets` groups by `AddressNormalizer.signature`, then merges a less complete copy (blank ZIP, street, country, …) into the single fuller address whose every part it shares — skipped when it fits more than one; works fullest-first so a chain of partial copies lands on the fullest. Each merge goes through `ModelDeduper#merge`, which repoints the merged address's inbound FKs — `contact_methods.address_id`, `invoices.bill_to_address_id`, `affiliations.organization_address_id` — onto the survivor. Within a set the survivor is the copy still in use — active before inactive, then the primary, then the oldest (`.keeper_order`) — since `Address.active` drives bill-to addresses, an org's program location and the geographic breakdowns; blank columns on the survivor (`CARRIED_COLUMNS` — the street/ZIP/country/county/district/type/locality, `phone` and the three `la_*` districts; "unknown" type/locality counts as blank) are filled from the merged copies. Then settles a single primary on the set holding the signature captured in the deduper's `merge_keeper` (the survivor's own pre-merge primary address, else the deleted record's), demoting the rest. Runs in the people and organizations dedupers' `after_merge`; `AddressDeduper.merge_duplicates(addressable)` runs it standalone (keeping the owner's current primary) for the **Dedupe addresses** admin page (`AddressesController#dedupe_index`/`#dedupe_perform`) - `AddressServices::DuplicateFinder` — Lists the people and organizations holding addresses that `AddressDeduper.matching_sets` would merge, for the **Dedupe addresses** page. Compares only within one owner — two orgs at one street are separate records, not duplicates @@ -591,7 +594,7 @@ RuboCop linting on PRs and pushes to main. ## Rake Tasks -Located in `lib/tasks/` (18 files): +Located in `lib/tasks/` (20 files): - `dev.rake` — Development database seeding from XML/CSV - `rhino_migrator.rake` — Rich text editor migration - `attachment_report.rake` — Attachment reporting @@ -604,6 +607,7 @@ Located in `lib/tasks/` (18 files): - `migrate_workshop_logs.rake` — Workshop log migration - `backfill_user_stamps.rake` — Backfill `created_by_id`/`updated_by_id` on legacy rows from the Ahoy trail (`data:backfill_user_stamps`) - `backfill_notification_stamps.rake` — Backfill notifications' `created_by_id`/`updated_by_id` from `sender_id`, nil-only (`data:backfill_notification_stamps`) +- `translate_addresses.rake` — Rewrite stored address countries/states and their form answers as the dropdowns store them (`data:translate_addresses`, via `AddressServices::TranslateExisting`, idempotent, DRY_RUN, lists values nothing translates; every run, dry or not, saves its report as an ActiveStorage blob and prints how to read it back; a real run credits its Ahoy events to `Setting.import_user` and aborts when none is chosen) - `fm_import.rake` — Load the FileMaker CSV exports into the `fm_*` archive tables (`fm:import`, reads each CSV from the `FM Archive` resources, batch `insert_all`, skips rows whose key is already stored) - `fm_import_people.rake` — Create people from the FileMaker Rolodex archive (`fm:import_people`, via `PersonServices::ImportFromRolodex`, links existing by name + email, idempotent, DRY_RUN; VERBOSE=true lists every row, FM_IDS=a,b re-runs just those rows past the idempotence guard; commits every 1,000 rows, prints progress, saves the full report as an ActiveStorage blob even on a crash, and aborts when no `TagAlias` rows exist or no `Setting.import_user` is chosen; logs every record it creates or fills as activity under a run name (`Current.source = "fm_import_people"`), which `fm:undo_import_people RUN=…` reverses via `PersonServices::UndoRolodexImport`) - `resolve_resource_organizations.rake` — Link resources to organizations from the legacy `agency` free-text name (`data:resolve_resource_organizations`, match-or-create, idempotent, DRY_RUN) diff --git a/app/controllers/event_registration_imports_controller.rb b/app/controllers/event_registration_imports_controller.rb index 854f524133..6fe61be3f3 100644 --- a/app/controllers/event_registration_imports_controller.rb +++ b/app/controllers/event_registration_imports_controller.rb @@ -116,6 +116,15 @@ def import_notice(result) "#{", #{result.payments_recorded} payments (#{helpers.dollars_from_cents(result.payments_amount_cents)})" if result.payments_recorded.positive?}" \ "#{", #{result.discounts_recorded} discounts (#{helpers.dollars_from_cents(result.discounts_amount_cents)})" if result.discounts_recorded.positive?}" \ "#{", #{result.ce_enrollments} CE enrollments" if result.ce_enrollments.positive?}" \ - "#{", #{result.skipped.size} rows skipped" if result.skipped.any?}." + "#{", #{result.skipped.size} rows skipped" if result.skipped.any?}" \ + "#{untranslated_notice(result.untranslated_addresses)}." + end + + def untranslated_notice(untranslated) + return if untranslated.empty? + + outcomes = untranslated.keys.map { |field| field == :country ? "country left blank" : "state kept" } + ", #{helpers.pluralize(untranslated.values.sum { |values| values.values.sum }, 'address value')} " \ + "the dropdowns don't offer (#{outcomes.join(', ')})" end end diff --git a/app/models/form_answer.rb b/app/models/form_answer.rb index ea8fd0950f..3d24ad0e18 100644 --- a/app/models/form_answer.rb +++ b/app/models/form_answer.rb @@ -1,6 +1,8 @@ # `submitted_answer` is stored verbatim — exactly what the applicant typed — so # their words are never silently altered (e.g. "revenue < $5k" or "C++ " -# survive intact). It is NOT HTML-sanitized on write. +# survive intact). It is NOT HTML-sanitized on write. The one exception is the +# address country/state answers, which AddressServices::TranslateExisting rewrites +# as the dropdowns store them (ADR-0009), logging every change as an Ahoy event. # # Safety is enforced at render time instead: every place that displays an answer # uses plain `<%= %>`, which auto-escapes, so any markup shows as inert text. diff --git a/app/services/address_services/translate_existing.rb b/app/services/address_services/translate_existing.rb new file mode 100644 index 0000000000..56e25e1e41 --- /dev/null +++ b/app/services/address_services/translate_existing.rb @@ -0,0 +1,96 @@ +module AddressServices + # Rewrites stored address countries and states, and the country/state answers on form + # submissions, as the dropdowns store them (ADR-0009), filling a locality only where it's + # Unknown. Each changed record is one autochange Ahoy event, all in one visit. Idempotent, + # so it runs before and after an import. + class TranslateExisting + Result = Data.define(:addresses, :answers, :changes, :unmatched) + + REASON = "address_translation" + # Each address answer, and the answer beside it that the translation reads alongside. + ANSWER_PAIRS = { + "mailing_country" => "mailing_state", "mailing_state" => "mailing_country", + "organization_country" => "organization_state", "organization_state" => "organization_country" + }.freeze + + # `log` receives one line per changed record, for the rake task's preview. + def initialize(dry_run: false, log: nil, user: nil) + @dry_run = dry_run + @log = log + @user = user + end + + def call + @translation = AddressTranslation.new + @tracker = Ahoy::Tracker.new(user: @user) + @changes = Hash.new(0) + addresses = answers = 0 + + ActiveRecord::Base.transaction do + Address.find_each { |address| addresses += 1 if translate_address(address) } + address_answers.find_each { |answer| answers += 1 if translate_answer(answer) } + end + + Result.new(addresses:, answers:, changes: @changes, unmatched: @translation.unmatched) + end + + private + + def translate_address(address) + state = @translation.state(address.state, country: address.country) + country = @translation.country(address.country, state: address.state) + updates = {} + updates[:state] = state if state.present? && state != address.state + updates[:country] = country if country != address.country + if address.locality.blank? || address.locality == Address::UNKNOWN_LOCALITY + locality = @translation.locality(city: address.city, state:, country:, county: address.county) + updates[:locality] = locality if locality != address.locality + end + apply(address, updates, title: address.name, addressable_type: address.addressable_type, addressable_id: address.addressable_id) + end + + def translate_answer(answer) + identifier = answer.form_field.field_identifier + paired = paired_answers.dig(answer.form_submission_id, ANSWER_PAIRS[identifier]) + raw = answer.submitted_answer + translated = if identifier.end_with?("_country") + @translation.country(raw, state: paired).to_s + else + @translation.state(raw, country: paired).to_s + end + return false if translated == raw.to_s || (translated.blank? && identifier.end_with?("_state")) + + apply(answer, { submitted_answer: translated }, title: answer.form_field.name, + form_submission_id: answer.form_submission_id, field_identifier: identifier) + end + + def apply(record, updates, title:, **extra) + return false if updates.empty? + + label = record.class.model_name.human + updates.each do |field, after| + @changes["#{label} #{field.to_s.humanize(capitalize: false)}: #{record[field].inspect} → #{after.inspect}"] += 1 + end + @log&.call(" #{label} ##{record.id}: #{updates.map { |field, after| "#{field} #{record[field].inspect} → #{after.inspect}" }.join(', ')}") + return true if @dry_run + + changes = updates.to_h { |field, after| [ field, { before: record[field], after: } ] } + record.update_columns(updates) + @tracker.track("autochange.#{record.class.model_name.singular}", { + resource_type: record.class.name, resource_id: record.id, resource_title: title, changes:, reason: REASON, **extra + }) + true + end + + def address_answers + FormAnswer.joins(:form_field).where(form_fields: { field_identifier: ANSWER_PAIRS.keys }).includes(:form_field) + end + + # { submission_id => { identifier => answer } }, read before any answer is rewritten so + # each translation sees what the person typed beside it. + def paired_answers + @paired_answers ||= address_answers.pluck(:form_submission_id, "form_fields.field_identifier", :submitted_answer) + .each_with_object(Hash.new { |hash, id| hash[id] = {} }) { |(id, identifier, value), map| map[id][identifier] = value } + end + end +end diff --git a/app/services/address_translation.rb b/app/services/address_translation.rb new file mode 100644 index 0000000000..39b95336ad --- /dev/null +++ b/app/services/address_translation.rb @@ -0,0 +1,145 @@ +# Turns the address values other systems and older free-text forms wrote — FileMaker, the +# LMS registration sheet, the portal's own answers from before the dropdowns — into what the +# portal's address dropdowns store (ADR-0009). Translation is by lookup table, never guessed: +# a country nothing answers to is cleared so every stored country is a dropdown one, and +# counted in #unmatched with the state values it can't read. +# +# Built once per run and reused, so an import's report can list everything it couldn't place. +class AddressTranslation + UNITED_STATES = "United States" + + # The words the archive and the forms use for a kind of address, phone, or email. + CONTACT_TYPES = { "home" => "personal", "personal" => "personal", "cell" => "personal", + "mobile" => "personal", "work" => "work", "business" => "work", + "professional" => "work", "mailing" => "mailing" }.freeze + + # Spellings found in our data that the countries gem doesn't read, by ISO code. A US city + # or county typed as the country is the US; the outlying islands aren't on the dropdown. + ALIASES = { + "unites states" => "US", "untied states" => "US", "los angeles" => "US", "la county" => "US", + "us minor outlying islands" => "US", "united states minor outlying islands" => "US", + "korea" => "KR", "republic of ireland" => "IE", "mexcio" => "MX", "scotland" => "GB", "republic of armenia" => "AM" + }.merge(GeographyHelper::PLAIN_COUNTRY_NAMES.to_h { |code, name| [ name.downcase, code ] }).freeze + + # Provinces and regions written in the state field, keyed as AddressNormalizer.words joins them. + REGIONS = { + "Canada" => [ "on", "ontario", "bc", "british columbia", "ab", "alberta", "ns", "nova scotia", "qc", "quebec", + "mb", "manitoba", "sk", "saskatchewan", "nb", "new brunswick", "nl", "newfoundland", + "newfoundland and labrador", "pe", "pei", "prince edward island", "yt", "yukon", "nu", "nunavut", + "northwest territories" ], + "Australia" => [ "nsw", "new south wales", "vic", "victoria", "qld", "queensland", "tas", "tasmania", + "south australia", "western australia" ], + "United Kingdom" => [ "england", "scotland", "wales", "northern ireland", "london", "bucks", "buckinghamshire", + "somerset", "argyle and bute", "argyll and bute" ], + "Vietnam" => [ "ha noi", "hanoi" ], + "Mexico" => [ "mexico city", "cdmx" ], + "Japan" => [ "tokyo" ], + "Philippines" => [ "metro manila" ], + "Ghana" => [ "greater accra" ], + "Trinidad and Tobago" => [ "trinidad", "tobago" ] + }.flat_map { |country, regions| regions.map { |region| [ region, country ] } }.to_h.freeze + + # Typed in place of a country to say there isn't one, keyed as AddressNormalizer.words joins them. + PLACEHOLDERS = [ "n a", "none", "unknown", "not applicable" ].to_set.freeze + + # A US state's full name anywhere in the value, so "CA - California" and "Maryland (MD)" read as a state. + US_STATE_NAME = /\b(#{AddressNormalizer::STATE_NAMES.keys.map { |name| Regexp.escape(name) }.join("|")})\b/ + + attr_reader :unmatched + + def initialize + @unmatched = Hash.new { |hash, field| hash[field] = Hash.new(0) } + end + + # A dropdown country: the one the value names, else the one its state implies, else nil + # (reported when there was a value to read). + def country(value, state:) + text = value.to_s.strip + text = "" if placeholder?(text) + return UNITED_STATES if us_state_code(text) && us_state_code(state) + + found = country_named(text) || country_from_state(state) + @unmatched[:country][text] += 1 if found.nil? && text.present? + found + end + + # A US state's USPS code; any other region as typed. A value that reads as neither, on an + # address in the US or with no country, is reported. + def state(value, country:) + text = value.to_s.strip + return if text.blank? + + abroad = country_named(country.to_s.strip).then { |named| named.present? && named != UNITED_STATES } + return text if abroad + + code = us_state_code(text) + return code.upcase if code + + @unmatched[:state][text] += 1 unless country_from_state(text) + text + end + + def address_type(value) + CONTACT_TYPES[value.to_s.strip.downcase].presence_in(Address::CONTACT_TYPES) + end + + def locality(city:, state:, country:, county: nil) + AddressLocality.for(city:, state:, country:, county:) + end + + # Address attributes with their state, country, and type translated and a locality worked out. + def address(attributes) + state = state(attributes[:state], country: attributes[:country]) + country = country(attributes[:country], state: attributes[:state]) + attributes.merge(state:, country:, address_type: address_type(attributes[:address_type]), + locality: locality(city: attributes[:city], state:, country:, county: attributes[:county])) + end + + private + + def country_named(text) + return if text.blank? + + code = ALIASES[text.downcase] + code ||= "US" if AddressNormalizer::US_COUNTRY_NAMES.include?(text.downcase.delete("^a-z")) + found = code ? ISO3166::Country.new(code) : gem_country(text) + name_for(found) if found + end + + def gem_country(text) + ISO3166::Country.find_country_by_alpha2(text) || + ISO3166::Country.find_country_by_alpha3(text) || + ISO3166::Country.find_country_by_any_name(text) + end + + # A US state, a known province or region, or a country named as the state ("England", "Togo"). + def country_from_state(state) + return UNITED_STATES if us_state_code(state) + + key = AddressNormalizer.words(state).join(" ") + return if key.blank? + return REGIONS[key] if REGIONS.key?(key) + + found = ISO3166::Country.find_country_by_any_name(state.to_s.strip) + name_for(found) if found + end + + def name_for(country) + return UNITED_STATES if ALIASES[country.common_name.downcase] == "US" + + GeographyHelper::PLAIN_COUNTRY_NAMES[country.alpha2] || country.common_name + end + + # The lowercase USPS code a value names, or nil: "CA", "Ca.", "California", "CA - California". + def us_state_code(value) + code = AddressNormalizer.state(value) + return code if AddressNormalizer::US_STATES.include?(code) + + name = AddressNormalizer.words(value).join(" ")[US_STATE_NAME] + AddressNormalizer::STATE_NAMES[name] if name + end + + def placeholder?(text) + !text.match?(/[[:alpha:]]/) || PLACEHOLDERS.include?(AddressNormalizer.words(text).join(" ")) + end +end diff --git a/app/services/event_registration_importer.rb b/app/services/event_registration_importer.rb index 688ebdd4dc..e6e20aeacf 100644 --- a/app/services/event_registration_importer.rb +++ b/app/services/event_registration_importer.rb @@ -132,7 +132,7 @@ def self.importable?(event) :payments_recorded, :payments_amount_cents, :scholarships_recorded, :scholarships_amount_cents, :discounts_recorded, :discounts_amount_cents, - :ce_enrollments, :unrecognized_headers, + :ce_enrollments, :unrecognized_headers, :untranslated_addresses, :skipped, :rows, keyword_init: true ) do @@ -188,10 +188,11 @@ def initialize(file_path:, event:, import_user: nil, source: nil, dry_run: false payments_recorded: 0, payments_amount_cents: 0, scholarships_recorded: 0, scholarships_amount_cents: 0, discounts_recorded: 0, discounts_amount_cents: 0, - ce_enrollments: 0, unrecognized_headers: [], + ce_enrollments: 0, unrecognized_headers: [], untranslated_addresses: {}, skipped: [], rows: [] ) @organization_matcher = OrganizationServices::NameMatcher.new + @address_translation = AddressTranslation.new @organization_cache = {} @form_field_cache = {} @seen_row_keys = {} @@ -207,6 +208,8 @@ def call @payment_batches.each_value { |members| apply_payment_batch(members) } raise ActiveRecord::Rollback if @dry_run end + unmatched = @address_translation.unmatched + @result.untranslated_addresses = %i[country state].select { |field| unmatched[field].any? }.index_with { |field| unmatched[field].to_h } @result end @@ -526,15 +529,27 @@ def build_answers(values, preview) answers["additional_sectors"] = sector_ids(values[:additional_sectors], preview, scope: Sector.published).map(&:to_s).presence answers["primary_age_group"] = age_group_ids(values[:primary_age_group], preview).first&.to_s answers["additional_age_groups"] = age_group_ids(values[:additional_age_groups], preview).map(&:to_s).presence + translate_address_answers(answers, "mailing") + translate_address_answers(answers, "organization") answers.compact end + # The sheet's free-text state and country, as the registration form's dropdowns store them. + def translate_address_answers(answers, prefix) + state = answers["#{prefix}_state"] + country = answers["#{prefix}_country"] + return if state.blank? && country.blank? + + answers["#{prefix}_state"] = @address_translation.state(state, country:) + answers["#{prefix}_country"] = @address_translation.country(country, state:) + end + def organization_type(raw) Organization::ORGANIZATION_TYPES.find { |type| type.casecmp?(raw) } || "Other: #{raw}" end def contact_type(raw) - CONTACT_TYPES.find { |type| type.casecmp?(raw.to_s) } + CONTACT_TYPES.find { |type| type.casecmp?(@address_translation.address_type(raw).to_s) } end def sector_ids(raw, preview, scope:) diff --git a/app/services/fm_address_attributes.rb b/app/services/fm_address_attributes.rb new file mode 100644 index 0000000000..596c4d3de7 --- /dev/null +++ b/app/services/fm_address_attributes.rb @@ -0,0 +1,75 @@ +# Turns an FmAddress row's fields into Address attributes — state, country, type and +# locality translated as the dropdowns store them (ADR-0009) — or nil when the row lacks the +# street, city or state an address needs. One instance serves a whole import: the archive's +# ZIP table is read once, and #unmatched gathers every value it couldn't translate. +class FmAddressAttributes + STRING_LIMIT = 255 + + def initialize + @translation = AddressTranslation.new + end + + def unmatched = @translation.unmatched + + def call(fields, address_type: nil) + street = [ street_line(fields), fields["AddrsLine2"] ].filter_map { |line| line.to_s.strip.presence }.join(", ") + city = clip(fields["AddrsCity"].to_s.strip) + state = clip(fields["AddrsState"].to_s.strip) + return if street.blank? || city.blank? || state.blank? + + @translation.address( + street_address: clip(street), + city:, + state:, + zip_code: zip_code(fields), + country: clip(fields["AddrsCountry"]), + county: county(fields), + address_type: fields["AddrsType"] + ).then { |attributes| address_type ? attributes.merge(address_type:) : attributes } + end + + # An address holding just a city and state — where the archive records a place (a + # project's city) with no street — translated the same way. + def place(city:, state:, county: nil) + @translation.address(street_address: "", zip_code: "", city: clip(city), state: clip(state), country: nil, + county: clip(county), address_type: nil) + end + + private + + # The house number has its own field, sometimes already repeated at the front of + # the first line. + def street_line(fields) + number = fields["StreetNo"].to_s.strip + line = fields["AddrsLine1"].to_s.strip + return line if number.blank? || line[/\A\S+/] == number + + "#{number} #{line}".strip + end + + # The address's own postal code, or the one it points at in the archive's ZIP table + # when it carries none itself — the field it mirrors, and where the county is read. + def zip_code(fields) + [ fields["AddrsPostalCode"], fields["PostalCodeLookup"] ] + .filter_map { |value| value.to_s.strip.presence }.first.to_s.truncate(STRING_LIMIT) + end + + def county(fields) + zip_counties[zip_lookup(fields)[/\A\d{5}/]] + end + + def zip_lookup(fields) + (fields["PostalCodeLookup"].presence || fields["AddrsPostalCode"]).to_s.strip + end + + # One query for the archive's ~80,000 zip codes beats one per address across a full import. + def zip_counties + @zip_counties ||= FmPostalCode.pluck(:fm_id, Arel.sql("JSON_UNQUOTE(JSON_EXTRACT(data, '$.County'))")) + .to_h { |zip, county| [ zip, clip(county) ] } + end + + # The archive's free text isn't bounded by the varchar(255) it lands in. + def clip(value) + value.presence && value.to_s.truncate(STRING_LIMIT) + end +end diff --git a/app/services/person_services/import_from_rolodex.rb b/app/services/person_services/import_from_rolodex.rb index bbd7eb85f1..a37c60a122 100644 --- a/app/services/person_services/import_from_rolodex.rb +++ b/app/services/person_services/import_from_rolodex.rb @@ -40,9 +40,7 @@ class ImportFromRolodex DATE_FORMATS = %w[%m/%d/%Y %Y-%m-%d %m-%d-%Y %Y/%m/%d].freeze # A two-digit year below this is read as 20xx, at or above it as 19xx. TWO_DIGIT_YEAR_PIVOT = 10 - CONTACT_TYPES = { "home" => "personal", "personal" => "personal", "cell" => "personal", - "mobile" => "personal", "work" => "work", "business" => "work", - "professional" => "work", "mailing" => "mailing" }.freeze + CONTACT_TYPES = AddressTranslation::CONTACT_TYPES # Answers to "which kind of email is this?" that aren't a kind at all. PRECEDENCE_TYPES = %w[preferred primary main].freeze EMAIL_PATTERN = /[^\s,;<>"']+@[^\s,;<>"']+\.[^\s,;<>"']+/ @@ -76,6 +74,7 @@ def initialize(scope: FmRolodex.all, relink: false, progress: nil, run: nil) @imported_codes = Person.where.not(filemaker_code: [ nil, "" ]).pluck(:filemaker_code) .flat_map { |value| Person.split_filemaker_codes(value) }.to_set @translation = TagTranslation.new + @address_attributes = FmAddressAttributes.new @staff = StaffTranslation.new @import_user_id = Setting.import_user&.id end @@ -104,6 +103,10 @@ def unmatched_staff @staff.unmatched end + def unmatched_addresses + @address_attributes.unmatched + end + private # Each batch commits on its own (a savepoint inside a dry run's rollback), so a run that @@ -396,7 +399,7 @@ def names(model, ids) def build_addresses(person, rolodex) held = person.addresses.map { |address| address_key(address.street_address, address.city) } attributes = rolodex.addresses.reject { |address| work_address?(rolodex, address) } - .filter_map { |address| address_attributes(address.data) } + .filter_map { |address| @address_attributes.call(address.data) } .reject { |attrs| held.include?(address_key(attrs[:street_address], attrs[:city])) } build_primary_first(person.addresses, attributes) end @@ -427,49 +430,6 @@ def phone_key(value) value.to_s.gsub(/\D/, "") end - def address_attributes(fields) - street = [ street_line(fields), fields["AddrsLine2"] ].filter_map { |line| line.to_s.strip.presence }.join(", ") - city = clip(fields["AddrsCity"].to_s.strip) - state = clip(fields["AddrsState"].to_s.strip) - return if street.blank? || city.blank? || state.blank? - - country = clip(fields["AddrsCountry"]) - county = county(fields) - { - street_address: clip(street), - city:, - state:, - zip_code: zip_code(fields), - country:, - county:, - address_type: contact_type(fields["AddrsType"], allowed: Address::CONTACT_TYPES), - # FileMaker records a locality only on a project, so a person's own address has - # none to carry over and the parts have to say. - locality: AddressLocality.for(city:, state:, country:, county:) - } - end - - # The address's own postal code, or the one it points at in the archive's ZIP table - # when it carries none itself — the field it mirrors, and where the county is read. - def zip_code(fields) - [ fields["AddrsPostalCode"], fields["PostalCodeLookup"] ] - .filter_map { |value| value.to_s.strip.presence }.first.to_s.truncate(STRING_LIMIT) - end - - def county(fields) - zip_counties[zip_lookup(fields)[/\A\d{5}/]] - end - - def zip_lookup(fields) - (fields["PostalCodeLookup"].presence || fields["AddrsPostalCode"]).to_s.strip - end - - # One query for the archive's ~80,000 zip codes beats one per address across a full import. - def zip_counties - @zip_counties ||= FmPostalCode.pluck(:fm_id, Arel.sql("JSON_UNQUOTE(JSON_EXTRACT(data, '$.County'))")) - .to_h { |zip, county| [ zip, clip(county) ] } - end - def phone_attributes(fields) number = clip(fields["PhoneNumber"]) return if number.blank? @@ -484,16 +444,6 @@ def contact_type(value, allowed:) CONTACT_TYPES[value.to_s.strip.downcase].presence_in(allowed) end - # The house number has its own field, sometimes already repeated at the front of - # the first line. - def street_line(fields) - number = fields["StreetNo"].to_s.strip - line = fields["AddrsLine1"].to_s.strip - return line if number.blank? || line[/\A\S+/] == number - - "#{number} #{line}".strip - end - # Every string copied here lands in a varchar(255) the archive's free text isn't # bounded by, and an over-long incidental field would otherwise cost the person — # or, for a column with no length validation behind it, raise mid-import. diff --git a/app/views/event_registration_imports/create.html.erb b/app/views/event_registration_imports/create.html.erb index 87d7687033..694697ac7d 100644 --- a/app/views/event_registration_imports/create.html.erb +++ b/app/views/event_registration_imports/create.html.erb @@ -41,6 +41,15 @@ <% end %> + <% if @result.untranslated_addresses.any? %> +
+

Address values the dropdowns don't offer

+ <% @result.untranslated_addresses.each do |field, values| %> +

<%= field == :country ? "Country (left blank)" : "State (kept as typed)" %>: <%= values.map { |value, count| "“#{value}”#{" (#{pluralize(count, 'row')})" if count > 1}" }.to_sentence %>

+ <% end %> +
+ <% end %> +