diff --git a/AGENTS.md b/AGENTS.md index 4e25420a52..4d8f83175d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -269,9 +269,8 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `ModelDeduper` — Deduplication logic. On a natural-key collision it now merges the losing row into the survivor (recursively, via itself) so the loser's own children move first — instead of destroying the loser and cascading them away or aborting on a protective `before_destroy` (e.g. a professional license with CE registrations). It finds reassignable children two ways: the model's own `has_many` declarations, and a scan of the DB's foreign keys. **A polymorphic child carries no foreign key**, so it is reachable only through an explicit `has_many … as:` — without one, `#unhandled_references` sees it and the merge is refused outright (this is why `Person`/`Organization` declare `has_many :invoices, as: :invoicee`). A config can also pass `reference_columns:` — column names that point at the model in any table without a DB foreign key (the users deduper names `user_id`/`created_by_id`/`updated_by_id`/`welcome_instructions_sent_by_id`). Self-references on the model's own table follow the keeper, except the keeper's own reference to the duplicate, which clears. The preview lists each moving record by its decorator's optional `#dedupe_label` (Affiliation, ContactMethod, Comment, Notification define one; text snippets cap at `SNIPPET_LENGTH`) - `ContinuingEducationDeduper` — Collapses duplicate CE registrations left sharing one event registration + license after a merge, moving the losers' allocations (payments) and comments onto the survivor. Keeps the real cost (larger of the two) rather than inventing one to cover the combined payments — if they now exceed it, `ContinuingEducationRegistration#over_allocated?` flags it (`ContinuingEducationRegistrationDecorator#overpayment_badge` → "Check payments") for an admin. CE has no DB unique index, so `ModelDeduper` never compares CE rows and just moves them all onto the keeper; a unique index is deliberately avoided because it would force the engine to move CE via validated `update!`, which trips `ContinuingEducationRegistration#license_belongs_to_registrant` mid-merge. Run from the people deduper's `after_merge` - `AddressNormalizer` — Comparison keys for address parts so spellings of one place match: street suffixes/directionals/ordinals (`Street`→`st`, `North`→`n`, `First`→`1st`), unit designators (`Suite`/`Apt`/`Unit`/`#` → `#`), `Saint`/`Mount`/`Fort` in cities, state names → USPS abbreviations, ZIP+4 → five digits, US country spellings (and a blank country in a US state) → `us`, a trailing "County", and an "unknown" type/locality → blank. `.signature(address)` is the whole-address key. Matching only — stored addresses keep their spelling. Used by `AddressDeduper`, `OrganizationServices::AddressMatcher`, `OrganizationServices::DuplicateFinder`'s shared-address signal, and `PersonServices::CaptureFromSubmission`'s mailing-address match -- `AddressTranslation` — Turns imported and legacy address values into what the address dropdowns store (ADR-0009): `#country(value, state:)` → a `GeographyHelper::COUNTRIES` name (aliases for our data's spellings, ISO codes, plain names, a country read from the state — a US state incl. "CA - California", a `REGIONS` province, or a country named as the state; placeholders like "-"/"n/a"/"unknown" → nil; "CA" next to state CA → US), `#state(value, country:)` → USPS code for a US state (other regions kept), `#address_type` (the archive's/forms' words via `CONTACT_TYPES`), `#locality` (`AddressLocality` on the translated parts), and `#address(attrs)` for a whole hash. Lookup tables only, never guessed (like `TagTranslation`): a country nothing answers to is cleared, so every stored country is a dropdown name or blank, and counted in `#unmatched`; an unreadable state is kept and counted. Used by `FmAddressAttributes` (every FileMaker import), `EventRegistrationImporter` (country/state answers + address/phone types; unmatched values listed on the import preview and counted in its notice), and `AddressServices::TranslateExisting`. No save hook — the dropdowns own new data +- `AddressTranslation` — Turns imported and legacy address values into what the address dropdowns store (ADR-0009): `#country(value, state:)` → a `GeographyHelper::COUNTRIES` name (aliases for our data's spellings, ISO codes, plain names, a country read from the state — a US state incl. "CA - California", a `REGIONS` province, or a country named as the state; placeholders like "-"/"n/a"/"unknown" → nil; "CA" next to state CA → US), `#state(value, country:)` → USPS code for a US state (other regions kept), `#address_type` (the archive's/forms' words via `CONTACT_TYPES`), `#locality` (`AddressLocality` on the translated parts), and `#address(attrs)` for a whole hash. Lookup tables only, never guessed (like `TagTranslation`): a country nothing answers to is cleared, so every stored country is a dropdown name or blank, and counted in `#unmatched`; an unreadable state is kept and counted. Used by `FmAddressAttributes` (every FileMaker import) and `EventRegistrationImporter` (country/state answers + address/phone types; unmatched values listed on the import preview and counted in its notice). No save hook — the dropdowns own new data - `FmAddressAttributes` — Turns an `FmAddress` row into `Address` attributes for any FileMaker import (street from `StreetNo` + lines, ZIP or its `PostalCodeLookup`, county from the archive's ZIP table read once, everything clipped to 255), with state/country/type/locality through `AddressTranslation`; nil when the row lacks a street, city or state. An `address_type:` overrides the row's (an organization's addresses are work); `#place(city:, state:, county:)` builds a street-less address for a place the archive records by city alone (a project). `#unmatched` gathers what it couldn't translate. Used by `PersonServices::ImportFromRolodex` (listed in `fm:import_people`'s report) and the FileMaker organizations/affiliations imports (ADR-0008) -- `AddressServices::TranslateExisting` — Rewrites stored `Address` countries/states and the `mailing_*`/`organization_*` country/state `FormAnswer`s through `AddressTranslation` (each answer read beside its paired state/country answer), fills a locality only where it's Unknown, clears (address) or blanks (answer) a country nothing translates, and logs each changed record as one `autochange.address`/`autochange.form_answer` Ahoy event (all in one visit, `reason: "address_translation"`). Idempotent; `dry_run:` reports without writing. Drives the `data:translate_addresses` rake task - `AddressLocality` — Works out which `Address::LOCALITIES` bucket an address falls in from its own city/state/country/county, for a writer that has an address but never asked the question (`PersonServices::ImportFromRolodex`, `PersonServices::CaptureFromSubmission`, `OrganizationServices::UpsertAddress`, `ProfileChangeRequests::Apply` — all on create only, so a locality a person picked is never overwritten). Reads a named non-US country as **Outside USA**, a non-CA US state as **Outside CA**, a Los Angeles mailing city as **LA City**, the rest of Los Angeles County as **LA County**, and Orange County as **Orange County**. Everything else stays **Unknown**: the Northern/Central/Southern line through California is AWBW's own and nothing records where it runs, and a state no US state answers to could as easily be a typo as a foreign region. Every comparison goes through `AddressNormalizer`. FileMaker records a `Locality` only on a **project**, so the organizations pass (ADR-0008) carries that across rather than deriving it; a person's own address has none - `AddressDeduper` — Merges a person's or organization's addresses that are one place into a single address. `.matching_sets` groups by `AddressNormalizer.signature`, then merges a less complete copy (blank ZIP, street, country, …) into the single fuller address whose every part it shares — skipped when it fits more than one; works fullest-first so a chain of partial copies lands on the fullest. Each merge goes through `ModelDeduper#merge`, which repoints the merged address's inbound FKs — `contact_methods.address_id`, `invoices.bill_to_address_id`, `affiliations.organization_address_id` — onto the survivor. Within a set the survivor is the copy still in use — active before inactive, then the primary, then the oldest (`.keeper_order`) — since `Address.active` drives bill-to addresses, an org's program location and the geographic breakdowns; blank columns on the survivor (`CARRIED_COLUMNS` — the street/ZIP/country/county/district/type/locality, `phone` and the three `la_*` districts; "unknown" type/locality counts as blank) are filled from the merged copies. Then settles a single primary on the set holding the signature captured in the deduper's `merge_keeper` (the survivor's own pre-merge primary address, else the deleted record's), demoting the rest. Runs in the people and organizations dedupers' `after_merge`; `AddressDeduper.merge_duplicates(addressable)` runs it standalone (keeping the owner's current primary) for the **Dedupe addresses** admin page (`AddressesController#dedupe_index`/`#dedupe_perform`) - `AddressServices::DuplicateFinder` — Lists the people and organizations holding addresses that `AddressDeduper.matching_sets` would merge, for the **Dedupe addresses** page. Compares only within one owner — two orgs at one street are separate records, not duplicates @@ -595,7 +594,7 @@ RuboCop linting on PRs and pushes to main. ## Rake Tasks -Located in `lib/tasks/` (19 files): +Located in `lib/tasks/` (18 files): - `dev.rake` — Development database seeding from XML/CSV - `rhino_migrator.rake` — Rich text editor migration - `attachment_report.rake` — Attachment reporting @@ -608,7 +607,6 @@ Located in `lib/tasks/` (19 files): - `migrate_workshop_logs.rake` — Workshop log migration - `backfill_user_stamps.rake` — Backfill `created_by_id`/`updated_by_id` on legacy rows from the Ahoy trail (`data:backfill_user_stamps`) - `backfill_notification_stamps.rake` — Backfill notifications' `created_by_id`/`updated_by_id` from `sender_id`, nil-only (`data:backfill_notification_stamps`) -- `translate_addresses.rake` — Rewrite stored address countries/states and their form answers as the dropdowns store them (`data:translate_addresses`, via `AddressServices::TranslateExisting`, idempotent, DRY_RUN, lists values nothing translates; every run, dry or not, saves its report — including every change as a CSV row (record, person/organization/submission ids, field, before, after; `VERBOSE=true` also prints them) — as an ActiveStorage blob and prints how to read it back; a real run credits its Ahoy events to `Setting.import_user` and aborts when none is chosen) - `fm_import.rake` — Load the FileMaker CSV exports into the `fm_*` archive tables (`fm:import`, reads each CSV from the `FM Archive` resources, batch `insert_all`, skips rows whose key is already stored) - `fm_import_people.rake` — Create people from the FileMaker Rolodex archive (`fm:import_people`, via `PersonServices::ImportFromRolodex`, links existing by name + email, idempotent, DRY_RUN; VERBOSE=true lists every row, FM_IDS=a,b re-runs just those rows past the idempotence guard; commits every 1,000 rows, prints progress, saves the full report as an ActiveStorage blob even on a crash, and aborts when no `TagAlias` rows exist or no `Setting.import_user` is chosen; logs every record it creates or fills as activity under a run name (`Current.source = "fm_import_people"`), which `fm:undo_import_people RUN=…` reverses via `PersonServices::UndoRolodexImport`) - `resolve_resource_organizations.rake` — Link resources to organizations from the legacy `agency` free-text name (`data:resolve_resource_organizations`, match-or-create, idempotent, DRY_RUN) diff --git a/app/models/form_answer.rb b/app/models/form_answer.rb index 3d24ad0e18..ea8fd0950f 100644 --- a/app/models/form_answer.rb +++ b/app/models/form_answer.rb @@ -1,8 +1,6 @@ # `submitted_answer` is stored verbatim — exactly what the applicant typed — so # their words are never silently altered (e.g. "revenue < $5k" or "C++ " -# survive intact). It is NOT HTML-sanitized on write. The one exception is the -# address country/state answers, which AddressServices::TranslateExisting rewrites -# as the dropdowns store them (ADR-0009), logging every change as an Ahoy event. +# survive intact). It is NOT HTML-sanitized on write. # # Safety is enforced at render time instead: every place that displays an answer # uses plain `<%= %>`, which auto-escapes, so any markup shows as inert text. diff --git a/app/services/address_services/translate_existing.rb b/app/services/address_services/translate_existing.rb deleted file mode 100644 index bf89e7e673..0000000000 --- a/app/services/address_services/translate_existing.rb +++ /dev/null @@ -1,105 +0,0 @@ -module AddressServices - # Rewrites stored address countries and states, and the country/state answers on form - # submissions, as the dropdowns store them (ADR-0009), filling a locality only where it's - # Unknown. Each changed record is one autochange Ahoy event, all in one visit. Idempotent, - # so it runs before and after an import. - class TranslateExisting - Result = Data.define(:addresses, :answers, :changes, :unmatched, :rows) - # One field one record changed (or would change, on a dry run), and whose record it is. - Row = Data.define(:record_type, :record_id, :person_id, :organization_id, :form_submission_id, :field, :before, :after) - ROW_HEADERS = Row.members.freeze - - REASON = "address_translation" - # Each address answer, and the answer beside it that the translation reads alongside. - ANSWER_PAIRS = { - "mailing_country" => "mailing_state", "mailing_state" => "mailing_country", - "organization_country" => "organization_state", "organization_state" => "organization_country" - }.freeze - - # `log` receives one line per changed record, for the rake task's preview. - def initialize(dry_run: false, log: nil, user: nil) - @dry_run = dry_run - @log = log - @user = user - end - - def call - @translation = AddressTranslation.new - @tracker = Ahoy::Tracker.new(user: @user) - @changes = Hash.new(0) - @rows = [] - addresses = answers = 0 - - ActiveRecord::Base.transaction do - Address.find_each { |address| addresses += 1 if translate_address(address) } - address_answers.find_each { |answer| answers += 1 if translate_answer(answer) } - end - - Result.new(addresses:, answers:, changes: @changes, unmatched: @translation.unmatched, rows: @rows) - end - - private - - def translate_address(address) - state = @translation.state(address.state, country: address.country) - country = @translation.country(address.country, state: address.state) - updates = {} - updates[:state] = state if state.present? && state != address.state - updates[:country] = country if country != address.country - if address.locality.blank? || address.locality == Address::UNKNOWN_LOCALITY - locality = @translation.locality(city: address.city, state:, country:, county: address.county) - updates[:locality] = locality if locality != address.locality - end - owner = { person_id: (address.addressable_id if address.addressable_type == "Person"), - organization_id: (address.addressable_id if address.addressable_type == "Organization") } - apply(address, updates, owner:, title: address.name, addressable_type: address.addressable_type, addressable_id: address.addressable_id) - end - - def translate_answer(answer) - identifier = answer.form_field.field_identifier - paired = paired_answers.dig(answer.form_submission_id, ANSWER_PAIRS[identifier]) - raw = answer.submitted_answer - translated = if identifier.end_with?("_country") - @translation.country(raw, state: paired).to_s - else - @translation.state(raw, country: paired).to_s - end - return false if translated == raw.to_s || (translated.blank? && identifier.end_with?("_state")) - - owner = { person_id: answer.form_submission&.person_id, form_submission_id: answer.form_submission_id } - apply(answer, { submitted_answer: translated }, owner:, title: answer.form_field.name, - form_submission_id: answer.form_submission_id, field_identifier: identifier) - end - - def apply(record, updates, owner:, title:, **extra) - return false if updates.empty? - - label = record.class.model_name.human - updates.each do |field, after| - @changes["#{label} #{field.to_s.humanize(capitalize: false)}: #{record[field].inspect} → #{after.inspect}"] += 1 - @rows << Row.new(record_type: label, record_id: record.id, person_id: nil, organization_id: nil, - form_submission_id: nil, **owner, field:, before: record[field], after:) - end - @log&.call(" #{label} ##{record.id}: #{updates.map { |field, after| "#{field} #{record[field].inspect} → #{after.inspect}" }.join(', ')}") - return true if @dry_run - - changes = updates.to_h { |field, after| [ field, { before: record[field], after: } ] } - record.update_columns(updates) - @tracker.track("autochange.#{record.class.model_name.singular}", { - resource_type: record.class.name, resource_id: record.id, resource_title: title, changes:, reason: REASON, **extra - }) - true - end - - def address_answers - FormAnswer.joins(:form_field).where(form_fields: { field_identifier: ANSWER_PAIRS.keys }).includes(:form_field, :form_submission) - end - - # { submission_id => { identifier => answer } }, read before any answer is rewritten so - # each translation sees what the person typed beside it. - def paired_answers - @paired_answers ||= address_answers.pluck(:form_submission_id, "form_fields.field_identifier", :submitted_answer) - .each_with_object(Hash.new { |hash, id| hash[id] = {} }) { |(id, identifier, value), map| map[id][identifier] = value } - end - end -end diff --git a/docs/adr/0009-legacy-address-value-translation.md b/docs/adr/0009-legacy-address-value-translation.md index 338ec3df86..2e084ad3ff 100644 --- a/docs/adr/0009-legacy-address-value-translation.md +++ b/docs/adr/0009-legacy-address-value-translation.md @@ -13,7 +13,7 @@ USPS code. Everything written before, and everything other systems send, is free |---|---| | FileMaker (`fm_addresses`) | `AddrsCountry`, `AddrsState`, `AddrsType` | | LMS registration spreadsheet (ongoing) | country, state, address type, phone type columns | -| Portal addresses and form answers from before the dropdowns | `addresses.country`/`state`, `mailing_*`/`organization_*` answers | +| Portal addresses and form answers from before the dropdowns | `addresses.country`/`state`; `mailing_*`/`organization_*` answers (read when an old submission is linked or captured) | The same place arrives many ways — "US", "USA", "United States of America", "Untied States", "California", "Ca.", "CA - California", a province typed in the state field with no country — @@ -22,11 +22,10 @@ list can't be preselected. ## Decision -One translator, `AddressTranslation`, used by every import and by the cleanup of existing -data. Like ADR-0007, translation is **by lookup table, and a value nothing answers to is +One translator, `AddressTranslation`, used by every import. Like ADR-0007, translation is **by lookup table, and a value nothing answers to is reported, never guessed** — it's counted in `#unmatched`. Every stored country ends up a dropdown name or blank: a country nothing answers to is cleared (its old value stays in the -cleanup's Ahoy log, the `fm_*` archive, or the import's report). A state nothing reads is kept, +`fm_*` archive or the import's report). A state nothing reads is kept, since a state is required and regions outside the US are legitimate. - **Country** → a dropdown name. In order: a placeholder ("-", blank, "n/a", "none", @@ -50,23 +49,38 @@ since a state is required and regions outside the US are legitimate. - **Imports, inline:** every FileMaker import builds addresses through `FmAddressAttributes` (the Rolodex import now, the organizations and affiliations imports of ADR-0008), and the LMS registration importer translates its answers. Each reports what it couldn't place. -- **Existing data, once and again:** `AddressServices::TranslateExisting` - (`data:translate_addresses`) rewrites stored address countries and states, and the - country/state form answers, filling a locality only where it's Unknown. It's idempotent, so it runs before and after an import. Every changed - record is an `autochange.*` Ahoy event with its before and after. +- **Existing data:** cleaned once by hand (below). No task rewrites stored addresses. +- **Where answers become addresses:** answers are stored on the address as submitted. Every + form's state and country fields are dropdowns, and the older answers that could still + become an address were fixed in the cleanup. - **Not on save.** The dropdowns own new data; nothing rewrites an address when someone saves a form. ### Form answers -Form answers are otherwise stored verbatim. Country and state answers are the one exception: -they're rewritten so form results and maps count one place once (a country answer nothing -answers to becomes blank), and each rewrite is logged. +Form answers are stored verbatim, as a record of what the person submitted; nothing rewrites +them. The one exception is the hand cleanup below. + +### Prod data cleanup, 2026-10-24 to 2026-10-25 + +Prod data was cleaned by hand over these two days, with addresses as the record: + +- **Addresses:** cities, states, and countries corrected in the portal. +- **Blank address fields filled in** from the rest of each address: country from the state, + county from the ZIP (FileMaker's ZIP table), and an Unknown locality from the city, county, + and the regions FileMaker projects give each California county. Values someone set were + kept. One `autochange.address_fills` Ahoy event points at the saved report of every fill. +- **Some form answers**, because they hadn't become addresses yet: organization address + answers on submissions not yet linked to an organization, which turn into the + organization's address when someone links one. Fixed: country spellings, placeholder values + ("N/A", "-"), and values typed into the wrong field. Each change is an + `autochange.form_answer` Ahoy event with its before value. + +All other answers stay as typed. ## Consequences -- A new spelling turns up in an import's or the cleanup's unmatched report; adding it is a - line in `ALIASES` or `REGIONS`, and a re-run fills the country it had cleared (from the - state, or by hand from the logged value). +- A new spelling turns up in an import's unmatched report; adding it is a line in `ALIASES` + or `REGIONS`. - The `fm_*` archive tables keep FileMaker's raw values. - Every stored country preselects in the dropdowns, so saving a form can't silently blank one. diff --git a/lib/tasks/translate_addresses.rake b/lib/tasks/translate_addresses.rake deleted file mode 100644 index 8c880d45f8..0000000000 --- a/lib/tasks/translate_addresses.rake +++ /dev/null @@ -1,61 +0,0 @@ -# frozen_string_literal: true - -# Rewrite stored address countries/states and country/state form answers as the dropdowns -# store them. See AddressServices::TranslateExisting. -# -# Dry-run by default. Set DRY_RUN=false to write changes (each one logged as an Ahoy event -# credited to Setting.import_user, which a real run requires). -# Every run, dry or not, also saves its report as an ActiveStorage blob — the command to -# read it back is printed at the end. -# bin/rails data:translate_addresses # preview, writes nothing -# VERBOSE=true bin/rails data:translate_addresses # also print every change as CSV rows -# DRY_RUN=false bin/rails data:translate_addresses # translate + log -namespace :data do - desc "Translate stored address countries/states and their form answers to dropdown values. Idempotent. DRY_RUN=false to execute." - task translate_addresses: :environment do - dry_run = ENV["DRY_RUN"] != "false" - verbose = ENV["VERBOSE"] == "true" - import_user = Setting.import_user - if !dry_run && import_user.nil? - abort "No import user chosen — pick one in admin Settings (Imports); every change this run logs is credited to them." - end - report = [] - say = ->(line = "") { puts line; report << line } - # Every change goes in the saved report; the console shows them only when VERBOSE=true. - note = ->(line) { puts line if verbose; report << line } - $stdout.sync = true - - begin - result = AddressServices::TranslateExisting.new(dry_run:, user: import_user).call - - say.("Changes (#{result.changes.values.sum}):") - result.changes.sort_by { |label, count| [ -count, label ] }.each { |label, count| say.(" #{count.to_s.rjust(5)} #{label}") } - result.unmatched.each do |field, values| - next if values.empty? - - outcome = field == :country ? "left blank, was logged" : "kept as typed" - say.("\n#{field.to_s.humanize} values nothing translates (#{outcome} — fix by hand if you know it):") - values.sort_by { |value, count| [ -count, value ] }.each { |value, count| say.(" #{count.to_s.rjust(5)} #{value}") } - end - say.("\nEvery change (CSV#{", in the saved report — VERBOSE=true prints them here" unless verbose}):") - note.(CSV.generate_line(AddressServices::TranslateExisting::ROW_HEADERS, row_sep: "")) - result.rows.each { |row| note.(CSV.generate_line(row.to_h.values, row_sep: "")) } - say.("\naddresses changed: #{result.addresses}") - say.("answers changed: #{result.answers}") - say.(dry_run ? "\nDRY RUN — nothing written. Set DRY_RUN=false to execute." : "\nDone.") - rescue StandardError => e - say.("\nStopped by #{e.class}: #{e.message}#{" — nothing was written; rerun to continue." unless dry_run}") - raise - ensure - begin - blob = ActiveStorage::Blob.create_and_upload!( - io: StringIO.new(report.join("\n")), content_type: "text/plain", - filename: "translate-addresses-#{Time.current.strftime("%Y%m%d-%H%M%S")}#{"-dry-run" if dry_run}.txt" - ) - puts "\nReport saved. Read it back with: bin/rails runner 'puts ActiveStorage::Blob.find(#{blob.id}).download'" - rescue StandardError => e - puts "\nCould not save the report (#{e.class}: #{e.message}) — copy the output above." - end - end - end -end diff --git a/spec/services/address_services/translate_existing_spec.rb b/spec/services/address_services/translate_existing_spec.rb deleted file mode 100644 index 12ec99ec5e..0000000000 --- a/spec/services/address_services/translate_existing_spec.rb +++ /dev/null @@ -1,163 +0,0 @@ -require "rails_helper" - -RSpec.describe AddressServices::TranslateExisting do - let(:us) { "United States" } - - # Stored as old rows hold them, past the model. - def address_with(country:, state:, locality: Address::UNKNOWN_LOCALITY, city: "Springfield") - create(:address, addressable: create(:person), city:).tap do |address| - address.update_columns(country:, state:, locality:) - end - end - - def stored(address) = Address.where(id: address.id).pick(:country, :state, :locality) - - def answer(submission, identifier, value) - field = create(:form_field, form: submission.form, field_identifier: identifier, name: identifier) - create(:form_answer, form_submission: submission, form_field: field, submitted_answer: value) - end - - it "translates stored countries and states, reading a missing country from the state" do - usa = address_with(country: "USA", state: "Texas") - blank = address_with(country: nil, state: "CA - California") - province = address_with(country: "-", state: "BC") - russia = address_with(country: "Russian Federation", state: "Moscow") - - described_class.new.call - - expect(stored(usa).first(2)).to eq([ us, "TX" ]) - expect(stored(blank).first(2)).to eq([ us, "CA" ]) - expect(stored(province).first(2)).to eq([ "Canada", "BC" ]) - expect(stored(russia).first).to eq("Russia") - end - - it "fills a locality only where it's Unknown" do - unknown = address_with(country: nil, state: "MA", city: "Cambridge") - chosen = address_with(country: nil, state: "CA", city: "Fresno", locality: "Central CA") - - described_class.new.call - - expect(stored(unknown).last).to eq(Address::OUTSIDE_CA) - expect(stored(chosen).last).to eq("Central CA") - end - - it "clears a placeholder country when the state names none" do - dash = address_with(country: "-", state: "Central Region") - unknown = address_with(country: "unknown", state: "-") - - described_class.new.call - - expect([ stored(dash).first, stored(unknown).first ]).to eq([ nil, nil ]) - end - - it "clears a country it can't place, logging what it was and reporting it" do - narnia = address_with(country: "Narnia", state: "Central Region") - - result = described_class.new.call - - expect(stored(narnia).first).to be_nil - expect(result.unmatched[:country]).to include("Narnia" => 1) - expect(Ahoy::Event.find_by!(name: "autochange.address", resource_id: narnia.id).properties["changes"]) - .to include("country" => { "before" => "Narnia", "after" => nil }) - end - - it "blanks a country answer it can't place" do - submission = create(:form_submission) - answer(submission, "mailing_state", "-") - country = answer(submission, "mailing_country", "West Africa") - - described_class.new.call - - expect(country.reload.submitted_answer).to eq("") - end - - it "leaves every stored country and country answer on the dropdown" do - address_with(country: "Narnia", state: "Central Region") - address_with(country: "USA", state: "TX") - submission = create(:form_submission) - answer(submission, "organization_country", "Atlantis") - - described_class.new.call - - expect(Address.where.not(country: [ nil, "" ]).pluck(:country) - GeographyHelper::COUNTRIES).to be_empty - expect(FormAnswer.where.not(submitted_answer: [ nil, "" ]).pluck(:submitted_answer) - GeographyHelper::COUNTRIES).to be_empty - end - - it "translates country and state answers using the submission's other answer" do - submission = create(:form_submission) - answer(submission, "mailing_state", "California") - country = answer(submission, "mailing_country", "") - org_country = answer(submission, "organization_country", "Untied States") - org_state = answer(submission, "organization_state", "Texas") - - described_class.new.call - - expect(country.reload.submitted_answer).to eq(us) - expect(org_country.reload.submitted_answer).to eq(us) - expect(org_state.reload.submitted_answer).to eq("TX") - expect(FormAnswer.joins(:form_field).find_by(form_fields: { field_identifier: "mailing_state" }).submitted_answer).to eq("CA") - end - - it "leaves a blank answer blank when its section was skipped" do - submission = create(:form_submission) - org_country = answer(submission, "organization_country", "") - - described_class.new.call - - expect(org_country.reload.submitted_answer).to eq("") - end - - it "logs each changed record as one autochange event, all in one visit" do - usa = address_with(country: "USA", state: "TX", locality: Address::OUTSIDE_CA) - submission = create(:form_submission) - answer(submission, "mailing_state", "TX") - country = answer(submission, "mailing_country", "USA") - - expect { described_class.new.call }.to change(Ahoy::Event, :count).by(2) - - address_event = Ahoy::Event.find_by!(name: "autochange.address", resource_id: usa.id) - expect(address_event.properties).to include("changes" => { "country" => { "before" => "USA", "after" => us } }, - "reason" => "address_translation") - answer_event = Ahoy::Event.find_by!(name: "autochange.form_answer", resource_id: country.id) - expect(answer_event.properties["changes"]).to eq("submitted_answer" => { "before" => "USA", "after" => us }) - expect(Ahoy::Event.where(name: %w[autochange.address autochange.form_answer]).distinct.count(:visit_id)).to eq(1) - end - - it "lists every change with its owner and before/after, on a dry run too" do - usa = address_with(country: "USA", state: "Texas", locality: Address::OUTSIDE_CA) - org_address = create(:address, addressable: create(:organization)).tap { |address| address.update_columns(country: "Usa", state: "TX", locality: Address::OUTSIDE_CA) } - submission = create(:form_submission, person: create(:person)) - answer(submission, "mailing_state", "TX") - country = answer(submission, "mailing_country", "US") - - rows = described_class.new(dry_run: true).call.rows - - expect(rows).to include( - have_attributes(record_type: "Address", record_id: usa.id, person_id: usa.addressable_id, organization_id: nil, - form_submission_id: nil, field: :country, before: "USA", after: "United States"), - have_attributes(record_type: "Address", record_id: usa.id, field: :state, before: "Texas", after: "TX"), - have_attributes(record_type: "Address", record_id: org_address.id, person_id: nil, organization_id: org_address.addressable_id, - field: :country, before: "Usa", after: "United States"), - have_attributes(record_type: "Form answer", record_id: country.id, person_id: submission.person_id, - form_submission_id: submission.id, field: :submitted_answer, before: "US", after: "United States") - ) - end - - it "writes and logs nothing on a dry run, but reports what it would change" do - usa = address_with(country: "USA", state: "TX") - - result = nil - expect { result = described_class.new(dry_run: true).call }.not_to change { [ Ahoy::Event.count, Ahoy::Visit.count ] } - - expect(stored(usa).first).to eq("USA") - expect(result.changes).to include("Address country: \"USA\" → \"United States\"" => 1) - end - - it "changes nothing the second time" do - address_with(country: "USA", state: "Texas") - - described_class.new.call - - expect { described_class.new.call }.not_to change(Ahoy::Event, :count) - end -end diff --git a/spec/tasks/translate_addresses_spec.rb b/spec/tasks/translate_addresses_spec.rb deleted file mode 100644 index b6aa4f8bcc..0000000000 --- a/spec/tasks/translate_addresses_spec.rb +++ /dev/null @@ -1,80 +0,0 @@ -require "rails_helper" -require "rake" - -RSpec.describe "data:translate_addresses" do - before(:all) do - Rails.application.load_tasks unless Rake::Task.task_defined?("data:translate_addresses") - end - - before { Rake::Task["data:translate_addresses"].reenable } - - after { ENV.delete("DRY_RUN") } - - def run_task - original = $stdout - $stdout = StringIO.new - Rake::Task["data:translate_addresses"].invoke - $stdout.string - ensure - $stdout = original - end - - def address_with(country:, state:) - create(:address, addressable: create(:person)).tap { |address| address.update_columns(country:, state:) } - end - - def saved_report = ActiveStorage::Blob.order(:id).last - - def report_text = saved_report.download.force_encoding(Encoding::UTF_8) - - it "previews by default, writing no addresses, and saves the report" do - usa = address_with(country: "USA", state: "TX") - address_with(country: "Narnia", state: "Central Region") - - output = nil - expect { output = run_task }.not_to change { [ Ahoy::Event.count, Ahoy::Visit.count ] } - - expect(usa.reload.country).to eq("USA") - expect(output).to include("DRY RUN", "Narnia", "Report saved") - expect(saved_report.filename.to_s).to match(/\Atranslate-addresses-\d{8}-\d{6}-dry-run\.txt\z/) - expect(report_text).to include("Address country: \"USA\" → \"United States\"", "Narnia") - end - - it "saves every change as CSV rows, printing them only with VERBOSE=true" do - usa = address_with(country: "USA", state: "TX") - row = "Address,#{usa.id},#{usa.addressable_id},,,country,USA,United States" - - output = run_task - - expect(report_text).to include("record_type,record_id,person_id,organization_id,form_submission_id,field,before,after", row) - expect(output).not_to include(row) - - Rake::Task["data:translate_addresses"].reenable - ENV["VERBOSE"] = "true" - expect(run_task).to include(row) - ensure - ENV.delete("VERBOSE") - end - - it "writes with DRY_RUN=false, credits each change to the import user, and saves that run's report" do - import_user = create(:user) - Setting.create!(import_user:) - usa = address_with(country: "USA", state: "TX") - ENV["DRY_RUN"] = "false" - - run_task - - expect(usa.reload.country).to eq("United States") - expect(Ahoy::Event.where(name: "autochange.address", resource_id: usa.id).pluck(:user_id)).to eq([ import_user.id ]) - expect(saved_report.filename.to_s).not_to include("dry-run") - expect(report_text).to include("Done.") - end - - it "won't write without an import user to credit" do - usa = address_with(country: "USA", state: "TX") - ENV["DRY_RUN"] = "false" - - expect { run_task }.to raise_error(SystemExit) - expect(usa.reload.country).to eq("USA") - end -end