diff --git a/AGENTS.md b/AGENTS.md index 050419f5a8..0bce588edd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -49,7 +49,7 @@ This codebase (Rails 8.1) | Directory | Purpose | Count | |---|---|---| | `app/models/` | ActiveRecord models | ~146 files | -| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~155 files | +| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~161 files | | `app/jobs/` | SolidQueue background jobs | 6 files | | `app/models/concerns/` | Shared model modules | 25 concerns | @@ -57,7 +57,7 @@ This codebase (Rails 8.1) | Directory | Purpose | Count | |---|---|---| -| `app/controllers/` | Rails controllers (admin/, events/, home/) | ~93 files | +| `app/controllers/` | Rails controllers (admin/, events/, home/) | ~110 files | | `app/views/` | ERB templates | ~824 files | | `app/decorators/` | Draper decorators for view logic | ~60 files | | `app/policies/` | ActionPolicy authorization rules | ~75 files | @@ -111,7 +111,7 @@ This codebase (Rails 8.1) | `StaffTagging` | Polymorphic join linking a `StaffTag` to the record it tags (`staff_taggable`); `created_by`/`updated_by` (stamped from `Current.user`) record which admin applied and last touched it. Has its own admin edit page (`StaffTaggingsController`, `/staff_taggings/:id/edit`) rendering the shared comments & communications section (`Communicable` + `commentable`) and a tag reassignment select, with communications keyed on the taggable person's email; plus a searchable/filterable index (`/staff_taggings`, `search_by_params` over tag + tagged person's name/email/org) | | `StaffAlias` | A name FileMaker stamped in `EnteredBy`/`ModifiedBy` and the portal `User` it means, so an import can fill `created_by`/`updated_by` (unrelated to `StaffTag`). Names a user **or** is `ignored` (a shared login like "Staff"), never both or neither; unique by name, case-insensitively. Loaded from a staff list kept outside the repo (it pairs real names with accounts) via `data:load_staff_aliases`; resolved by `StaffTranslation` | | `OtherResponse` | A free-text "Other" typed on a form question, captured at submission time (registration, scholarship, bulk payment). Polymorphic `owner`: a **sector** "Other" is owned by the `Person` (promotable into a `Sector`, shown on their profile/edit chip); an **organization_type** "Other" is owned by the `Organization` (stored now, not promotable until `OrganizationType` is a model). `generic` questions aren't captured — that stays searchable in the form answers. `field_identifier` records the question; `kind` is derived. Curated at `/other_responses` (grouped by kind/question): `promote` (sectors only), `keep`, `dismiss`. `dismissed` hides the chip from the profile but stays in the review queue (still promotable later); only `promoted` leaves the queue. Admins deep-link there from a person's chip. | -| `Organization` | Groups with affiliations, addresses, logos via ActiveStorage; self-nests under a parent org (`parent_id` self-FK, `#ancestors`/`#descendants`/`#root`, structural only) per ADR-0004. Has no sectors/age groups of its own: `active_facilitator_sectors` / `active_facilitator_age_groups` roll up its active Facilitator affiliations' primary tags for profile/index/edit chips; search (index filters, tag search, tag matrix, public tag list via its own `sector_names_all` / `category_names_all` scopes) and the populations-served page count primary + additional | +| `Organization` | Groups with affiliations, addresses, phones (`contact_methods`, as on `Person`; `#phone_number` falls back to an active address's `phone`), logos via ActiveStorage; self-nests under a parent org (`parent_id` self-FK, `#ancestors`/`#descendants`/`#root`, structural only) per ADR-0004. Has no sectors/age groups of its own: `active_facilitator_sectors` / `active_facilitator_age_groups` roll up its active Facilitator affiliations' primary tags for profile/index/edit chips; search (index filters, tag search, tag matrix, public tag list via its own `sector_names_all` / `category_names_all` scopes) and the populations-served page count primary + additional | | `Grant` | Funds (polymorphic `funder`: Organization or Person) with eligibility criteria, tasks, deadlines; parent of `Scholarship`. Scholarship totals cannot exceed the grant amount | | `Scholarship` | Award to a `Person`; optionally drawn from a `Grant`, syncs to event registration `Allocation`. Tri-state `agreement_response_status` (pending/accepted/declined) drives the agreement; declined awards zero their allocation and drop out of all totals | | `ScholarshipAgreementResponse` | Append-only history of a scholarship's accept ↔ decline back-and-forth (status, reason, responder, amount at the time); the scholarship's `agreement_response_status` is the denormalized latest row, and `responded_at`/reason are read from the latest response, not stored on the scholarship | @@ -150,7 +150,7 @@ This codebase (Rails 8.1) | `Communicable` | Gives a record the `notifications` association, nested attributes, a default `communications_scope`, and a `before_validation` addressing a hand-logged communication to the record's `communications_email`. Included by every model whose form renders `shared/_comments_and_communications` | | `AuthorCreditable` | Author attribution. Credits are formatted by the credited **person's profile** (`Person#display_name_preference`), not by the record. The record's `author_credit_preference` is the consent snapshot taken at create time and is human-editable only on the author credit divergences page. Anonymity is a one-way latch: the profile or the record can set it, neither can strip it from the other. `credits_to_org` flips the unattributed label to "AWBW Staff"; `credits_creator` (idea models, workshop logs) credits the submitting account's person when no author is named | | `Featureable` | `featured`, `publicly_featured` scopes | -| `FilemakerCoded` | Reads `filemaker_code` as the trimmed, comma-separated **list** it is (Person, Organization, Affiliation, Payment, Workshop, Resource): `#filemaker_codes` for the set a record answers to, `.join_filemaker_codes` to combine any mix of strings/lists into one normalized column value (both dedupers union the codes on merge). `filemaker_archived_in` names the `Fm*` archive table(s) a model's codes point into (`.filemaker_archives`), and `.with_filemaker_code(code)` finds records carrying a code. Nothing should compare the whole column against a single code — `FilemakerHelper#filemaker_code_links` renders one archive link per code, and `fm_archives#show` links back to the portal records carrying its code | +| `FilemakerCoded` | Reads `filemaker_code` as the trimmed, comma-separated **list** it is (Person, Organization, Affiliation, Payment, Workshop, Resource, Address, ContactMethod): `#filemaker_codes` for the set a record answers to, `.join_filemaker_codes` to combine any mix of strings/lists into one normalized column value (both dedupers union the codes on merge, as do `AddressDeduper` and `ContactMethodDeduper`). `filemaker_archived_in` names the `Fm*` archive table(s) a model's codes point into (`.filemaker_archives`), and `.with_filemaker_code(code)` finds records carrying a code. Nothing should compare the whole column against a single code — `FilemakerHelper#filemaker_code_links` renders one archive link per code, and `fm_archives#show` links back to the portal records carrying its code | | `Mentionable` | ActionText mention grouping, plus `connected_records(klass)` — records linked by a mention in either direction (the only Resource↔Workshop link: `Resource#connected_workshops`, `Workshop#connected_resources`) | | `NameFilterable` | Name-based filtering | | `OutsideUsaTypeable` | `typed_outside_usa :state` adds the `_outside_usa` box `shared/_outside_usa` reveals beside a US-state dropdown; a typed value wins whichever is assigned first (ProfessionalLicense, Address, ProfileChangeRequest) | @@ -223,7 +223,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `Analytics::EventReferenceLoader` — Batch-loads the records referenced in a page of Ahoy event properties (association changes, associated records), one query per type, so the activity table's Details column links each to its show page without an N+1 - `Analytics::ContentCreationBreakdown` — Counts content created in the selected time range for the Ahoy charts page's Total / User-generated / Admin-generated cards. Each record is credited by its author's account (super_user → admin), falling back to the creating account when no author is set, so promoted ideas and admin-entered facilitator content land on the user side and the two cards always sum to the total -- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) +- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled with an old-app role — Leader, Liaison, Assistant or "default" (`LegacyRoleAffiliations::LEGACY_TITLES`; report-only; each row badges whether the person has a Facilitator row at that org, lists those rows with how their stretch meets the Leader row's — `#facilitators_for`, `#twin_for`, `#stretch_label` — notes Facilitator rows at other orgs, and opens the person's edit page at the affiliation), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) - `DataHealth::Worklists` — Counts for review pages that live elsewhere (possible duplicates per dedupe page via each `DuplicateFinder` / `Dedupable.name_groups`, and author credit divergence sections, each linked to its section via `?section=` which renders the divergence results with the page so the anchor can scroll), rendered in Data health's lazy `data_health_worklists` frame because the finders load records in Ruby - `ResourceServices::ResolveOrganizations` — Links resources to an organization from the legacy free-text `agency` name (case-insensitive exact match, else creates one with status "Unknown"). Idempotent; drives both the `data:resolve_resource_organizations` rake task (dry-run by default) and the Data health repair ### Business Logic @@ -276,15 +276,15 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `AddressTranslation` — Turns imported and legacy address values into what the address dropdowns store (ADR-0009): `#country(value, state:)` → a `GeographyHelper::COUNTRIES` name (aliases for our data's spellings, ISO codes, plain names, a country read from the state — a US state incl. "CA - California", a `REGIONS` province, or a country named as the state; placeholders like "-"/"n/a"/"unknown" → nil; "CA" next to state CA → US), `#state(value, country:)` → USPS code for a US state (other regions kept), `#address_type` (the archive's/forms' words via `CONTACT_TYPES`), `#locality` (`AddressLocality` on the translated parts), and `#address(attrs)` for a whole hash. Lookup tables only, never guessed (like `TagTranslation`): a country nothing answers to is cleared, so every stored country is a dropdown name or blank, and counted in `#unmatched`; an unreadable state is kept and counted. Used by `FmAddressAttributes` (every FileMaker import) and `EventRegistrationImporter` (country/state answers + address/phone types; unmatched values listed on the import preview and counted in its notice). No save hook — the dropdowns own new data - `FmAddressAttributes` — Turns an `FmAddress` row into `Address` attributes for any FileMaker import (street from `StreetNo` + lines, ZIP or its `PostalCodeLookup`, county from the archive's ZIP table read once, everything clipped to 255), with state/country/type/locality through `AddressTranslation`; nil when the row lacks a street, city or state. An `address_type:` overrides the row's (an organization's addresses are work); `#place(city:, state:, county:)` builds a street-less address for a place the archive records by city alone (a project). `#unmatched` gathers what it couldn't translate. Used by `PersonServices::ImportFromRolodex` (listed in `fm:import_people`'s report) and the FileMaker organizations/affiliations imports (ADR-0008) - `AddressDefaults` — The country, county and locality an address leaves blank, from its other parts: a US state → United States (`Address::DEFAULT_COUNTRY`), a US ZIP → its county (`FmPostalCode.county_for`, FileMaker's ZIP table, skipped when the ZIP is in another state), then `AddressLocality`. Used on create by `OrganizationServices::UpsertAddress` (both org-linking flows and public registration capture), `PersonServices::CaptureFromSubmission` and `ProfileChangeRequests::Apply`; `UpsertAddress` also fills a matched address's blank country/county and blank or Unknown locality, never replacing a value someone set -- `AddressLocality` — Works out which `Address::LOCALITIES` bucket an address falls in from its own city/state/country/county, for a writer that has an address but never asked the question (`PersonServices::ImportFromRolodex`, and through `AddressDefaults` for the rest — a locality a person picked is never overwritten). Reads a named non-US country as **Outside USA**, a non-CA US state as **Outside CA**, a mailing city inside the City of Los Angeles — Los Angeles or a neighborhood (`LA_CITY_MAILING_CITIES`) — as **LA City**, the rest of Los Angeles County as **LA County**, Orange County as **Orange County**, and any other California county by `CALIFORNIA_REGIONS` (the region most FileMaker projects in that county name, plus a hand-checked few) as **Northern/Central/Southern CA**. A state no US state answers to, with no country, is **Outside USA** (the forms only offer US states). Everything else stays **Unknown**: a California address with no or an unplaced county, and a US address whose state no US state answers to. Every comparison goes through `AddressNormalizer`. FileMaker records a `Locality` only on a **project**, so the organizations pass (ADR-0008) carries that across rather than deriving it; a person's own address has none +- `AddressLocality` — Works out which `Address::LOCALITIES` bucket an address falls in from its own city/state/country/county, for a writer that has an address but never asked the question (`PersonServices::ImportFromRolodex`, and through `AddressDefaults` for the rest — a locality a person picked is never overwritten). Reads a named non-US country as **Outside USA**, a non-CA US state as **Outside CA**, a mailing city inside the City of Los Angeles — Los Angeles or a neighborhood (`LA_CITY_MAILING_CITIES`) — as **LA City**, the rest of Los Angeles County as **LA County**, Orange County as **Orange County**, and any other California county by `CALIFORNIA_REGIONS` (the region most FileMaker projects in that county name, plus a hand-checked few) as **Northern/Central/Southern CA**. A state no US state answers to, with no country, is **Outside USA** (the forms only offer US states). Everything else stays **Unknown**: a California address with no or an unplaced county, and a US address whose state no US state answers to. Every comparison goes through `AddressNormalizer`. FileMaker records a `Locality` only on a **project**, so `OrganizationServices::ImportFromFilemaker` carries that across rather than deriving it (falling back here when a project names none); a person's own address has none - `OutsideUsaState` — Resolves a US-state dropdown + "Not on list?" box pair: `.pick(dropdown, typed)` for raw params (CE callout, admin CE registration) and `.form_field_answers(params, scope)`, which lays `form_fields_outside_usa` answers over `form_fields` for every `_state` question (public forms, registration, callout forms). The box itself is `shared/_outside_usa` — a no-JS `
` - `AddressDeduper` — Merges a person's or organization's addresses that are one place into a single address. `.matching_sets` groups by `AddressNormalizer.signature`, then merges a less complete copy (blank ZIP, street, country, …) into the single fuller address whose every part it shares — skipped when it fits more than one; works fullest-first so a chain of partial copies lands on the fullest. Each merge goes through `ModelDeduper#merge`, which repoints the merged address's inbound FKs — `contact_methods.address_id`, `invoices.bill_to_address_id`, `affiliations.organization_address_id` — onto the survivor. Within a set the survivor is the copy still in use — active before inactive, then the primary, then the oldest (`.keeper_order`) — since `Address.active` drives bill-to addresses, an org's program location and the geographic breakdowns; blank columns on the survivor (`CARRIED_COLUMNS` — the street/ZIP/country/county/district/type/locality, `phone` and the three `la_*` districts; "unknown" type/locality counts as blank) are filled from the merged copies. Then settles a single primary on the set holding the signature captured in the deduper's `merge_keeper` (the survivor's own pre-merge primary address, else the deleted record's), demoting the rest. Runs in the people and organizations dedupers' `after_merge`; `AddressDeduper.merge_duplicates(addressable)` runs it standalone (keeping the owner's current primary) for the **Dedupe addresses** admin page (`AddressesController#dedupe_index`/`#dedupe_perform`) +- `ContactMethodDeduper` — After a person or organization merge, merges duplicate contact methods (phone/sms/whatsapp) the generic merge moves onto the survivor wholesale (contact_methods have no unique index). Groups by normalized `kind`+`value`+`contact_type` and merges duplicates via `ModelDeduper#merge` — survivor chosen active before inactive, then primary, then oldest (`#phone_number` reads only active phones), carrying the duplicate's `address_id` up when the survivor has none and unioning `filemaker_code` — then keeps one primary per kind matching the signatures captured in `merge_keeper` (the survivor's own pre-merge primary of that kind, else the deleted record's), demoting the rest. Runs in `after_merge` - `AddressServices::DuplicateFinder` — Lists the people and organizations holding addresses that `AddressDeduper.matching_sets` would merge, for the **Dedupe addresses** page. Compares only within one owner — two orgs at one street are separate records, not duplicates - `PersonServices::ReconcilePrimaryDesignations` — After a person merge, keeps the survivor's own primary sector + primary age range and demotes the duplicate's, which would otherwise leave the kept person with two primaries and trip Person's single-primary validations. The survivor's pre-merge primary ids are captured in the people deduper's `merge_keeper` (they're indistinguishable once merged); reconciliation runs in `after_merge` - `ProfileChangeRequests::Apply` — Applies an auto-appliable `ProfileChangeRequest` when an admin clicks **Approve**: primary email via `UserServices::ProcessEmailChange` (sets `unconfirmed_email`, sends the confirmation), secondary email by writing `email_2` (+ `email_2_type`), the requested email type onto the person (and onto the login with an address change, so the confirm-time sync keeps it), organization name by renaming the person's `primary_organization`, and the structured affiliation categories (title, dates, org rename, remove — a Facilitator row closes the program via `AffiliationServices::CloseProgram` — add an existing org (ending affiliations at the requester's `left_organization_ids`, plus a Facilitator row when the reviewer passes `confer_facilitator:`), a work address picked from the org's list or entered as street/city/state/ZIP). Free-text affiliation requests return `applied: false` (the admin edits by hand, then marks the request resolved). Returns a `Result(applied, message)` - `PersonServices::ReconcileProfessionalLicenses` — After a person merge, merges redundant licenses the generic `(kind, number)` collapse leaves behind. Within each kind, every blank-number license (nil or `""`) merges into a sibling — preferring one with a real number — via `ModelDeduper`, so its CE registrations move onto the survivor. Left as separate rows: two different real numbers of the same kind, and licenses of different kinds (a shared number across kinds is allowed). Runs first in the people deduper's `after_merge`, so any CE it moves is then collapsed by `ContinuingEducationDeduper` - `PersonServices::ReconcileDefaultPayCustomer` — After a person merge, settles the survivor to a single active default Pay customer. Merging two billable people can leave the survivor owning two live `pay_customers` both flagged `default` (the soft-delete unique index doesn't bound them in MySQL, so `ModelDeduper` moves both instead of collapsing), which makes Pay's `payment_processor` (`has_one … default: true`) ambiguous — and a later checkout (`set_payment_processor` clears every default, then re-picks `.first` active customer) could silently flip it. Picks the winner — the customer carrying an active, uncanceled subscription (`Pay::Subscription.active_uncanceled`, so membership autopay reading `payment_processor.subscribed?` off the default keeps working), else the survivor's own pre-merge default (id captured in the people deduper's `merge_keeper`), else the newest — marks it `default: true`, and **soft-deletes** each redundant same-processor sibling (sets `deleted_at`; charges/subscriptions are preserved, not destroyed) so it drops out of the active/checkout scope. A customer with an active subscription is never soft-deleted. No-op unless more than one live customer remains. Runs in `after_merge`. Two people who each carry an active subscription can't be merged — `PeopleController`'s `merge_blocks` blocks it (via `Dedupable`), telling the admin to cancel one first -- `PersonServices::ReconcileContactMethods` — After a person merge, merges duplicate contact methods (phone/sms/whatsapp) the generic merge moves onto the survivor wholesale (contact_methods have no unique index). Groups by normalized `kind`+`value`+`contact_type` and merges duplicates via `ModelDeduper#merge` — survivor chosen active before inactive, then primary, then oldest (`Person#phone_number` reads only active phones), carrying the duplicate's `address_id` up when the survivor has none — then keeps one primary per kind matching the signatures captured in `merge_keeper` (the survivor's own pre-merge primary of that kind, else the deleted person's), demoting the rest. Runs in `after_merge` - `RichTextMigrator` — Rich text migration utility - `EventRegistrationImporter` — Bulk-imports registrants into one existing event (live or on-demand, training or not) from an uploaded CSV (stdlib `csv`). Only FirstName/LastName are required; `Columns::TEMPLATE_HEADERS` backs a downloadable CSV template (one spelling per readable column, `#template`, spec-held to `HEADER_ALIASES`) and `EventRegistrationImporter::Columns` maps loosely-matched headers (case/spacing/punctuation ignored) for status, Day 1–5, registered date, person extras, org + org address + position, mailing address/phone, sectors/age groups (by name), CE license, scholarship, payment, and follow-up flags, and the preview lists unrecognized headers. **Person:** a `FileMakerCode`/`ParticipantID` the Rolodex import already recorded on somebody matches them outright (`Person.with_filemaker_code`), and makes the name columns optional for that row; otherwise, with an email, `PersonMatcher` (PublicRegistration's rule); without, `PersonNameMatcher` (name at the row's org, else a name only one person has); a miss creates the person (attributed to the importing admin), an email-less one with `Person#next_free_placeholder_email` (numbered per name, so the same placeholder serves two different names). **Status:** the Status column (blank → `attended`, or derived from Day N columns when only those are filled), applied only where the portal holds no deliberate outcome — an attended / cancelled / no-show / transferred-out / incomplete-attendance registration keeps its status and the sheet's lands as a comment on it (`PORTAL_WINS_STATUSES`, topic "Spreadsheet import"), with the row's Day columns left out since they describe the outcome that lost, so one person still holds one registration per event; attended marks every day complete; a status the import does apply is saved without emails (`skip_cancellation_emails`). **Org:** `OrganizationServices::NameMatcher` (exact, then punctuation/legal-suffix tolerant), then `OrganizationServices::CaptureFromSubmission` (website/type/work address/job + facilitator affiliations) with `answers:` and `overwrite: false`; an unmatched name lands in the link-organization queue via the import's OWN `FormSubmission` (role registration, marked `imported_from`, created alongside never in place of a real one, carrying the answers this event's form asks). **Profile:** `PersonServices::CaptureFromSubmission` with `answers:` — a matched person's values are kept (`overwrite: false` only fills blanks and adds new addresses/phones as non-primary); what fill-only declined comes back on `#declined` (and, for the org, from `OrganizationServices::ProfileDiff`) and is reported as a row warning **and** a comment on that person/organization, so a curated sheet's value is never dropped silently. **CE** via `ContinuingEducationRegistration.enroll!` when the event offers hours and the row asks (`Columns.requested?` — a Yes/No cell or FileMaker's `CEHours` count). `Notes`/`Comments` become comments on the registration; a row whose `NoOfPersons` is above 1 stands for a group: the person it names is registered, the payment becomes a bulk payment for that many heads with the unnamed ones padded as `UNNAMED_ATTENDEE` ("TBD") for an admin to add and link, and the count is noted on the registration. **Money:** planned per row (`set_money_preview` records what's owed), then applied once every row is read, so rows sharing a `BulkPayment` reference — or one check number — become **one payment split across their registrations** rather than a full payment each (`apply_payment_batch`; an amount covering the group's whole bill is read as the payment, anything less as a share of it, and the payer is the organization they share, else the first registrant); a group payment also gets a role `bulk_payment` `FormSubmission` (attendees + linked registrations) so it lists on the event's **Bulk payments** page, reusing `Event#bulk_payment_form` or attaching a `Form.standalone` bulk payment form, creating one only when the install has none. Order per member: an accepted Scholarship, then the payment (registration, then a CE *the row asked for* — a CE it doesn't mention is left untouched), then a Discount comping what a paying row still owes. One calculation drives both the preview and the run; a row the scholarship already covers reports `:nothing_owed` and records no payment. Idempotent, sends **no emails**, never creates organizations. Requires a registration form with an org field (`.importable?`). Runs in a transaction with a `dry_run` mode that rolls back for the preview; returns a `Result` + per-row `RowPreview`s. Drives `EventRegistrationImportsController` (`event_registrations/import`, admin-only via `EventRegistrationPolicy#import?`; linked from the registrants page's Bulk actions menu and admin home's Actions) - `StoryImporter` — Imports stories from the curated stories spreadsheet (the columns this importer's own preview produces, reviewed/edited by staff and re-uploaded; headers in `StoryImporter::TEMPLATE_HEADERS`). Every importable row becomes a Story; the `status` column drives publish state ("Published story" vs "Draft story"), whether a StoryIdea is also promoted ("… + story idea"; organization optional), and `Skipped — …` rows. Column headers may be the curated sheet's human labels (`COLUMN_ALIASES`, any order) or the snake_case keys, and a duplicate human-label header row is skipped. Trusts the sheet's resolved taxonomy — looks up `sectors`/`window_type` (exact match; unknown window type skips the row) by name and `categories` as "Type: Name" — an age group (Children/Teens/Adults/Elders) always resolves to its `AgeRange` category whichever type the sheet filed it under, so the retired StoryPopulation age twins (`Teens_`) are never tagged; either kind of audience can be the starred `primary_story_population`, and flags `is_primary` on the join rows matching `primary_sector` / `primary_story_population` (tagging them even if absent from the sectors/categories lists); taxonomy, grants, or facilitator names that can't be matched are preserved as Comments on the story rather than invented. Strips HTML from titles and renames a duplicate title to `[COPY N]` instead of skipping. Resolves up to two author People (first + optional co-author) from parallel `facilitator_*`/`co_facilitator_*` columns, each with its own `name_display`/`anonymous` — matched through the shared `PersonMatcher` (email in any of the person's three addresses + last name + stored-or-legal first name), falling back to an exact name for a row with no email — `AWBW` rows pull the real name out of the last-name column and flag the profile anonymous; a co-author sticks only behind a distinct first author. Credit is synced to each author's profile and left blank on the story (both `author_credit_preference` and `co_author_credit_preference`) unless that person's rows disagree. Both authors and the workshop link carry onto the promoted StoryIdea as well; the workshop links through the `story_workshops`/`story_idea_workshops` join (matched Workshop or free-text external title). Links `grants` through the author's Scholarship, attaches comma-separated `professional_licenses` (by kind) to the author, records the source `wp_id` as a Comment, converts `content` via wpautop, enqueues a `StoryAssetImportJob` per story to download its `image_urls` (with per-image `image_alt_titles`) in the background, and returns a row-by-row preview for the dry-run interstitial @@ -304,6 +304,8 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `EventRegistrationServices::CalloutFormSubmission` — Records a registrant's answers to one of the forms a registration-ticket callout delivers inline (`call(registration:, callout:, form:, form_params:)`, returns the service instance with `#submission`/`#profile_changes`). Find-or-creates one `FormSubmission` per `(registrant, form, event)` carrying the form's own role — a re-submit edits the answers in place via `FormSubmission#persist_answer`. That a callout collected it is recorded separately in `FormSubmission#metadata` (`collected_via: "callout"`, `collected_via_callout_id`) via `#record_callout_collection!` — the flag (`#collected_via_callout?`), not the role, tells a callout-collected submission apart. There is no survey-role flag — the extra side effects self-gate on the form's own fields, so they no-op on a form that doesn't ask those questions: fields carrying the `anonymous_contributions` / `display_name_preference` identifiers write through to the `Person` via `PersonServices::SyncSharingPreferences` (the same write-through the registration form runs), exposing `#profile_changes` so the controller Ahoy-tracks real edits; and submitting any form on the **`scholarship`** callout stamps a recipient's `post_survey_completed_at`. On an **anonymous** form (`Form#anonymous?`) the registrant is credited only when the answers give first name, last name, and email (`Form#identified_by?`); otherwise the answers are saved with no person (a credited submission edited down to a partial identity is detached), no profile write-through runs, a `FormCompletion` records that the registrant finished, and a later submit is refused (`#already_completed?`) — as is one missing a required answer (`#incomplete?`). Drives `Events::CalloutsController#callout`/`#submit_callout` (the person-first `registration/:slug/forms/:callout_id` page and its `.../:form_id` submit) - `TagTranslation` — Turns a legacy word for who someone serves into tags: `#resolve` answers with both the sector ids and the age-group ids a term means, since the vocabularies cross (`Elder Abuse` is a life experience that means the Elders *age range*). A tag's own name resolves only while it's **published** — a retired tag is unpublished, not deleted — but a recorded `TagAlias` may point at a retired one deliberately (FileMaker's `Poverty`). Splits a repeating field on line breaks and semicolons — **not commas**, which the archive's own wording contains — and drops a trailing parenthetical. Built once and reused (it reads the whole vocabulary up front); a term nothing answers to lands in `#unmatched` rather than a guess. Primacy is the caller's call, since it follows from the field a term sat in - `TagAliasSeeder` — Records what the FileMaker archive, the older registration/scholarship forms, Story Share and the portal's earlier sector list each called a tag the portal now names its own way (**ADR-0007**) as `TagAlias` rows, so the translation is data an admin can extend and a record of what a tag used to be called. Terms AWBW decided are recorded as-is; semantic reads of the archive's hand-typed free text are marked **`proposed`** for a pass with staff (they still resolve, so no import waits on it), and text naming an organization or a facilitator's own circumstances is left alone. Idempotent on (term, tag) and never touches a term added by hand; a term whose tag isn't published is reported, not created. Shared with `data:seed_tag_aliases` +- `FilemakerValues` — How every FileMaker archive import reads a raw field: `parse_date` (the archive's mixed formats, two-digit years given a century), `flag?`, `contact_type` (archive type words → the slice a model accepts), `email_addresses` (the valid addresses inside a free-text field), and `clip` to a varchar. Module functions, also included by the importers +- `FilemakerCodeIndex` — Which records of a `FilemakerCoded` model carry each code, read once so a bulk import looks codes up in memory; `#add` records codes the import writes as it goes - `StaffTranslation` — Turns a FileMaker `EnteredBy`/`ModifiedBy` value into a user id through the recorded `StaffAlias` rows (`#user_id_for`), case- and spacing-insensitive. Built once and reused (reads every alias up front); a name nothing answers to lands in `#unmatched` unless it's recorded as ignored - `StaffAliasLoader` — Records `StaffAlias` rows from a CSV/TSV staff list (`staff`, `user_ids_by_email`, `portal_user_email`, optional `ignored` columns), resolving the user by id then email. Idempotent: a name already recorded is left alone; a name with no user yet is reported in `#unresolved`, not recorded. Shared with `data:load_staff_aliases` - `ProfileCategoriesToSectors` — Gives people an additional sector for each profile-specific category tag (the retired workshop settings), reading each category name through `TagTranslation` so the tag aliases decide what it means. Idempotent, keeps the category tags and any existing sector's primary flag; organization tags are only counted (their sectors roll up from facilitators). Drives `data:convert_profile_categories_to_sectors` (DRY_RUN) @@ -314,8 +316,8 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `OrganizationServices::NameMatcher` — Resolves a typed organization name tolerating spreadsheet spelling drift: exact (case-insensitive), then `DuplicateFinder.normalized_name` (punctuation), then `DuplicateFinder.suffixless_name` (Inc/LLC/…); a loose match counts only when it fits one org. Indexes names once per instance. Used by the registration CSV import - `OrganizationServices::CaptureFromSubmission` — The shared org-capture for **both** the event registration (`PublicRegistration`) and standalone (`PublicFormSubmission`) flows: resolves the submitted organization by name (`FormField::ORGANIZATION_NAME_FIELD_IDENTIFIER`, case-insensitive per the column collation; never creates one — an unmatched name is left for an admin), syncs its website/type (`SyncProfile`, latest-wins) and work address (`UpsertAddress`, which adds another work address rather than rewriting a different one). `AffiliationServices::CreateFromRegistration` then mints the person's job affiliation from the typed position, plus the standing Facilitator one when `facilitator_training:` — which only an event supplies (`@event.facilitator_training`), so a standalone form gets the job affiliation alone, the same as registering for a non-training event. **Agreement-role submissions never come through here** — they run `LinkSubmittedOrganization`, which confers the scenario's Facilitator affiliation. Returns the org + the autofill changes, which the event flow records on its registration↔org link. `answers:` (identifier → value) stands in for form + params and `organization:` for the name lookup when the caller resolved them (the CSV import); `overwrite: false` only fills the org's blanks - `PersonServices::CaptureFromSubmission` — The shared person-capture for **both** the event registration (`PublicRegistration`) and standalone (`PublicFormSubmission`) flows, run for a newly-created **and** an existing person, so the person smart fields on the Smart form settings page behave identically across both forms. Fills identity extras (pronouns, name `pronunciation`, legal first name, primary email type, secondary email + type), the profile (`racial_ethnic_identity` + `SyncSharingPreferences`), the mailing address, the phone contact method, and sector/age tags (on the person only — organizations roll theirs up from their active facilitators). A non-blank answer overwrites what's on file (newest submission wins); a blank never clobbers. Sector and age tags follow one rule: submitting both a primary and additional set **overwrites** the person's tags with the latest (`replace: true`), submitting only a primary **reassigns** the primary and leaves the rest (additive). Self-gates on each field identifier, so a form that doesn't ask a question no-ops for it. `answers:` (identifier → value) stands in for form + params (the CSV import); `overwrite: false` keeps the person's name, primary address/phone, and primary tags, only filling blanks and adding new entries beside them -- `PersonServices::ImportFromRolodex` — Creates people from FileMaker Rolodex archive rows (`FmRolodex`), carrying the row's ID as `filemaker_code` plus every field `FmRolodex::PORTAL_FIELDS` maps onto a person (profile fields, email type — a precedence word like "Preferred" records none, since the email's own slot already says that — facilitator start → `member_since`, service areas, `Settings`, `LifeExperiences` and `PrimaryAgeGroup` → sectors and age groups translated through `TagTranslation` (ADR-0007; primary fields → primary, the rest additional; one primary sector and one primary age group per person — each kind takes it from the field that owns it, and an answer naming several lets the other field answer instead) via `tag_sectors` / `tag_age_groups`, and the archive's own entry/modification dates as `created_at`/`updated_at`, plus `EnteredBy`/`ModifiedBy` as `created_by`/`updated_by` through `StaffTranslation`, falling back to `Setting.import_user` (picked on `/admin/settings` → Imports) when the name is blank, a shared login, or unresolved, so every person carries both stamps; a linked person keeps any stamps they already have; a name no `StaffAlias` answers to is reported in `#unmatched_staff`). Plus every address and phone the row answers to (`FmRolodex#addresses` / `#phone_numbers`, primary first; both tables are read once via `FmRolodex.contact_index` + `#preload_contacts`), except work addresses (the worksite one or any typed Work), which the affiliations import will bring in. Links an existing person through `PersonMatcher`, trying each address the row holds in `EMail` (`EmailPrimary` is a flag, never an address: `1` → the first address becomes `email`, a second one the person doesn't already hold `email_2`; `2` → the address and `EmailType` land in `email_2`/`email_2_type`; for a row with no address at all, a same-name person with no email) by adding the row's ID to their `filemaker_code` list — every match when several answer, and alongside any code they already carry — and fills them from the row: blank fields take the archive's value, missing addresses/phones/tags are added (an existing primary stays primary), and a field the archive disagrees on keeps the portal value and is noted as a `Comment` on the person and in `#differences`; creates a new person when nobody matches, even when the email belongs to a differently named person; skips agencies, soft-deleted, and nameless rows. A match that doesn't validate is reported under `failed` and left out, so the row's other matches still link; `relink: true` reads named rows past the idempotence guard to pick one up once the person is valid. Exposes `TagTranslation#unmatched` as `#unmatched_tags`. Idempotent on `filemaker_code`. Driven by `fm:import_people` -- `PersonServices::UndoRolodexImport` — Reverses one `fm:import_people` run from the Ahoy lifecycle events `ImportFromRolodex` tagged with its `import_run`: newest first, deletes each record the run created and reverts each field it filled, unless the record was edited since (its `updated_at` is newer than the event) or the field has changed — those are kept and reported. Runs under `no_touching` so deleting a person's address doesn't make the person look edited. Driven by `fm:undo_import_people` +- `PersonServices::ImportFromRolodex` — Creates people from FileMaker Rolodex archive rows (`FmRolodex`), carrying the row's ID as `filemaker_code` plus every field `FmRolodex::PORTAL_FIELDS` maps onto a person (profile fields, email type — a precedence word like "Preferred" records none, since the email's own slot already says that — facilitator start → `member_since`, service areas, `Settings`, `LifeExperiences` and `PrimaryAgeGroup` → sectors and age groups translated through `TagTranslation` (ADR-0007; primary fields → primary, the rest additional; one primary sector and one primary age group per person — each kind takes it from the field that owns it, and an answer naming several lets the other field answer instead) via `tag_sectors` / `tag_age_groups`, and the archive's own entry/modification dates as `created_at`/`updated_at`, plus `EnteredBy`/`ModifiedBy` as `created_by`/`updated_by` through `StaffTranslation`, falling back to `Setting.import_user` (picked on `/admin/settings` → Imports) when the name is blank, a shared login, or unresolved, so every person carries both stamps; a linked person keeps any stamps they already have; a name no `StaffAlias` answers to is reported in `#unmatched_staff`). Plus every address and phone the row answers to (`FmRolodex#addresses` / `#phone_numbers`, primary first; both tables are read once via `FmRolodex.contact_index` + `#preload_contacts`), except organization addresses (`FmRolodex#organization_address?`: the worksite one, any typed Work, and every address on an agency's card), which `AffiliationServices::ImportFromFilemaker` and `OrganizationServices::ImportFromFilemaker` place. Each address and phone carries its archive row's ID (`AddrsID` / `PhoneID`) as its own `filemaker_code`; a row the person already holds (same address or number) adds its code to that record instead of a copy. Links an existing person through `PersonMatcher`, trying each address the row holds in `EMail` (`EmailPrimary` is a flag, never an address: `1` → the first address becomes `email`, a second one the person doesn't already hold `email_2`; `2` → the address and `EmailType` land in `email_2`/`email_2_type`; for a row with no address at all, a same-name person with no email) by adding the row's ID to their `filemaker_code` list — every match when several answer, and alongside any code they already carry — and fills them from the row: blank fields take the archive's value, missing addresses/phones/tags are added (an existing primary stays primary), and a field the archive disagrees on keeps the portal value and is noted as a `Comment` on the person and in `#differences`; creates a new person when nobody matches, even when the email belongs to a differently named person; an agency card naming a contact person becomes that person; skips soft-deleted and nameless rows (a nameless agency card is the agency itself, which the organizations pass reads). Reads raw values through `FilemakerValues` and addresses through `FmAddressAttributes`. A match that doesn't validate is reported under `failed` and left out, so the row's other matches still link; `relink: true` reads named rows past the idempotence guard to pick one up once the person is valid. Exposes `TagTranslation#unmatched` as `#unmatched_tags`. Idempotent on `filemaker_code`. Driven by `fm:import_people`, and by `fm:import_organizations` for agency cards only +- `PersonServices::UndoRolodexImport` — Reverses one FileMaker import run (`fm:import_people` or `fm:import_organizations`) from the Ahoy lifecycle events it tagged with its `import_run`: newest first, deletes each record the run created and reverts each field it filled, unless the record was edited since (its `updated_at` is newer than the event) or the field has changed — those are kept and reported. Runs under `no_touching` so deleting a person's address doesn't make the person look edited. Driven by `fm:undo_import_people` (alias `fm:undo_import`) - `PersonServices::SyncSharingPreferences` — Writes the two content-sharing answers (`anonymous_contributions`, `display_name_preference`) onto the `Person`, mapping each submitted option label back to its stored value; an unanswered or unrecognized answer leaves the profile alone, so a form that doesn't ask the questions is a no-op and every flow can call it unconditionally. The person-side counterpart to `OrganizationServices::SyncProfile`. Shared by `PersonServices::CaptureFromSubmission` (so a submission's answers land in the "what this submission changed" audit via the Ahoy trail) and `CalloutFormSubmission` (whose `#profile_changes` re-reports the same edits for its own Ahoy tracking) - `EventRegistrationServices::TransferContinuingEducation` — Splits/relocates a registrant's CE when they transfer events (issue #1944): a simple forward transfer leaves a paid, zero-hours **stub** on the source (its payments count at the original event) and creates a **live** record on the destination carrying the hours and the outstanding balance; when the reg being transferred out is itself a transfer-in (a collapsing double transfer, or a transfer back to the origin) its live record is relocated forward — merging back into the origin's stub — instead of split again, so no third record appears. Runs inside the transfer transaction, after the destination is saved and before a collapsing middle reg is destroyed - `EventRegistrationServices::ReconcileMergedContinuingEducation` — Run from the event-registration deduper's `after_merge`. A CE registration moved from the deleted registration still points at that (deleted) registrant's license — invalid, since the registration now belongs to the kept registrant — so this repoints each to the kept registrant's equivalent license (same kind + number, found or created), then merges the resulting duplicates into one with `ContinuingEducationDeduper` so payments are preserved @@ -329,6 +331,8 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations +- `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Used by the `LegacyRoleAffiliations` Data health check to find each row's Facilitator twin +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing row titled with an old-app role (Leader, Liaison, …) matches as a Facilitator; a row whose project wasn't imported lands at the org on the person's Rolodex card (its `OrgID`, else its typed name via `NameMatcher`, created when nothing matches); a row naming no person — or only an agency's own card — gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled) only when its dates agree — same or blank on one side, still start-before-end — and it records no other row of that table; differing dates come in as a new affiliation. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) @@ -353,6 +357,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Organizations +- `OrganizationServices::ImportFromFilemaker` — The FileMaker organizations pass (ADR-0008): each `FmOrganization` and its `FmProject`s become portal orgs — an org with exactly one project stays one org carrying both codes, otherwise the projects nest under it (`parent`). Matches by code (`FilemakerCodeIndex`), then exact name only to an org carrying no code from that table (a namesake gets `"Name (code)"`). Fills blanks only (website, description, mission, `organization_type` from `OrgType`/`AgencyType` via `ORGANIZATION_TYPE_PATTERNS` — any other word lands as Other with the word in `organization_type_other`, tallied in `#unmatched_types`), noting differences as comments; adds the org's `fm_addresses` and `fm_phone_numbers` (phones as `contact_methods`, the archive's primary first), each carrying its row's ID as `filemaker_code` (an address or number the org already holds takes the code instead of a copy), and its `fm_email_addresses`; copies each project's `Locality`/`District`/`SPA`/`LASupervisorialDistrict` onto the program's address in its city (a city-only one when none); folds a merged-away project's code (`MergerWithID`) into the survivor; and reads agencies' own Rolodex cards (website, mission, addresses, and email when no contact person is named) onto their org, creating it by name when nothing answers. Driven by `fm:import_organizations` - `OrganizationServices::AutofillChange` — One thing a registrant's answers wrote onto an org: `field`, human `label`, the `value` that landed in it, `change_type` (`"new"` filled a blank / `"update"` replaced something), `previous_value` on an update, and an optional `scope` naming which work address. `change_type` is derived from `previous_value` so the two can't disagree, but is written to the JSON anyway so the stored row reads on its own. Round-trips through the `form_autofill_changes` column; `#description` renders "Website" or "ZIP on the Austin work address" for a flash, while the linking page shows the value and what it displaced. `#key` is (field, scope), which is what makes a later submission's value replace an earlier one instead of stacking. Drops a stored entry with no `field` rather than raising — it's a display aid, not a ledger. **Not a substitute for the Ahoy audit trail**, which records every before/after on Organization and Address independently; this is the denormalized, registration-scoped copy the linking page can render without correlating events - `OrganizationServices::AddressMatcher` — Given a submitted city/state/street/ZIP, finds the org address it corresponds to so `UpsertAddress`/`ProfileDiff` update the right one instead of duplicating. Tries same city + state (preferring one whose street also matches, so an org with several offices in one city doesn't get the wrong one rewritten), then same city + street (unifies an address with a submission that merely adds the state/country we were missing), then — across cities — same street + same ZIP, which reunites an address whose city was renamed ("Ventura" / "San Buenaventura") without merging two offices that share a street name in different towns. Every comparison goes through `AddressNormalizer`, so known spellings ("St. Louis" / "Saint Louis", "TX" / "Texas", "N Main St" / "North Main Street") match at every step. Module function, used by both address services - `OrganizationServices::UpsertAddress` — Find-or-create an organization's "work" address from a registrant's submitted organization fields (street/city/state/zip/country), matching via `AddressMatcher`; updates the match, else adds a new address; never demotes the org's existing primary (a registrant's address becomes primary only when the org has none yet). `overwrite: true` (default, public flow) replaces the matching address's details; `overwrite: false` (admin org-linking) fills only its blank fields so a conflicting saved address is kept and flagged instead. City and state are always fill-only (never flipped, in either mode), so a cross-city ZIP match keeps the saved city and lets `ProfileDiff` flag the difference; that fill still earns its keep because a legacy row can hold `""` in either column, and the street + ZIP match is the one path that reaches such a row to repair it. Returns a `Result` (`address`, `created`, and `changes` — an `AutofillChange` per field actually written, carrying the value) so callers can report what was really saved. Every field is trimmed before it's stored, so a padded answer isn't later read back as a difference from itself. Tolerates optional answers: a skipped street/ZIP is stored blank (both are `NOT NULL` columns), and `address` is nil — nothing saved — when there's no city, or no state and no existing address to update, rather than failing the whole registration. Shared by `PublicRegistration` and the admin org-linking actions @@ -601,7 +606,7 @@ RuboCop linting on PRs and pushes to main. ## Rake Tasks -Located in `lib/tasks/` (18 files): +Located in `lib/tasks/` (19 files): - `dev.rake` — Development database seeding from XML/CSV - `rhino_migrator.rake` — Rich text editor migration - `attachment_report.rake` — Attachment reporting @@ -616,6 +621,7 @@ Located in `lib/tasks/` (18 files): - `backfill_notification_stamps.rake` — Backfill notifications' `created_by_id`/`updated_by_id` from `sender_id`, nil-only (`data:backfill_notification_stamps`) - `fm_import.rake` — Load the FileMaker CSV exports into the `fm_*` archive tables (`fm:import`, reads each CSV from the `FM Archive` resources, batch `insert_all`, skips rows whose key is already stored) - `fm_import_people.rake` — Create people from the FileMaker Rolodex archive (`fm:import_people`, via `PersonServices::ImportFromRolodex`, links existing by name + email, idempotent, DRY_RUN; VERBOSE=true lists every row, FM_IDS=a,b re-runs just those rows past the idempotence guard; commits every 1,000 rows, prints progress, saves the full report as an ActiveStorage blob even on a crash, and aborts when no `TagAlias` rows exist or no `Setting.import_user` is chosen; logs every record it creates or fills as activity under a run name (`Current.source = "fm_import_people"`), which `fm:undo_import_people RUN=…` reverses via `PersonServices::UndoRolodexImport`) +- `fm_import_organizations.rake` — Create organizations and affiliations from the FileMaker archive after `fm:import_people` (`fm:import_organizations`: `PersonServices::ImportFromRolodex` over `FmRolodex.agency_cards` — the contact people `fm:import_people` skipped — then `OrganizationServices::ImportFromFilemaker`, then `AffiliationServices::ImportFromFilemaker`; idempotent, DRY_RUN rolls back as a whole while a real run commits per batch, VERBOSE=true lists every row; prints progress, the skipped/failed rows, differences, unmatched org types and the Personnel status/role tallies, and saves the report as an ActiveStorage blob; aborts with no `Setting.import_user`; logs under a run name that `fm:undo_import RUN=…` reverses) - `resolve_resource_organizations.rake` — Link resources to organizations from the legacy `agency` free-text name (`data:resolve_resource_organizations`, match-or-create, idempotent, DRY_RUN) - `seed_tag_aliases.rake` — Record the legacy sector and age-group terms from ADR-0007 as tag aliases (`data:seed_tag_aliases`, via `TagAliasSeeder`, idempotent; also part of `db:seed`) - `load_staff_aliases.rake` — Record FileMaker staff names → portal users as staff aliases from `FILE=` (`data:load_staff_aliases`, via `StaffAliasLoader`, idempotent) diff --git a/app/controllers/fm_archives_controller.rb b/app/controllers/fm_archives_controller.rb index 901e525cc5..52eb42073f 100644 --- a/app/controllers/fm_archives_controller.rb +++ b/app/controllers/fm_archives_controller.rb @@ -63,7 +63,7 @@ class FmArchivesController < ApplicationController { label: "Other", icon: "📎", tables: %w[notes] } ] - PORTAL_MODELS = [ Person, Organization, Affiliation, Payment, Workshop ].freeze + PORTAL_MODELS = [ Person, Organization, Affiliation, Payment, Workshop, Address, ContactMethod ].freeze def index authorize! :fm_archive diff --git a/app/controllers/organizations_controller.rb b/app/controllers/organizations_controller.rb index 760f51e7b4..25a42ee9f0 100644 --- a/app/controllers/organizations_controller.rb +++ b/app/controllers/organizations_controller.rb @@ -207,7 +207,7 @@ def populations_served def set_organization @organization = Organization.includes( - :organization_status, :windows_type, :addresses, + :organization_status, :windows_type, :addresses, :contact_methods, :categorizable_items, { comments: [ :created_by, :updated_by ] }, affiliations: { person: [ :user, :affiliations, *Organization::PEOPLE_TAGGINGS ] } @@ -238,6 +238,7 @@ def organization_params ], comments_attributes: [ :id, :topic, :body, :flagged, :_destroy ], notifications_attributes: Notification::PERMITTED_LOG_ATTRIBUTES, + contact_methods_attributes: [ :id, :kind, :contact_type, :value, :primary, :inactive, :_destroy ], addresses_attributes: [ :id, :address_type, @@ -275,9 +276,11 @@ def dedupe_config merge_keeper: ->(keep, delete) { keep.filemaker_code = Organization.join_filemaker_codes(keep.filemaker_code, delete.filemaker_code) @keeper_primary_address_signature = AddressDeduper.primary_signature(keep, delete) + @keeper_primary_contact_signatures = ContactMethodDeduper.primary_signatures(keep, delete) }, after_merge: ->(keep) { AddressDeduper.new(keep, primary_signature: @keeper_primary_address_signature).call + ContactMethodDeduper.new(keep, primary_signatures: @keeper_primary_contact_signatures).call }, belongs_to_options: -> { { diff --git a/app/controllers/people_controller.rb b/app/controllers/people_controller.rb index 5f9ef75eee..83d6e0c417 100644 --- a/app/controllers/people_controller.rb +++ b/app/controllers/people_controller.rb @@ -429,7 +429,7 @@ def dedupe_config delete.age_range_categorizable_items.find_by(is_primary: true)&.category_id @keeper_default_pay_customer_id = keep.pay_customers.active.find_by(default: true)&.id @keeper_primary_address_signature = AddressDeduper.primary_signature(keep, delete) - @keeper_primary_contact_signatures = PersonServices::ReconcileContactMethods.primary_signatures(keep, delete) + @keeper_primary_contact_signatures = ContactMethodDeduper.primary_signatures(keep, delete) @primary_login_id = primary_login(keep, delete)&.id }, # Reconcile what the generic merge leaves inconsistent on the kept person: @@ -456,7 +456,7 @@ def dedupe_config preferred_customer_id: @keeper_default_pay_customer_id).call AddressDeduper.new(keep, primary_signature: @keeper_primary_address_signature).call - PersonServices::ReconcileContactMethods.new(keep, + ContactMethodDeduper.new(keep, primary_signatures: @keeper_primary_contact_signatures).call sync_primary_login_email(keep) }, diff --git a/app/helpers/filemaker_helper.rb b/app/helpers/filemaker_helper.rb index d153b30c5a..2b2baaf7ee 100644 --- a/app/helpers/filemaker_helper.rb +++ b/app/helpers/filemaker_helper.rb @@ -44,6 +44,8 @@ def filemaker_portal_link(record) when Affiliation then [ "#{record.person&.full_name} at #{record.organization&.name}", edit_affiliation_path(record) ] when Payment then [ [ dollars_from_cents(record.amount_cents), person_name(record.payer) ].compact.join(" from "), payment_path(record) ] when Workshop then [ record.title, workshop_path(record) ] + when Address then [ "#{record.addressable.name}: #{record.street_address}", polymorphic_path(record.addressable) ] + when ContactMethod then [ "#{record.contactable.name}: #{record.value}", polymorphic_path(record.contactable) ] end link_to text, path, class: button_classes(:primary) diff --git a/app/models/address.rb b/app/models/address.rb index c8304c562b..19cdf984b5 100644 --- a/app/models/address.rb +++ b/app/models/address.rb @@ -1,5 +1,7 @@ class Address < ApplicationRecord + include FilemakerCoded include OutsideUsaTypeable + filemaker_archived_in FmAddress belongs_to :created_by, class_name: "User", optional: true belongs_to :updated_by, class_name: "User", optional: true diff --git a/app/models/affiliation.rb b/app/models/affiliation.rb index 0a9a4a1168..fdbe26ff3a 100644 --- a/app/models/affiliation.rb +++ b/app/models/affiliation.rb @@ -1,7 +1,7 @@ class Affiliation < ApplicationRecord include Communicable include FilemakerCoded - filemaker_archived_in FmProject, FmPersonnel + filemaker_archived_in FmProject, FmPersonnel, FmPeopleToOrganization # Standing title given to the "facilitator affiliation" we create on registration # and org linking. Matches the `facilitators` scope / `facilitator?` predicate diff --git a/app/models/contact_method.rb b/app/models/contact_method.rb index f3ed4e0dc4..420e421a1e 100644 --- a/app/models/contact_method.rb +++ b/app/models/contact_method.rb @@ -1,4 +1,7 @@ class ContactMethod < ApplicationRecord + include FilemakerCoded + filemaker_archived_in FmPhoneNumber + UNKNOWN_CONTACT_TYPE = "unknown" CONTACT_TYPES = [ nil, "work", "personal", UNKNOWN_CONTACT_TYPE ].freeze diff --git a/app/models/fm_rolodex.rb b/app/models/fm_rolodex.rb index 2abdd07784..81f4cb8267 100644 --- a/app/models/fm_rolodex.rb +++ b/app/models/fm_rolodex.rb @@ -225,6 +225,24 @@ def addresses @addresses ||= contact_records(FmAddress, primary_ids: address_ids) end + # Cards flagged as an agency's, narrowed in SQL; #agency? reads the flag exactly. + scope :agency_cards, -> { where("JSON_UNQUOTE(JSON_EXTRACT(data, '$.isAgency__c')) NOT IN ('', '0', 'n', 'no', 'false', 'null')") } + + # The row is an agency's own contact card, sometimes naming the person to reach there. + def agency? + FilemakerValues.flag?(data["isAgency__c"]) + end + + def work_address?(address) + address.fm_id == data["WorksiteAddrsID"] || + FilemakerValues.contact_type(address.data["AddrsType"], allowed: Address::CONTACT_TYPES) == "work" + end + + # Where the person works rather than lives — every address on an agency's card is the agency's. + def organization_address?(address) + agency? || work_address?(address) + end + def job_affiliations @job_affiliations ||= FmPeopleToOrganization.find_by_data("PersonID", fm_id).order(:fm_id).to_a end diff --git a/app/models/organization.rb b/app/models/organization.rb index e00f5281a8..08a159b04e 100644 --- a/app/models/organization.rb +++ b/app/models/organization.rb @@ -16,6 +16,7 @@ class Organization < ApplicationRecord has_many :children, class_name: "Organization", foreign_key: :parent_id, inverse_of: :parent, dependent: :nullify has_many :addresses, as: :addressable, dependent: :destroy has_many :active_addresses, -> { active }, as: :addressable, class_name: "Address", inverse_of: :addressable + has_many :contact_methods, as: :contactable, dependent: :destroy has_many :bookmarks, as: :bookmarkable, dependent: :destroy has_many :other_responses, as: :owner, dependent: :destroy has_many :invoices, as: :invoicee, dependent: :restrict_with_error @@ -26,6 +27,12 @@ class Organization < ApplicationRecord has_many :users, through: :people has_many :comments, -> { newest_first }, as: :commentable, dependent: :destroy + # Its primary active phone, else any active one, else a phone an active address holds. + def phone_number + phones = contact_methods.select { |method| method.phone? && !method.inactive? } + (phones.find(&:primary?) || phones.first)&.value || addresses.reject(&:inactive?).filter_map(&:phone).first + end + # An organization is its own correspondent — there is no person behind it. def communications_email email @@ -77,6 +84,7 @@ def communications_email # Nested attributes accepts_nested_attributes_for :addresses, allow_destroy: true, reject_if: proc { |attrs| Address.unfilled_attributes?(attrs) } + accepts_nested_attributes_for :contact_methods, allow_destroy: true, reject_if: :all_blank accepts_nested_attributes_for :affiliations, allow_destroy: true, reject_if: proc { |attrs| attrs["person_id"].blank? } accepts_nested_attributes_for :comments, allow_destroy: true, reject_if: proc { |attrs| attrs["body"].blank? } diff --git a/app/services/address_deduper.rb b/app/services/address_deduper.rb index 309041019d..ba7409081c 100644 --- a/app/services/address_deduper.rb +++ b/app/services/address_deduper.rb @@ -8,6 +8,7 @@ class AddressDeduper # Filled in on the survivor from a merged copy when the survivor has none, so a # merge never drops a phone number, a ZIP, or a geocoded district only one copy had. + # FileMaker codes are unioned. CARRIED_COLUMNS = %w[ street_address zip_code country county district address_type locality phone la_city_council_district la_service_planning_area la_supervisorial_district @@ -86,9 +87,9 @@ def merge(loser, into:) def carry_over(loser, survivor) filled = CARRIED_COLUMNS.select { |column| missing?(survivor, column) && !missing?(loser, column) } - return if filled.empty? - - survivor.update!(filled.index_with { |column| loser.public_send(column) }) + survivor.assign_attributes(filled.index_with { |column| loser.public_send(column) }) + survivor.filemaker_code = Address.join_filemaker_codes(survivor.filemaker_code, loser.filemaker_code) + survivor.save! if survivor.changed? end def missing?(address, column) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb new file mode 100644 index 0000000000..fe2a4a4b98 --- /dev/null +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -0,0 +1,424 @@ +module AffiliationServices + # Turns the FileMaker archive's ties between people and organizations into affiliations + # (ADR-0008), after the Rolodex and organizations passes have made both ends. Reads, in + # order: + # + # 1. Personnel (FmPersonnel) — a person staffing a project's art program becomes a + # Facilitator affiliation, dated from when the row was entered to when the project + # closed, with a comment naming the role FileMaker gave. Only a Leader or Liaison is a + # primary contact — turned on for a Facilitator row the portal already had. It links + # an existing Facilitator row only when their dates agree (#same_dates); other dates + # are a separate stretch and become their own affiliation. Each carries the row's PrsnlRecID as its filemaker_code. A row + # whose project wasn't imported lands at the organization on the person's Rolodex card + # (created from its name when nothing matches); one naming no person — or only an + # agency's own card — gets AWBW Placeholder where nothing else affiliates anyone. + # 2. People-to-organizations (FmPeopleToOrganization) — a job at an organization, with + # its title, carrying the row's RecordID. + # 3. The Rolodex card itself (FmRolodex) — the organization its OrgID (or its typed + # organization name) names, with its Title, and its work addresses: each lands on + # the organization it belongs to and the person's affiliations there point at it. + # A work address no organization can be found for stays with the person. + # + # People are found by their Rolodex code, organizations by the project or organization + # code the earlier pass gave them. A row whose person or organization wasn't imported is + # skipped and reported. An existing affiliation of the same person, organization and + # title is linked rather than duplicated: it takes the row's code and fills its blank + # dates. + # + # None of these is minted from a training, so event_registration_id stays NULL: the + # reconcile page treats them as hand-entered rows. + class ImportFromFilemaker + include FilemakerValues + + OUTCOMES = %i[created linked already_imported skipped failed].freeze + BATCH_SIZE = 1_000 + # Personnel Status values that mean the person still staffs the program. Anything else — + # Inactive, Pending, Suspended, a blank, or two values typed over each other — reads as + # ended, since over-reporting active programs would skew program status. + ACTIVE_STATUSES = %w[active].freeze + # Every Personnel role is a Facilitator, the shape prod's January 2026 conversion gave the + # old app's roles. Only a role FileMaker names as Leader or Liaison makes a primary contact. + PRIMARY_ROLES = %w[leader leaders liaison].freeze + ROLE_COMMENT_TOPIC = "FileMaker import" + # Stands in where a Personnel row names a project but no person. + PLACEHOLDER_NAME = { first_name: "AWBW", last_name: "Placeholder" }.freeze + + attr_reader :results, :statuses, :roles + + def unmatched_addresses = @address_attributes.unmatched + + def initialize(progress: nil, run: nil) + @progress = progress + @run = run + @results = OUTCOMES.index_with { [] } + @statuses = Hash.new(0) + @roles = Hash.new(0) + @staff = StaffTranslation.new + @import_user_id = Setting.import_user&.id + @address_attributes = FmAddressAttributes.new + @name_matcher = OrganizationServices::NameMatcher.new + end + + def call + @tracker = Ahoy::Tracker.new + @people = FilemakerCodeIndex.new(Person) + @organizations = FilemakerCodeIndex.new(Organization) + @affiliations = FilemakerCodeIndex.new(Affiliation) + @projects = FmProject.all.index_by(&:fm_id) + @personnel_ids = FmPersonnel.pluck(:fm_id).to_set + @job_ids = FmPeopleToOrganization.pluck(:fm_id).to_set + @cards = FmRolodex.where(fm_id: FmPersonnel.distinct.pluck(Arel.sql("JSON_UNQUOTE(JSON_EXTRACT(data, '$.PersonID'))")).compact_blank) + .index_by(&:fm_id) + @unstaffed = [] + @read = 0 + + each_batch(FmPersonnel.all) { |row| import_personnel(row) } + each_batch(FmPeopleToOrganization.all) { |row| import_job(row) } + addresses = FmRolodex.contact_index(FmAddress) + phones = FmRolodex.contact_index(FmPhoneNumber) + each_batch(FmRolodex.all) { |rolodex| import_rolodex(rolodex.preload_contacts(addresses:, phone_numbers: phones)) } + in_batch(@unstaffed) { |row, organization, label| import_unstaffed(row, organization, label) } + self + end + + private + + # Each batch commits on its own, so a run that stops partway keeps what it finished. + # Its activity events, tagged with the run, commit with it — UndoRolodexImport reads them. + # Untouched, so a new affiliation doesn't stamp its person as edited now. + def each_batch(scope, &import) + scope.find_in_batches(batch_size: BATCH_SIZE) do |batch| + in_batch(batch, &import) + @progress&.call(@read += batch.size) + end + end + + def in_batch(rows, &import) + ActiveRecord::Base.transaction(requires_new: true) do + ActiveRecord::Base.no_touching do + Analytics::LifecycleBuffer.tagging(@run ? { import_run: @run } : {}) { rows.each(&import) } + end + Analytics::LifecycleBuffer.flush_to(@tracker) + end + end + + def import_personnel(row) + data = row.data + label = "#{row.fm_id} #{data["Name"].to_s.squish}".squish + return record(:already_imported, label) if @affiliations.include?(row.fm_id) + + card = @cards[data["PersonID"].to_s] + organization = organization_for(data["ProjectID"]) || card_organization(card, label) + return record(:skipped, "#{label}: #{missing_organization(data, card)}") unless organization + + people = @people.records(data["PersonID"]) + return @unstaffed << [ row, organization, label ] if data["PersonID"].blank? || (people.empty? && card&.agency?) + return record(:skipped, "#{label}: person #{data["PersonID"]} wasn't imported") if people.empty? + + dates = personnel_dates(data, @projects[data["ProjectID"].to_s]) + role = data["Role"].to_s.squish + attributes = dates.merge(primary_contact: primary_contact?(role)) + people.each do |person| + affiliation = affiliate(person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, attributes:, codes: @personnel_ids, dated: true) + note_role(affiliation, row, role) if affiliation && role.present? + end + end + + # A row whose project wasn't imported (or names none) lands at the organization on the + # person's Rolodex card — the one its OrgID names, else the one its typed name means, + # created when nothing answers to it, since the person staffed a program there. + def card_organization(card, label) + return unless card + + organization_for(card.data["OrgID"]) || named_organization(card.data["Organization"]) || + create_card_organization(card.data["Organization"].to_s.squish, label) + end + + def create_card_organization(name, label) + return if name.blank? + + organization = Organization.create(name: clip(name), created_by_id: @import_user_id, updated_by_id: @import_user_id) + unless organization.persisted? + failed(organization, label) + return + end + + record(:created, "#{label} → organization ##{organization.id} (#{organization.name}), named on the person's Rolodex card") + organization + end + + def missing_organization(data, card) + project = data["ProjectID"].presence + prefix = project ? "project #{project} wasn't imported" : "no project" + return "#{prefix}, and no Rolodex card for person #{data["PersonID"].presence || "(none)"}" unless card + + "#{prefix}, and the person's Rolodex card names no organization" + end + + # A Personnel row naming no one still records that the program ran. Read last, once every + # other row has had its say, so the placeholder only stands in at an organization nothing + # else affiliates anyone with — one placeholder there is enough. + def import_unstaffed(row, organization, label) + if organization.affiliations.exists? + return record(:skipped, "#{label}: names no person, and organization ##{organization.id} already has affiliations") + end + + dates = personnel_dates(row.data, @projects[row.data["ProjectID"].to_s]) + affiliation = affiliate(placeholder_person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, + attributes: dates.merge(primary_contact: false), codes: @personnel_ids, dated: true) + return unless affiliation + + affiliation.comments.create!(topic: ROLE_COMMENT_TOPIC, + body: "FileMaker Personnel #{row.fm_id} names no person, so #{PLACEHOLDER_NAME.values.join(" ")} " \ + "stands in to keep the program's FileMaker history.") + end + + def placeholder_person + @placeholder_person ||= Person.where("LOWER(first_name) = ? AND LOWER(last_name) = ?", *PLACEHOLDER_NAME.values.map(&:downcase)) + .order(:id).first || + Person.create!(**PLACEHOLDER_NAME, created_by_id: @import_user_id, updated_by_id: @import_user_id) + end + + # Every role FileMaker gave folds into Facilitator, so the affiliation keeps the word — + # and says when the import turned on primary contact for a row the portal already had. + def note_role(affiliation, row, role) + flipped = !affiliation.previously_new_record? && affiliation.saved_change_to_primary_contact? + affiliation.comments.create!(topic: ROLE_COMMENT_TOPIC, + body: "Marked as #{role} in FileMaker (Personnel #{row.fm_id}) at the time of the FileMaker import." \ + "#{" Primary contact turned on." if flipped}") + end + + def primary_contact?(role) + primary = PRIMARY_ROLES.include?(role.downcase) + @roles["#{role.presence || "(blank)"} → Facilitator#{", primary contact" if primary}"] += 1 + primary + end + + # Personnel rows carry no dates of their own: the assignment starts when it was entered + # and ends when its project closed. A status that isn't active ends it now, undated. + def personnel_dates(data, project) + status = data["Status"].to_s.squish + active = ACTIVE_STATUSES.include?(status.downcase) + @statuses["#{status.presence || "(blank)"} → #{active ? "active" : "ended"}"] += 1 + start_date = parse_date(data["DateEntered"]) + end_date = project && (parse_date(project.data["CloseDate"]) || parse_date(project.data["EndDate"])) + end_date = nil if start_date && end_date && end_date < start_date + { start_date:, end_date:, inactive: !active } + end + + def import_job(row) + data = row.data + label = "#{row.fm_id} #{data["PersonID"]} at #{data["OrgID"]}" + return record(:already_imported, label) if @affiliations.include?(row.fm_id) + + people = @people.records(data["PersonID"]) + organization = organization_for(data["OrgID"]) + return record(:skipped, "#{label}: person wasn't imported") if people.empty? + return record(:skipped, "#{label}: organization wasn't imported") unless organization + + people.each { |person| affiliate(person, organization, row, label, title: clip(data["Title"].to_s.squish), attributes: {}, codes: @job_ids) } + end + + # The Rolodex card names the person's organization and their title there, and lists the + # work addresses the people pass left for this one. + def import_rolodex(rolodex) + data = rolodex.data + work = rolodex.addresses.select { |address| rolodex.organization_address?(address) } + return if flag?(data["Delete"]) || (data["OrgID"].blank? && data["Organization"].blank? && work.empty?) + + label = "#{rolodex.fm_id} #{rolodex.name}" + people = @people.records(rolodex.fm_id) + return record(:skipped, "#{label}: person wasn't imported") if people.empty? + + organization = organization_for(data["OrgID"]) || named_organization(data["Organization"]) + if organization.nil? && (data["OrgID"].present? || data["Organization"].present?) + record(:skipped, "#{label}: organization #{data["OrgID"].presence || data["Organization"].squish} wasn't found") + end + title = clip(data["Title"].to_s.squish) + people.each do |person| + next unless organization + next if title.blank? && person.affiliations.exists?(organization:) + + affiliate(person, organization, rolodex, label, title:, attributes: {}, codes: nil) + end + place_work_addresses(rolodex, people, organization, work, label) + end + + def named_organization(name) + name = name.to_s.squish + @name_matcher.call(name)&.organization if name.present? + end + + # The affiliation this row describes: one the person already has at the organization + # with the same title is linked, otherwise one is created. `codes` is the set of IDs + # the row's own table holds — an affiliation already carrying another of them records + # a different row (another stint on the same project), so it's never the match. A row + # with no code of its own (the Rolodex card) only fills in what's missing. + def affiliate(person, organization, row, label, title:, attributes:, codes:, dated: false) + existing = person.affiliations.where(organization:).to_a + candidates = existing.select { |affiliation| same_title?(affiliation.title, title) && !records_another_row?(affiliation, codes) } + match = dated ? same_dates(candidates, attributes) : candidates.first + match ||= existing.find { |affiliation| affiliation.title.blank? } if title.present? && codes.nil? + return link(match, row, label, title:, attributes:, codes:) if match + + create(person, organization, row, label, title:, attributes:, code: codes && row.fm_id) + end + + # A Personnel row links only an affiliation whose dates agree with it — each the same, or + # blank on one side — and that still reads start-before-end once the blanks fill. Any other + # dates are a separate stretch, so the row becomes its own affiliation. Identical dates win, + # then the one sharing the most. + def same_dates(candidates, dates) + candidates.select { |affiliation| dates_agree?(affiliation, dates) } + .min_by { |affiliation| [ identical_dates?(affiliation, dates) ? 0 : 1, -shared_dates(affiliation, dates) ] } + end + + def dates_agree?(affiliation, dates) + starts = affiliation.start_date || dates[:start_date] + ends = affiliation.end_date || dates[:end_date] + %i[start_date end_date].all? { |attribute| affiliation[attribute].nil? || dates[attribute].nil? || affiliation[attribute] == dates[attribute] } && + (starts.nil? || ends.nil? || starts <= ends) + end + + def identical_dates?(affiliation, dates) + affiliation.start_date == dates[:start_date] && affiliation.end_date == dates[:end_date] + end + + def shared_dates(affiliation, dates) + %i[start_date end_date].count { |attribute| affiliation[attribute] && affiliation[attribute] == dates[attribute] } + end + + def records_another_row?(affiliation, codes) + codes && affiliation.filemaker_codes.any? { |code| codes.include?(code) } + end + + # Some prod rows still carry an old-app role title (Leader, Liaison, …), each a Facilitator. + def same_title?(held, title) + held.to_s.squish.casecmp?(title.to_s.squish) || + (title == Affiliation::FACILITATOR_TITLE && DataHealth::LegacyRoleAffiliations.legacy_title?(held)) + end + + def link(affiliation, row, label, title:, attributes:, codes:) + return record(:already_imported, "#{label} → affiliation ##{affiliation.id}") if codes.nil? && affiliation.title.present? + + affiliation.filemaker_code = Affiliation.join_filemaker_codes(affiliation.filemaker_code, row.fm_id) if codes + affiliation.title = title if affiliation.title.blank? + affiliation.primary_contact = true if attributes[:primary_contact] + fill_dates(affiliation, attributes) + affiliation.updated_by_id ||= updated_by(row.data) + return failed(affiliation, label) unless affiliation.save + + @affiliations.add(row.fm_id, affiliation.id) if codes + record(:linked, "#{label} → affiliation ##{affiliation.id}") + affiliation + end + + # A blank date takes the archive's. The archive's status only ends an affiliation the + # portal still holds open with no dates of its own to say otherwise. + def fill_dates(affiliation, attributes) + attributes.slice(:start_date, :end_date).each { |attribute, value| affiliation[attribute] ||= value } + return unless attributes[:inactive] && !affiliation.inactive? && affiliation.end_date.nil? + + affiliation.inactive = true + affiliation.inactive_supplied = true + end + + def create(person, organization, row, label, title:, attributes:, code:) + data = row.data + entered = parse_date(data["DateEntered"]) + affiliation = person.affiliations.build( + organization:, title:, filemaker_code: code, **attributes, + inactive_supplied: attributes[:inactive], + created_at: entered, updated_at: parse_date(data["DateTimeModified"].presence || data["ModificationTimeStamp"]) || entered, + created_by_id: @staff.user_id_for(data["EnteredBy"]) || @import_user_id, updated_by_id: updated_by(data) + ) + return failed(affiliation, label) unless affiliation.save + + @affiliations.add(code, affiliation.id) if code + record(:created, "#{label} → affiliation ##{affiliation.id}#{" (#{title})" if title}") + affiliation + end + + def updated_by(data) + @staff.user_id_for(data["ModifiedBy"]) || @import_user_id + end + + # Each work address lands on the organization it belongs to — the one FileMaker filed it + # under, else the card's organization, else the person's only one — and the person's + # affiliations there that name no address yet point at it. The worksite FileMaker named + # goes first, so it's the one they point at. With nowhere to put it, it stays with the + # person as their own work address. + def place_work_addresses(rolodex, people, organization, work, label) + work.sort_by { |address| address.fm_id == rolodex.data["WorksiteAddrsID"] ? 0 : 1 }.each do |fm_address| + attributes = @address_attributes.call(fm_address.data, address_type: "work")&.merge(filemaker_code: fm_address.fm_id) + next unless attributes + + people.each do |person| + target = organization_for(fm_address.data["OrgID"]) || organization || sole_organization(person) + next keep_with_person(person, attributes, label) unless target + + address = organization_address(target, attributes, label) + next failed(address, label) unless address.persisted? + + person.affiliations.where(organization: target, organization_address_id: nil).find_each do |affiliation| + affiliation.update(organization_address: address) || failed(affiliation, label) + end + end + end + end + + def sole_organization(person) + ids = person.affiliations.distinct.pluck(:organization_id) + Organization.find_by(id: ids.first) if ids.one? + end + + def organization_address(organization, attributes, label) + held = held_address(organization, attributes) + return organization.addresses.create(attributes.merge(primary: organization.addresses.where(primary: true).none?)) unless held + + add_code(held, attributes[:filemaker_code], label) + held + end + + def keep_with_person(person, attributes, label) + held = held_address(person, attributes) + return add_code(held, attributes[:filemaker_code], label) if held + + address = person.addresses.create(attributes.merge(primary: person.addresses.where(primary: true).none?)) + return failed(address, label) unless address.persisted? + + record(:created, "#{label} → work address ##{address.id} kept on person ##{person.id}, no organization to place it on") + end + + def held_address(addressable, attributes) + key = address_key(attributes[:street_address], attributes[:city]) + addressable.addresses.find { |address| address_key(address.street_address, address.city) == key } + end + + # An address already on file takes the archive row's code rather than a copy. + def add_code(address, code, label) + address.update(filemaker_code: Address.join_filemaker_codes(address.filemaker_code, code)) || failed(address, label) + end + + def address_key(street, city) + [ AddressNormalizer.street(street), AddressNormalizer.city(city) ] + end + + def organization_for(code) + code = code.to_s.strip + return if code.blank? + + ids = @organizations.ids(code) + Organization.find_by(id: ids.first) if ids.any? + end + + def failed(record, label) + record(:failed, "#{label}: #{record.class.name} #{record.errors.full_messages.to_sentence.presence || "matches one already on file"}") + false + end + + def record(outcome, message) + @results[outcome] << message + end + end +end diff --git a/app/services/affiliation_stint.rb b/app/services/affiliation_stint.rb new file mode 100644 index 0000000000..b43457ae9a --- /dev/null +++ b/app/services/affiliation_stint.rb @@ -0,0 +1,17 @@ +# One affiliation's stretch of time, for telling a person's separate stints at one +# organization apart. A blank start reads as "since whenever" and a blank end as "still +# going" — except an ended row with no end date, which reads as the moment it started, so +# it can't reach a later stretch. +AffiliationStint = Data.define(:starts, :ends) do + def self.for(start_date:, end_date:, ended:) + new(starts: start_date, ends: end_date || (start_date if ended)) + end + + def self.of(affiliation) + self.for(start_date: affiliation.start_date, end_date: affiliation.end_date, ended: affiliation.inactive?) + end + + def overlap?(other) + (starts.nil? || other.ends.nil? || starts <= other.ends) && (other.starts.nil? || ends.nil? || other.starts <= ends) + end +end diff --git a/app/services/contact_method_deduper.rb b/app/services/contact_method_deduper.rb new file mode 100644 index 0000000000..f7f928709e --- /dev/null +++ b/app/services/contact_method_deduper.rb @@ -0,0 +1,91 @@ +# After a person or organization merge, the generic merge moves both records' contact +# methods onto the keeper wholesale (contact_methods have no unique index), leaving +# identical duplicates and possibly two primary phones of the same kind. Merge contact +# methods that share the same kind + value into one, then settle a single primary per +# kind: the keeper's own pre-merge primary, or the deleted record's when the keeper had +# none. Runs in the people and organizations dedupers' after_merge. +class ContactMethodDeduper + IDENTITY_COLUMNS = %w[kind value contact_type].freeze + + # Outside the identity, so a merge never drops the address link only one copy had. + # FileMaker codes are unioned. + CARRIED_COLUMNS = %w[address_id].freeze + + def self.signature(contact_method) + return unless contact_method + + IDENTITY_COLUMNS.map do |column| + value = contact_method.public_send(column).to_s.strip.downcase + # A row written before an untyped number said so holds a blank, so unknown and + # blank have to read alike or an otherwise identical pair won't fold. + value == ContactMethod::UNKNOWN_CONTACT_TYPE && column == "contact_type" ? "" : value + end + end + + # One preserved-primary signature per kind, read before the merge while the two + # records' contact methods can still be told apart: the keeper's own primary of + # that kind, else the deleted record's. + def self.primary_signatures(keep, delete) + methods = keep.contact_methods.to_a + delete.contact_methods.to_a + methods.map(&:kind).uniq.filter_map do |kind| + primary = methods.find { |method| method.kind == kind && method.primary? && method.contactable == keep } || + methods.find { |method| method.kind == kind && method.primary? } + signature(primary) + end + end + + def initialize(contactable, primary_signatures: []) + @contactable = contactable + @primary_signatures = primary_signatures + end + + def call + survivors = {} + merge_candidates.each do |contact_method| + signature = self.class.signature(contact_method) + if (survivor = survivors[signature]) + merge(contact_method, into: survivor) + else + survivors[signature] = contact_method + end + end + + settle_primary(survivors.values) + end + + private + + # Active before inactive, then the primary, then oldest — so a merge keeps the + # copy still in use. `#phone_number` reads only active phones, so letting an + # inactive duplicate win would leave the record looking phoneless. + def merge_candidates + @contactable.contact_methods.reload.sort_by do |contact_method| + [ contact_method.inactive? ? 1 : 0, contact_method.primary? ? 0 : 1, contact_method.id ] + end + end + + def merge(loser, into:) + carry_over(loser, into) + deduper.merge(into, loser) + end + + def carry_over(loser, survivor) + filled = CARRIED_COLUMNS.select do |column| + survivor.public_send(column).blank? && loser.public_send(column).present? + end + survivor.assign_attributes(filled.index_with { |column| loser.public_send(column) }) + survivor.filemaker_code = ContactMethod.join_filemaker_codes(survivor.filemaker_code, loser.filemaker_code) + survivor.save! if survivor.changed? + end + + def settle_primary(contact_methods) + contact_methods.each do |contact_method| + primary = @primary_signatures.include?(self.class.signature(contact_method)) + contact_method.update!(primary: primary) if contact_method.primary? != primary + end + end + + def deduper + @deduper ||= ModelDeduper.new(model_class: ContactMethod, logger: Rails.logger, dry_run: false) + end +end diff --git a/app/services/data_health.rb b/app/services/data_health.rb index fce4f4dc7a..1a2762ead7 100644 --- a/app/services/data_health.rb +++ b/app/services/data_health.rb @@ -8,6 +8,7 @@ module DataHealth "Affiliations" => [ FacilitatorAffiliationsFromNonTrainings, MisalignedAffiliationProvenance, + LegacyRoleAffiliations, LegacyOrganizationStatusDrift ], "People" => [ diff --git a/app/services/data_health/legacy_role_affiliations.rb b/app/services/data_health/legacy_role_affiliations.rb new file mode 100644 index 0000000000..0af2e7f297 --- /dev/null +++ b/app/services/data_health/legacy_role_affiliations.rb @@ -0,0 +1,62 @@ +module DataHealth + # Affiliations still titled with one of the old app's roles, which the January 2026 + # conversion copied into titles: a Leader or Liaison was a Facilitator who is a primary + # contact, an Assistant a Facilitator who isn't. None of those titles counts toward program + # status. Report-only: each row shows the person's Facilitator rows at that organization and + # opens their edit page to settle by hand. + class LegacyRoleAffiliations < Check + LEGACY_TITLES = %w[Leader Liaison Assistant default].freeze + + def title = "Affiliations still titled with an old-app role" + + def explanation + "The old app's Leader, Liaison and Assistant roles (and its blank \"default\") became titles in " \ + "January, but only a Facilitator affiliation counts toward program status. Each row is listed " \ + "with the person's Facilitator rows at that organization; open the person to merge, rename or remove it." + end + + def scope + Affiliation.where(title: LEGACY_TITLES).includes(:person, :organization).order(:title, :organization_id, :person_id, :id) + end + + def self.legacy_title?(title) + LEGACY_TITLES.any? { |legacy| legacy.casecmp?(title.to_s.squish) } + end + + def row_partial = "admin/data_health/legacy_role_row" + + def describe(affiliation) + "#{affiliation.person&.name} — #{affiliation.organization&.name} (#{affiliation.title})" + end + + # The person's Facilitator rows at the affiliation's organization, oldest first. + def facilitators_for(affiliation) + Affiliation.facilitators.where(person_id: affiliation.person_id, organization_id: affiliation.organization_id) + .order(:start_date, :id).to_a + end + + # How many organizations other than this one the person facilitates at. + def facilitator_elsewhere_count(affiliation) + Affiliation.facilitators.where(person_id: affiliation.person_id).where.not(organization_id: affiliation.organization_id) + .distinct.count(:organization_id) + end + + # The Facilitator row recording the same stretch (AffiliationStint), the same dates first. + def twin_for(affiliation, facilitators = facilitators_for(affiliation)) + stint = AffiliationStint.of(affiliation) + facilitators.select { |facilitator| AffiliationStint.of(facilitator).overlap?(stint) } + .min_by { |facilitator| same_dates?(facilitator, affiliation) ? 0 : 1 } + end + + def stretch_label(facilitator, affiliation, twin) + return "same dates" if same_dates?(facilitator, affiliation) + return "overlaps with different dates" if facilitator == twin + + "a different stretch" + end + + def same_dates?(facilitator, affiliation) + facilitator.start_date == affiliation.start_date && facilitator.end_date == affiliation.end_date + end + end +end diff --git a/app/services/filemaker_code_index.rb b/app/services/filemaker_code_index.rb new file mode 100644 index 0000000000..23153999f7 --- /dev/null +++ b/app/services/filemaker_code_index.rb @@ -0,0 +1,30 @@ +# Which records of a FilemakerCoded model carry each code, read once so a bulk import +# looks codes up in memory rather than scanning the table per archive row. Records the +# import codes as it goes are added with #add, so a later row finds them. +class FilemakerCodeIndex + def initialize(model) + @model = model + @ids = Hash.new { |hash, code| hash[code] = [] } + model.where.not(filemaker_code: [ nil, "" ]).pluck(:id, :filemaker_code).each do |id, value| + model.split_filemaker_codes(value).each { |code| @ids[code] << id } + end + end + + def ids(code) + @ids.fetch(@model.strip_filemaker_code(code), []) + end + + def include?(code) + ids(code).any? + end + + def records(code) + ids = ids(code) + ids.empty? ? [] : @model.where(id: ids).order(:id).to_a + end + + def add(code, id) + code = @model.strip_filemaker_code(code) + @ids[code] |= [ id ] if code.present? + end +end diff --git a/app/services/filemaker_values.rb b/app/services/filemaker_values.rb new file mode 100644 index 0000000000..33cd9e2a52 --- /dev/null +++ b/app/services/filemaker_values.rb @@ -0,0 +1,73 @@ +# How every FileMaker archive import reads a raw field value: dates in the archive's mixed +# formats, yes/no flags, contact types, and free text bounded to the column it lands in. +# Included by the importers so each reads the archive the same way. +module FilemakerValues + module_function + + STRING_LIMIT = 255 + EMAIL_PATTERN = /[^\s,;<>"']+@[^\s,;<>"']+\.[^\s,;<>"']+/ + FALSE_FLAGS = %w[0 n no false].freeze + DATE_FORMATS = %w[%m/%d/%Y %Y-%m-%d %m-%d-%Y %Y/%m/%d].freeze + # A two-digit year below this is read as 20xx, at or above it as 19xx. + TWO_DIGIT_YEAR_PIVOT = 10 + CONTACT_TYPES = AddressTranslation::CONTACT_TYPES + + # Every string copied here lands in a varchar(255) the archive's free text isn't + # bounded by, and an over-long incidental field would otherwise cost the record — + # or, for a column with no length validation behind it, raise mid-import. + def clip(value) + value.presence && value.to_s.truncate(STRING_LIMIT) + end + + # The archive words an email field freely ("Attn: someone", a note, two addresses at + # once), so it's read for the addresses in it the portal's email validation accepts. + def email_addresses(value) + value.to_s.scan(EMAIL_PATTERN).select { |email| valid_email?(email) } + end + + def valid_email?(email) + email.length <= STRING_LIMIT && email.match?(URI::MailTo::EMAIL_REGEXP) + end + + def flag?(value) + value.to_s.strip.presence && !FALSE_FLAGS.include?(value.to_s.strip.downcase) + end + + # The archive's types map onto one set of ours, but each model declares the slice it + # accepts — an address can be the mailing one, a phone number can't. A type we can't + # place is left blank, which both models then record as unknown. + def contact_type(value, allowed:) + CONTACT_TYPES[value.to_s.strip.downcase].presence_in(allowed) + end + + def parse_date(value) + value = value.to_s.strip + return if value.blank? + + DATE_FORMATS.each do |format| + parsed = parse_with(value, format) + return parsed if parsed + end + nil + end + + # Parsing stops where the format ends, so "10-5-1983" read as %Y-%m-%d takes 19 + # as the day and drops the 83 — digits left over mean the format split the value + # in the wrong place, so the next one gets its turn. + def parse_with(value, format) + parts = Date._strptime(value, format) + return if parts.nil? || parts[:leftover].to_s.match?(/\A\d/) + + expand_century(Date.new(parts[:year], parts[:mon], parts[:mday])) + rescue Date::Error, TypeError + nil + end + + # %Y reads a two-digit year as the year itself, so "10/5/83" is 83 AD. The archive + # mixes two- and four-digit years, so a bare pair gets a century: 83 is 1983, 05 is 2005. + def expand_century(date) + return date if date.year >= 100 + + date.change(year: date.year + (date.year < TWO_DIGIT_YEAR_PIVOT ? 2000 : 1900)) + end +end diff --git a/app/services/organization_services/import_from_filemaker.rb b/app/services/organization_services/import_from_filemaker.rb new file mode 100644 index 0000000000..7a0d81a523 --- /dev/null +++ b/app/services/organization_services/import_from_filemaker.rb @@ -0,0 +1,452 @@ +module OrganizationServices + # Turns the FileMaker archive's organizations (FmOrganization) and the art programs run at + # them (FmProject) into portal organizations (ADR-0008). Runs after the Rolodex pass and + # before AffiliationServices::ImportFromFilemaker, which finds what this creates by code. + # + # An app organization is usually a FileMaker project; the FileMaker organization above it + # becomes its parent (ADR-0004). An organization with exactly one project is one place, + # not two, so it stays one portal organization carrying both codes. Each archive row is + # matched to an existing organization by any shared filemaker_code first, then by exact + # name — but only to one carrying no code from that row's table yet, since a name taken + # by another FileMaker record is a namesake, not a match. A match takes the row's code. + # + # Every row lands in one outcome bucket of #results: + # * created — a new organization + # * linked — an existing organization took the row's code and was filled + # * already_imported — an organization already carried the row's code + # * skipped — nameless rows, and merged projects with nowhere to fold into + # * failed — the Organization didn't validate + # + # Fields fill blanks only. Where the portal already holds a different value it keeps its + # own, noted in #differences and as a comment on the organization. Also carried: + # * the organization's addresses (fm_addresses by OrgID), its phones (fm_phone_numbers), + # and its email (fm_email_addresses) — each address and phone carrying its row's ID as + # its own filemaker_code + # * each project's City/State with its Locality, District, SPA and supervisorial + # district, onto the program's address in that city + # * a merged-away project's code, onto the program it merged into + # * an agency's own Rolodex card: its website, mission and addresses (and its email, + # when the card names no contact person whose email it would be) + class ImportFromFilemaker + include FilemakerValues + + OUTCOMES = %i[created linked already_imported skipped failed].freeze + BATCH_SIZE = 500 + # FileMaker's OrgType / AgencyType words onto Organization::ORGANIZATION_TYPES, first + # match winning — so a value naming two types ("501c3/Nonprofit Government Agency"), as + # FileMaker's multi-select stored them, takes the first. Any other word ("School", + # "Grassroots") is Other, with the word kept as its description. + ORGANIZATION_TYPE_PATTERNS = { + /501|non.?profit/i => "501c3/nonprofit", + /for.?profit|\bLLC\b/i => "For-profit", + /govt|government/i => "Government agency", + /\Aother\z/i => Organization::ORGANIZATION_TYPE_OTHER + }.freeze + TYPE_FIELDS = %i[organization_type organization_type_other].freeze + COMMENT_SOURCES = { FmOrganization => "FileMaker organization", FmProject => "FileMaker project", + FmRolodex => "FileMaker Rolodex agency" }.freeze + + attr_reader :results, :differences, :unmatched_types + + def unmatched_addresses = @address_attributes.unmatched + + def initialize(progress: nil, run: nil) + @progress = progress + @run = run + @results = OUTCOMES.index_with { [] } + @differences = [] + @unmatched_types = Hash.new(0) + @staff = StaffTranslation.new + @import_user_id = Setting.import_user&.id + @address_attributes = FmAddressAttributes.new + @name_matcher = NameMatcher.new + end + + def call + @tracker = Ahoy::Tracker.new + @codes = FilemakerCodeIndex.new(Organization) + @organization_ids = FmOrganization.pluck(:fm_id).to_set + projects = FmProject.all.to_a + @projects = projects.index_by(&:fm_id) + @project_ids = @projects.keys.to_set + @addresses = FmAddress.all.group_by { |address| address.data["OrgID"].to_s } + @phones = FmPhoneNumber.all.group_by { |phone| phone.data["OrgID"].to_s } + @emails = FmEmailAddress.all.group_by { |email| email.data["OrgID"].to_s } + + merged, live = projects.partition { |project| merged_into(project) } + by_organization = live.group_by { |project| project.data["OrgID"].to_s } + read = 0 + FmOrganization.find_in_batches(batch_size: BATCH_SIZE) do |batch| + import_batch(batch) { |organization| import_organization(organization, by_organization.delete(organization.fm_id) || []) } + @progress&.call(read += batch.size) + end + import_batch(by_organization.values.flatten) { |project| import_projects(nil, [ project ]) } + import_batch(merged) { |project| fold_merged(project) } + import_batch(agency_cards) { |rolodex| import_agency(rolodex) } + self + end + + private + + # Each batch commits on its own, so a run that stops partway keeps what it finished. + # Its activity events, tagged with the run, commit with it — UndoRolodexImport reads them. + # Untouched, so an address added to an organization doesn't stamp it as edited now. + def import_batch(rows, &import) + ActiveRecord::Base.transaction(requires_new: true) do + ActiveRecord::Base.no_touching do + Analytics::LifecycleBuffer.tagging(@run ? { import_run: @run } : {}) { rows.each(&import) } + end + Analytics::LifecycleBuffer.flush_to(@tracker) + end + end + + def import_organization(fm_organization, projects) + data = fm_organization.data + name = clip(data["OrgName"].to_s.squish) + label = "#{fm_organization.fm_id} #{name}".squish + if name.blank? + record(:skipped, "#{label}: no name") + return import_projects(nil, projects) + end + + umbrella = coded(fm_organization.fm_id) || named(name, @organization_ids) + return import_single_program(fm_organization, projects.sole, umbrella, label) if projects.one? + + umbrella ||= Organization.new(name: unique_name(name, fm_organization.fm_id)) + return unless save(umbrella, fm_organization, label, organization_attributes(data)) + + import_projects(umbrella, projects) + end + + # One organization running one program is a single place, so it stays one portal + # organization carrying both codes — unless the portal already keeps them apart. + def import_single_program(fm_organization, project, umbrella, label) + program = coded(project.fm_id) || named(project_name(project), @project_ids) + if umbrella && program && umbrella != program + return unless save(umbrella, fm_organization, label, organization_attributes(fm_organization.data)) + + return import_projects(umbrella, [ project ]) + end + + organization = umbrella || program || Organization.new(name: unique_name(clip(fm_organization.data["OrgName"].to_s.squish), project.fm_id)) + organization_side = organization_attributes(fm_organization.data) + attributes = organization_side.merge(project_attributes(project.data)) { |_key, mine, theirs| mine.presence || theirs } + attributes.merge!(organization_side.slice(*TYPE_FIELDS)) if organization_side[:organization_type] + save(organization, fm_organization, "#{label} + #{project.fm_id}", attributes, also: project) + end + + def import_projects(umbrella, projects) + projects.each do |project| + name = project_name(project) + label = "#{project.fm_id} #{name}".squish + next record(:skipped, "#{label}: no name") if name.blank? + + program = coded(project.fm_id) || named(name, @project_ids) || Organization.new(name: unique_name(name, project.fm_id)) + nest(program, umbrella, label) if umbrella && program != umbrella + save(program, project, label, project_attributes(project.data)) + end + end + + # A program the portal already nests elsewhere stays where it is. + def nest(program, umbrella, label) + return program.parent = umbrella if program.parent_id.nil? + return if program.parent_id == umbrella.id + + @differences << "#{label} → ##{program.id}: kept parent ##{program.parent_id}, FileMaker nests it under ##{umbrella.id}" + end + + # Writes one archive row (and, for a single-program organization, its project too) onto + # the organization: its code, blank fields, addresses, phone and email. + def save(organization, row, label, attributes, also: nil) + rows = [ row, also ].compact + outcome = outcome_for(organization, rows) + organization.filemaker_code = Organization.join_filemaker_codes(organization.filemaker_code, rows.map(&:fm_id)) + kept = fill(organization, attributes) + stamp(organization, row.data) + add_addresses(organization, @addresses.fetch(row.fm_id, [])) + rows.select { |source| source.is_a?(FmProject) }.each { |project| place_project(organization, project.data) } + add_phones(organization, @phones.fetch(row.fm_id, [])) + fill_email(organization, @emails.fetch(row.fm_id, []).map { |email| email.data["EmailAddress"] }) + return failed(organization, label) unless organization.save + + rows.each { |source| @codes.add(source.fm_id, organization.id) } + note_differences(organization, row, label, kept) if kept.any? + record(outcome, "#{label} → ##{organization.id}") + organization + end + + def outcome_for(organization, rows) + return :created if organization.new_record? + + rows.all? { |row| organization.filemaker_codes.include?(row.fm_id) } ? :already_imported : :linked + end + + # FileMaker kept a merged-away project's row, pointing at the one it merged into, so its + # code joins that program and the records still naming it find their way there. + def fold_merged(project) + label = "#{project.fm_id} #{project_name(project)}".squish + survivor = merged_into(project) + program = coded(survivor.fm_id) + own = coded(project.fm_id) + return record(:skipped, "#{label}: merged into #{survivor.fm_id}, which wasn't imported") unless program + return record(:already_imported, "#{label} → ##{program.id}") if own == program + if own + return record(:skipped, "#{label}: merged into #{survivor.fm_id} (##{program.id}) in FileMaker but still its own organization ##{own.id} — merge them by hand") + end + + program.filemaker_code = Organization.join_filemaker_codes(program.filemaker_code, project.fm_id) + return failed(program, label) unless program.save + + @codes.add(project.fm_id, program.id) + record(:linked, "#{label} → ##{program.id} (merged into #{survivor.fm_id})") + end + + # The project a merged-away one ended up in, following a chain of merges to its end. + def merged_into(project) + seen = Set[project.fm_id] + survivor = nil + while (next_id = (survivor || project).data["MergerWithID"].to_s.strip.presence) && seen.add?(next_id) + break unless @projects[next_id] + + survivor = @projects[next_id] + end + survivor + end + + # An agency's own Rolodex card describes the agency: it fills the organization its OrgID + # names, or the one its name means, which is created when nothing answers to it. + def import_agency(rolodex) + data = rolodex.data + name = clip(data["Organization"].to_s.squish) + label = "#{rolodex.fm_id} #{name}".squish + organization = coded(data["OrgID"]) || (@name_matcher.call(name)&.organization if name) + return record(:skipped, "#{label}: agency card with no organization named") if organization.nil? && name.blank? + + organization ||= Organization.new(name:) + kept = fill(organization, website_url: clip(data["WebSite"]), mission_vision_values: clip(data["Mission"])) + stamp(organization, data) + add_addresses(organization, rolodex.addresses) + add_phones(organization, rolodex.phone_numbers) + fill_email(organization, [ data["EMail"] ]) unless names_a_person?(data) + outcome = agency_outcome(organization) + return failed(organization, label) unless organization.save + + note_differences(organization, rolodex, label, kept) if kept.any? + record(outcome, "#{label} → ##{organization.id} (agency card)") + end + + def agency_outcome(organization) + return :created if organization.new_record? + + organization.changed_for_autosave? ? :linked : :already_imported + end + + def agency_cards + FmRolodex.agency_cards.order(:fm_id).select(&:agency?) + .reject { |rolodex| flag?(rolodex.data["Delete"]) } + end + + # A card naming a contact person carries that person's email, not the agency's. + def names_a_person?(data) + data["FirstName"].present? && data["LastName"].present? + end + + def coded(code) + code = code.to_s.strip + return if code.blank? + + ids = @codes.ids(code) + Organization.find_by(id: ids.first) if ids.any? + end + + # The organization by exact name, unless it already carries another code from the same + # archive table — then it's a namesake of a different FileMaker record. + def named(name, table_ids) + return if name.blank? + + organization = Organization.find_by(name:) + organization unless organization&.filemaker_codes&.any? { |code| table_ids.include?(code) } + end + + # Organization names are unique, so a FileMaker record whose name a namesake already + # holds is told apart by its code. + def unique_name(name, code) + Organization.exists?(name:) ? clip("#{name} (#{code})") : name + end + + def project_name(project) + clip(project.data["ProjectName"].to_s.squish) + end + + def organization_attributes(data) + { + website_url: clip(data["Website"]), + description: data["Description"].to_s.strip.presence, + mission_vision_values: clip(data["Mission"]), + **organization_type(data["OrgType"]), + **archive_timestamps(data["CreationDate"], data["ModificationDateTime"]) + } + end + + def project_attributes(data) + { + description: data["Description"].to_s.strip.presence, + **organization_type(data["AgencyType"]), + **archive_timestamps(data["DateEntered"], data["DateTimeModified"]) + } + end + + # The archive's own dates, so a created organization dates from when FileMaker first + # knew it. Only a new record takes them; fill passes them over on an existing one. + def archive_timestamps(entered, modified) + entered = parse_date(entered) + { created_at: entered, updated_at: parse_date(modified) || entered } + end + + def organization_type(value) + value = value.to_s.squish + return {} if value.blank? + + type = ORGANIZATION_TYPE_PATTERNS.find { |pattern, _type| value.match?(pattern) }&.last + return { organization_type: type } if type + + @unmatched_types[value] += 1 + { organization_type: Organization::ORGANIZATION_TYPE_OTHER, organization_type_other: clip(value) } + end + + # Staff edited both systems, so neither side is newer by rule: a blank takes the + # archive's value, and a disagreement keeps the portal's and is returned as + # [ label, portal value, archive value ] to note. + def fill(organization, attributes) + timestamps, fields = attributes.partition { |attribute, _value| %i[created_at updated_at].include?(attribute) } + timestamps.each { |attribute, value| organization[attribute] = value if organization.new_record? && value } + fields.filter_map do |attribute, value| + next if value.blank? + # The description only belongs to an Other type, which the type just settled. + next if attribute == :organization_type_other && organization.organization_type != Organization::ORGANIZATION_TYPE_OTHER + + if organization[attribute].blank? + organization[attribute] = value + next + end + next if organization[attribute].to_s.squish.casecmp?(value.to_s.squish) + + [ Organization.human_attribute_name(attribute), organization[attribute].to_s.truncate(80), value.to_s.truncate(80) ] + end + end + + # The staff FileMaker recorded as entering and last changing the row, or the settings' + # import user. An existing organization keeps any stamps it already has. + def stamp(organization, data) + organization.created_by_id ||= @staff.user_id_for(data["CreatedBy"].presence || data["EnteredBy"]) || @import_user_id + organization.updated_by_id ||= @staff.user_id_for(data["ModifiedBy"]) || @import_user_id + end + + # An organization gathers addresses from every source, so one it already has takes the + # archive row's code and the others are added; the first becomes its primary when it has none. + def add_addresses(organization, fm_addresses) + fm_addresses.each do |fm_address| + attributes = @address_attributes.call(fm_address.data, address_type: "work") + next if attributes.nil? + + held = held_address(organization, attributes) + next held.filemaker_code = Address.join_filemaker_codes(held.filemaker_code, fm_address.fm_id) if held + + primary = organization.addresses.none? { |address| address.primary? && !address.marked_for_destruction? } + organization.addresses.build(attributes.merge(primary:, filemaker_code: fm_address.fm_id)) + end + end + + def held_address(organization, attributes) + key = address_key(attributes[:street_address], attributes[:city]) + organization.addresses.find { |address| address_key(address.street_address, address.city) == key } + end + + def address_key(street, city) + [ AddressNormalizer.street(street), AddressNormalizer.city(city) ] + end + + # FileMaker records a project's locality and districts, which the portal keeps on an + # address — so they land on the program's address in the project's city, or on a new + # one holding just the city when the program has no address there yet. + def place_project(organization, data) + city = clip(data["City"].to_s.squish) + state = clip(data["State"].to_s.squish) + return if city.blank? + + geography = project_geography(data) + address = organization.addresses.find { |held| AddressNormalizer.city(held.city) == AddressNormalizer.city(city) } + return fill_geography(address, geography) if address + return if state.blank? + + primary = organization.addresses.none?(&:primary?) + place = @address_attributes.place(city:, state:, county: geography[:county]) + organization.addresses.build(place.merge(geography.compact, address_type: "work", primary:)) + end + + def project_geography(data) + { + locality: Address::LOCALITIES.find { |locality| locality.casecmp?(data["Locality"].to_s.squish) }, + county: clip(data["County"].to_s.squish), + district: clip(data["District"].to_s.squish), + la_service_planning_area: district_number(data["SPA"]), + la_supervisorial_district: district_number(data["LASupervisorialDistrict"]) + }.compact_blank + end + + # A locality worked out from the address's parts is a guess the project's own answer + # replaces when the guess was Unknown. + def fill_geography(address, geography) + geography.each do |attribute, value| + current = address[attribute] + address[attribute] = value if current.blank? || (attribute == :locality && current == Address::UNKNOWN_LOCALITY) + end + end + + def district_number(value) + value.to_s[/\d+/]&.to_i + end + + # Each number lands on a phone record carrying its archive row's code — the one the + # organization already holds for that number, else a new one, the archive's primary first. + def add_phones(organization, fm_phones) + fm_phones.sort_by { |phone| flag?(phone.data["Primacy"]) ? 0 : 1 }.each do |phone| + number = clip(phone.data["PhoneNumber"].to_s.strip) + next if number.blank? + + held = organization.contact_methods.find { |method| phone_key(method.value) == phone_key(number) } + next held.filemaker_code = ContactMethod.join_filemaker_codes(held.filemaker_code, phone.fm_id) if held + + organization.contact_methods.build( + kind: "phone", value: number, filemaker_code: phone.fm_id, + contact_type: contact_type(phone.data["PhoneType"], allowed: ContactMethod::CONTACT_TYPES), + primary: organization.contact_methods.none?(&:primary?) + ) + end + end + + def phone_key(value) + value.to_s.gsub(/\D/, "").presence + end + + def fill_email(organization, values) + organization.email = values.flat_map { |value| email_addresses(value) }.first if organization.email.blank? + end + + def note_differences(organization, row, label, kept) + lines = kept.map { |field, portal, archive| "#{field}: kept “#{portal}”, FileMaker has “#{archive}”" } + Comment.create!(commentable: organization, + body: "#{COMMENT_SOURCES[row.class]} #{row.fm_id} differs from this organization:\n#{lines.join("\n")}") + @differences << "#{label} → ##{organization.id}: #{lines.join('; ')}" + end + + def failed(organization, label) + record(:failed, "#{label}: #{"##{organization.id} " if organization.persisted?}#{organization.errors.full_messages.to_sentence}") + nil + end + + def record(outcome, message) + @results[outcome] << message + end + end +end diff --git a/app/services/person_services/import_from_rolodex.rb b/app/services/person_services/import_from_rolodex.rb index a37c60a122..6d8d1ddf3c 100644 --- a/app/services/person_services/import_from_rolodex.rb +++ b/app/services/person_services/import_from_rolodex.rb @@ -15,9 +15,14 @@ module PersonServices # field the archive disagrees on keeps the person's value — # noted in #differences and as a comment on the person # * already_imported — a person already carries the row's ID - # * skipped — agency, soft-deleted, or nameless rows + # * skipped — soft-deleted or nameless rows (an agency row naming no contact + # person is the agency itself, which the organizations import reads) # * failed — the Person didn't validate # + # Each address and phone the row brings carries its archive row's ID as its own + # filemaker_code, added to the one the person already holds when it's the same place + # or number. + # # An email field holding no address the person's email validation accepts ("ada@@x.org", # "ada at x dot org") or an address it rejects is left off — the row imports without it. # When that leaves the person with no email on file, the field's whole value is left as a @@ -35,21 +40,16 @@ module PersonServices # named handful of rows (the task's FM_IDS), since a row whose created person has since # changed their email no longer matches and would be created again. class ImportFromRolodex + include FilemakerValues + OUTCOMES = %i[created linked already_imported skipped failed].freeze - FALSE_FLAGS = %w[0 n no false].freeze - DATE_FORMATS = %w[%m/%d/%Y %Y-%m-%d %m-%d-%Y %Y/%m/%d].freeze - # A two-digit year below this is read as 20xx, at or above it as 19xx. - TWO_DIGIT_YEAR_PIVOT = 10 - CONTACT_TYPES = AddressTranslation::CONTACT_TYPES # Answers to "which kind of email is this?" that aren't a kind at all. PRECEDENCE_TYPES = %w[preferred primary main].freeze - EMAIL_PATTERN = /[^\s,;<>"']+@[^\s,;<>"']+\.[^\s,;<>"']+/ # EmailPrimary holds a flag, never an address, so only EMail is read for one. The flag # says whether EMail is the person's primary email ("1") or their secondary ("2"). EMAIL_FIELDS = %w[EMail].freeze EMAIL_FLAG_FIELD = "EmailPrimary" SECONDARY_EMAIL_FLAG = "2" - STRING_LIMIT = 255 BATCH_SIZE = 1_000 # Whether a translated tag lands as primary or additional follows the field (ADR-0007). @@ -128,8 +128,8 @@ def import(rolodex) label = "#{rolodex.fm_id} #{first_name} #{last_name}".squish return record(:already_imported, label) if !@relink && @imported_codes.include?(rolodex.fm_id) - return record(:skipped, "#{label}: agency") if flag?(data["isAgency__c"]) return record(:skipped, "#{label}: marked deleted") if flag?(data["Delete"]) + return record(:skipped, "#{label}: agency with no contact person — the organizations import reads it") if rolodex.agency? && (first_name.blank? || last_name.blank?) return record(:skipped, "#{label}: missing first or last name") if first_name.blank? || last_name.blank? row_emails = emails(data) @@ -264,7 +264,7 @@ def fill_email_2(person, data) # The archive words an email field freely ("Attn: someone", a note, two addresses at # once), so each is read for the addresses in it; the same one twice counts once. def emails(data) - EMAIL_FIELDS.flat_map { |field| data[field].to_s.scan(EMAIL_PATTERN) }.select { |email| valid_email?(email) }.uniq(&:downcase) + EMAIL_FIELDS.flat_map { |field| email_addresses(data[field]) }.uniq(&:downcase) end def invalid_email_values(data) @@ -277,10 +277,6 @@ def usable_email_value?(value) addresses.any? && addresses.all? { |email| valid_email?(email) } end - def valid_email?(email) - email.length <= STRING_LIMIT && email.match?(URI::MailTo::EMAIL_REGEXP) - end - def same?(portal, archive) portal.to_s.squish.casecmp?(archive.to_s.squish) end @@ -394,32 +390,34 @@ def names(model, ids) # Every address the row answers to, not just the one it names as primary — # FmRolodex#addresses lists the primary first, so the first that can be built is the - # person's primary and the rest ride along. A work address, one the person already - # has, and a primary they already hold are all left as they are. + # person's primary and the rest ride along. An organization's address (a work address, + # or any address on an agency's row) and a primary they already hold are left as they + # are; the organizations import places the first. def build_addresses(person, rolodex) - held = person.addresses.map { |address| address_key(address.street_address, address.city) } - attributes = rolodex.addresses.reject { |address| work_address?(rolodex, address) } - .filter_map { |address| @address_attributes.call(address.data) } - .reject { |attrs| held.include?(address_key(attrs[:street_address], attrs[:city])) } - build_primary_first(person.addresses, attributes) - end - - # A work address belongs to the person's organization, so the affiliations import brings it in. - def work_address?(rolodex, address) - address.fm_id == rolodex.data["WorksiteAddrsID"] || - contact_type(address.data["AddrsType"], allowed: Address::CONTACT_TYPES) == "work" + attributes = rolodex.addresses.reject { |address| rolodex.organization_address?(address) } + .filter_map { |address| @address_attributes.call(address.data)&.merge(filemaker_code: address.fm_id) } + attach(person.addresses, attributes) { |address| address_key(address[:street_address], address[:city]) } end def build_phones(person, rolodex) - held = person.contact_methods.map { |phone| phone_key(phone.value) } - attributes = rolodex.phone_numbers.filter_map { |phone| phone_attributes(phone.data) } - .reject { |attrs| held.include?(phone_key(attrs[:value])) } - build_primary_first(person.contact_methods, attributes) + attributes = rolodex.phone_numbers.filter_map { |phone| phone_attributes(phone) } + attach(person.contact_methods, attributes) { |phone| phone_key(phone[:value]) } end - def build_primary_first(association, attributes) + # Each archive row lands on one record carrying its code: the one the person already + # holds for it (the same address or number), or a new one — the first of which is + # primary when the person has none. + def attach(association, attributes) has_primary = association.any?(&:primary?) - attributes.each_with_index { |attrs, index| association.build(attrs.merge(primary: index.zero? && !has_primary)) } + attributes.each do |attrs| + held = association.find { |record| yield(record) == yield(attrs) } + if held + held.filemaker_code = held.class.join_filemaker_codes(held.filemaker_code, attrs[:filemaker_code]) + else + association.build(attrs.merge(primary: !has_primary)) + has_primary = true + end + end end def address_key(street, city) @@ -430,60 +428,12 @@ def phone_key(value) value.to_s.gsub(/\D/, "") end - def phone_attributes(fields) - number = clip(fields["PhoneNumber"]) + def phone_attributes(phone) + number = clip(phone.data["PhoneNumber"]) return if number.blank? - { kind: "phone", value: number, contact_type: contact_type(fields["PhoneType"], allowed: ContactMethod::CONTACT_TYPES) } - end - - # The archive's types map onto one set of ours, but each model declares the slice it - # accepts — an address can be the mailing one, a phone number can't. A type we can't - # place is left blank, which both models then record as unknown. - def contact_type(value, allowed:) - CONTACT_TYPES[value.to_s.strip.downcase].presence_in(allowed) - end - - # Every string copied here lands in a varchar(255) the archive's free text isn't - # bounded by, and an over-long incidental field would otherwise cost the person — - # or, for a column with no length validation behind it, raise mid-import. - def clip(value) - value.presence && value.to_s.truncate(STRING_LIMIT) - end - - def parse_date(value) - value = value.to_s.strip - return if value.blank? - - DATE_FORMATS.each do |format| - parsed = parse_with(value, format) - return parsed if parsed - end - nil - end - - # Parsing stops where the format ends, so "10-5-1983" read as %Y-%m-%d takes 19 - # as the day and drops the 83 — digits left over mean the format split the value - # in the wrong place, so the next one gets its turn. - def parse_with(value, format) - parts = Date._strptime(value, format) - return if parts.nil? || parts[:leftover].to_s.match?(/\A\d/) - - expand_century(Date.new(parts[:year], parts[:mon], parts[:mday])) - rescue Date::Error, TypeError - nil - end - - # %Y reads a two-digit year as the year itself, so "10/5/83" is 83 AD. The archive - # mixes two- and four-digit years, so a bare pair gets a century: 83 is 1983, 05 is 2005. - def expand_century(date) - return date if date.year >= 100 - - date.change(year: date.year + (date.year < TWO_DIGIT_YEAR_PIVOT ? 2000 : 1900)) - end - - def flag?(value) - value.to_s.strip.presence && !FALSE_FLAGS.include?(value.to_s.strip.downcase) + { kind: "phone", value: number, contact_type: contact_type(phone.data["PhoneType"], allowed: ContactMethod::CONTACT_TYPES), + filemaker_code: phone.fm_id } end def ids(people) diff --git a/app/services/person_services/reconcile_contact_methods.rb b/app/services/person_services/reconcile_contact_methods.rb deleted file mode 100644 index fb53a97cd1..0000000000 --- a/app/services/person_services/reconcile_contact_methods.rb +++ /dev/null @@ -1,92 +0,0 @@ -module PersonServices - # After a person merge, the generic merge moves both people's contact methods onto - # the keeper wholesale (contact_methods have no unique index), leaving identical - # duplicates and possibly two primary phones of the same kind. Merge contact methods - # that share the same kind + value into one, then settle a single primary per kind: - # the keeper's own pre-merge primary, or the deleted person's when the keeper had - # none. Runs in the people deduper's after_merge. - class ReconcileContactMethods - IDENTITY_COLUMNS = %w[kind value contact_type].freeze - - # Outside the identity, so a merge never drops the address link only one copy had. - CARRIED_COLUMNS = %w[address_id].freeze - - def self.signature(contact_method) - return unless contact_method - - IDENTITY_COLUMNS.map do |column| - value = contact_method.public_send(column).to_s.strip.downcase - # A row written before an untyped number said so holds a blank, so unknown and - # blank have to read alike or an otherwise identical pair won't fold. - value == ContactMethod::UNKNOWN_CONTACT_TYPE && column == "contact_type" ? "" : value - end - end - - # One preserved-primary signature per kind, read before the merge while the two - # people's contact methods can still be told apart: the keeper's own primary of - # that kind, else the deleted person's. - def self.primary_signatures(keep, delete) - methods = keep.contact_methods.to_a + delete.contact_methods.to_a - methods.map(&:kind).uniq.filter_map do |kind| - primary = methods.find { |method| method.kind == kind && method.primary? && method.contactable == keep } || - methods.find { |method| method.kind == kind && method.primary? } - signature(primary) - end - end - - def initialize(person, primary_signatures: []) - @person = person - @primary_signatures = primary_signatures - end - - def call - survivors = {} - merge_candidates.each do |contact_method| - signature = self.class.signature(contact_method) - if (survivor = survivors[signature]) - merge(contact_method, into: survivor) - else - survivors[signature] = contact_method - end - end - - settle_primary(survivors.values) - end - - private - - # Active before inactive, then the primary, then oldest — so a merge keeps the - # copy still in use. `Person#phone_number` reads only active phones, so letting - # an inactive duplicate win would leave the person looking phoneless. - def merge_candidates - @person.contact_methods.reload.sort_by do |contact_method| - [ contact_method.inactive? ? 1 : 0, contact_method.primary? ? 0 : 1, contact_method.id ] - end - end - - def merge(loser, into:) - carry_over(loser, into) - deduper.merge(into, loser) - end - - def carry_over(loser, survivor) - filled = CARRIED_COLUMNS.select do |column| - survivor.public_send(column).blank? && loser.public_send(column).present? - end - return if filled.empty? - - survivor.update!(filled.index_with { |column| loser.public_send(column) }) - end - - def settle_primary(contact_methods) - contact_methods.each do |contact_method| - primary = @primary_signatures.include?(self.class.signature(contact_method)) - contact_method.update!(primary: primary) if contact_method.primary? != primary - end - end - - def deduper - @deduper ||= ModelDeduper.new(model_class: ContactMethod, logger: Rails.logger, dry_run: false) - end - end -end diff --git a/app/services/person_services/undo_rolodex_import.rb b/app/services/person_services/undo_rolodex_import.rb index 84be389436..89c121a390 100644 --- a/app/services/person_services/undo_rolodex_import.rb +++ b/app/services/person_services/undo_rolodex_import.rb @@ -1,7 +1,7 @@ module PersonServices - # Reverses one fm:import_people run from the activity events ImportFromRolodex logged under - # it (`import_run`). Newest first, so a person's addresses, tags and comments go before the - # person. A record created by the run is deleted, and a field the run filled goes back to + # Reverses one FileMaker import run (fm:import_people, fm:import_organizations) from the + # activity events it logged under it (`import_run`). Newest first, so a record's addresses, + # tags, comments and affiliations go before the record. A record created by the run is deleted, and a field the run filled goes back to # its value before the run — unless someone has changed it since, which is kept and reported. class UndoRolodexImport OUTCOMES = %i[deleted reverted kept gone].freeze diff --git a/app/views/admin/data_health/_legacy_role_row.html.erb b/app/views/admin/data_health/_legacy_role_row.html.erb new file mode 100644 index 0000000000..74c13158f4 --- /dev/null +++ b/app/views/admin/data_health/_legacy_role_row.html.erb @@ -0,0 +1,37 @@ +<%# Locals: record (an Affiliation titled with an old-app role). The person opens their edit + page at the affiliation; below it, whether they already have a Facilitator row here and + which one, if any, records the same stretch. %> +<% check = DataHealth::LegacyRoleAffiliations.new %> +<% facilitators = check.facilitators_for(record) %> +<% twin = check.twin_for(record, facilitators) %> +<% elsewhere = check.facilitator_elsewhere_count(record) %> +<% back = { return_to: "data_health", check: check.key } %> +
+ <% if record.person %> + <%= link_to person_name(record.person), edit_person_path(record.person, **back, anchor: dom_id(record)), target: "_blank", rel: "noopener", + class: "font-medium text-gray-800 hover:text-blue-700 hover:underline" %> + <% end %> + — <%= record.organization&.name %> + <% if facilitators.any? %> + <%= render "shared/badge", label: "Has Facilitator row here", classes: "bg-green-50 text-green-800 border-green-300", icon: "fa-solid fa-circle-check" %> + <% else %> + <%= render "shared/badge", label: "No Facilitator row here", classes: "bg-amber-50 text-amber-800 border-amber-300", icon: "fa-solid fa-circle-exclamation" %> + <% end %> +
+
+ <%= record.title %> · <%= record.decorate.period_label.presence || "No dates" %><%= " · primary contact" if record.primary_contact? %> + <% if record.filemaker_code.present? %> · FileMaker <%= record.filemaker_code %><% end %> +
+<% facilitators.each do |facilitator| %> + <% same = check.same_dates?(facilitator, record) %> +
"> + Facilitator #<%= facilitator.id %> · <%= facilitator.decorate.period_label.presence || "No dates" %><%= " · primary contact" if facilitator.primary_contact? %> + · <%= check.stretch_label(facilitator, record, twin) %> +
+<% end %> +<% if facilitators.any? && twin.nil? %> +
None of these covers this row's stretch.
+<% end %> +<% if elsewhere.positive? %> +
Also a Facilitator at <%= pluralize(elsewhere, "other organization") %>.
+<% end %> diff --git a/app/views/organizations/_form.html.erb b/app/views/organizations/_form.html.erb index 7c415ba3b4..da9e236fe7 100644 --- a/app/views/organizations/_form.html.erb +++ b/app/views/organizations/_form.html.erb @@ -96,20 +96,32 @@ input_html: { class: "w-full rounded-md border-gray-300 shadow-sm focus:ring-blue-500 focus:border-blue-500" } %> -
+
Phones
- <% phones = f.object.decorate.address_phones %> - <% numbered = f.object.addresses.size > 1 %> - <% if phones.any? %> -
    - <% phones.each do |number, phone| %> -
  1. <% if numbered %><%= number %>. <% end %><%= phone %>
    2. - <% end %> -
- <% else %> -

None yet.

+ <%= f.simple_fields_for :contact_methods do |sfi| %> + <%= render "contact_methods/contact_method_fields", f: sfi %> + <% end %> + + <%= link_to_add_association "➕ Add phone", + f, + :contact_methods, + partial: "contact_methods/contact_method_fields", + data: { association_insertion_node: "this" }, + class: button_classes(:secondary_outline) %> + + <% address_phones = f.object.decorate.address_phones %> + <% if address_phones.any? %> + <% numbered = f.object.addresses.size > 1 %> +
+
On addresses
+
    + <% address_phones.each do |number, phone| %> +
  • <% if numbered %><%= number %>. <% end %><%= phone %>
    • + <% end %> +
+

Saved per <%= link_to (numbered ? "address number" : "address"), "#addresses", class: "underline" %>

+
<% end %> -

Saved per <%= link_to (numbered ? "address number" : "address"), "#addresses", class: "underline" %>

diff --git a/app/views/organizations/show.html.erb b/app/views/organizations/show.html.erb index fe45c2e5b9..cac2beac2f 100644 --- a/app/views/organizations/show.html.erb +++ b/app/views/organizations/show.html.erb @@ -86,11 +86,7 @@ <% end %> <% if @organization.profile_show_phone? %> - <% phone = if @organization.addresses.loaded? - @organization.addresses.reject(&:inactive?).filter_map(&:phone).first - else - @organization.addresses.active.filter_map(&:phone).first - end %> + <% phone = @organization.phone_number %> <% if phone.present? %> <%= link_to phone, tel_href(phone), class: "hover:text-primary" %> diff --git a/db/migrate/20261025200004_add_filemaker_code_to_addresses_and_contact_methods.rb b/db/migrate/20261025200004_add_filemaker_code_to_addresses_and_contact_methods.rb new file mode 100644 index 0000000000..6cf9ddd2f1 --- /dev/null +++ b/db/migrate/20261025200004_add_filemaker_code_to_addresses_and_contact_methods.rb @@ -0,0 +1,11 @@ +class AddFilemakerCodeToAddressesAndContactMethods < ActiveRecord::Migration[8.1] + def up + add_column :addresses, :filemaker_code, :string + add_column :contact_methods, :filemaker_code, :string + end + + def down + remove_column :contact_methods, :filemaker_code, if_exists: true + remove_column :addresses, :filemaker_code, if_exists: true + end +end diff --git a/db/schema.rb b/db/schema.rb index 7a60730000..59fa4cf573 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -10,7 +10,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema[8.1].define(version: 2026_10_25_125134) do +ActiveRecord::Schema[8.1].define(version: 2026_10_25_200004) do create_table "action_text_mentions", charset: "utf8mb4", collation: "utf8mb4_unicode_ci", force: :cascade do |t| t.bigint "action_text_rich_text_id", null: false t.datetime "created_at", null: false @@ -89,6 +89,7 @@ t.integer "updated_by_id" t.string "zip_code", null: false t.boolean "invoice_address" + t.string "filemaker_code" t.index ["addressable_type", "addressable_id", "invoice_address"], name: "index_addresses_on_addressable_and_invoice_role" t.index ["addressable_type", "addressable_id"], name: "index_addresses_on_addressable" t.index ["created_by_id"], name: "index_addresses_on_created_by_id" @@ -459,6 +460,7 @@ t.datetime "updated_at", null: false t.integer "updated_by_id" t.string "value", null: false + t.string "filemaker_code" t.index ["address_id"], name: "index_contact_methods_on_address_id" t.index ["contactable_type", "contactable_id"], name: "index_contact_methods_on_contactable" t.index ["created_by_id"], name: "index_contact_methods_on_created_by_id" diff --git a/db/seeds/features/2026-10-25-filemaker-codes-on-addresses-and-phones.yml b/db/seeds/features/2026-10-25-filemaker-codes-on-addresses-and-phones.yml new file mode 100644 index 0000000000..e1bff6abbb --- /dev/null +++ b/db/seeds/features/2026-10-25-filemaker-codes-on-addresses-and-phones.yml @@ -0,0 +1,13 @@ +name: "FileMaker codes on imported addresses and phones" +area: people +display_status: admin_facing +released_on: 2026-10-25 +pr_number: 2860 +summary: >- + Addresses and phone numbers brought in from FileMaker — for people and + organizations — now carry the FileMaker record they came from, so each one + can be traced back to the archive. +pro_tips: + - "An address or phone's page in the FileMaker archive links back to the person or organization holding it." + - "When FileMaker has an address or number the record already had, the existing one takes the FileMaker code rather than getting a copy." + - "Merging duplicate addresses or phones keeps every FileMaker code they carried." diff --git a/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml b/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml new file mode 100644 index 0000000000..abf00d286e --- /dev/null +++ b/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml @@ -0,0 +1,31 @@ +name: "FileMaker organizations, programs, and affiliations" +area: people +display_status: admin_facing +released_on: 2026-10-25 +pr_number: 2788 +summary: >- + Organizations and art programs from the old FileMaker system can now be + brought into the portal, along with who worked or facilitated where and the + work addresses that go with them. +description: >- +
  • An organization running one art program comes in as one organization. + One running several gets each program nested under it.
  • Each program + carries FileMaker's locality and districts onto its address.
  • FileMaker's + facilitator assignments become Facilitator affiliations, dated from when they + were entered to when the program closed. Leaders and liaisons are marked as + primary contacts, and a comment on each records the role FileMaker gave.
  • A + person with a separate stretch of time at the same organization keeps a separate + affiliation for it.
  • An assignment whose program wasn't in FileMaker lands at + the organization on the person's contact card.
  • A program FileMaker lists + with nobody assigned keeps its history through the placeholder person AWBW + Placeholder.
  • Job titles from FileMaker become job affiliations.
  • Work + addresses land on the organization, and the person's affiliation there points at + them.
  • Agency contact cards become organizations with their addresses as + work addresses, and a contact person they name comes in as a person + affiliated there, at that work address.
+pro_tips: + - "An organization or affiliation already in the portal is linked rather than duplicated, and only its blank fields are filled in." + - "Where FileMaker has an organization's website, mission or type differently, the portal's is kept and a comment on the organization lists what FileMaker had." + - "An affiliation with different dates, an address the organization doesn't have, or a new phone number comes in as its own record, leaving the portal's as it was." + - "When either the portal's affiliation or FileMaker's has no dates, the portal's is updated instead of adding a second one." +action_path: "/organizations" diff --git a/db/seeds/features/2026-10-25-legacy-role-affiliations-in-data-health.yml b/db/seeds/features/2026-10-25-legacy-role-affiliations-in-data-health.yml new file mode 100644 index 0000000000..ec436cedca --- /dev/null +++ b/db/seeds/features/2026-10-25-legacy-role-affiliations-in-data-health.yml @@ -0,0 +1,16 @@ +name: "See leftover old-app role affiliations in Data health" +area: people +display_status: admin_facing +released_on: 2026-10-25 +pr_number: 2788 +summary: >- + Data health lists affiliations still titled Leader, Liaison, Assistant or + "default" — the old app's roles, which don't count toward program status — each + with the person's Facilitator affiliations there, so you can open the person + and clean it up. +pro_tips: + - "Find it under Data health → Affiliations → Affiliations still titled with an old-app role." + - "Each row says whether the person already has a Facilitator affiliation at that organization, and lists it." + - "A row marked “same dates” sits beside a Facilitator affiliation that already covers it; one marked “overlaps with different dates” needs a look at which dates are right." + - "Each name opens the person's edit page at that affiliation, and its back link returns you to Data health." +action_path: "/admin/data_health" diff --git a/db/seeds/features/2026-10-25-organization-phones.yml b/db/seeds/features/2026-10-25-organization-phones.yml new file mode 100644 index 0000000000..41c1996e56 --- /dev/null +++ b/db/seeds/features/2026-10-25-organization-phones.yml @@ -0,0 +1,14 @@ +name: "Phone numbers on organizations" +area: people +display_status: admin_facing +released_on: 2026-10-25 +pr_number: 2860 +summary: >- + Organizations now keep their own list of phone numbers, edited the same way + as a person's, with one marked primary. The FileMaker import brings an + organization's numbers in there. +pro_tips: + - "Add phones in the Phones section of the organization's edit page." + - "The profile shows the primary phone, or a phone saved on one of its addresses when it has none." + - "Merging two organizations keeps both sets of phones, folding a shared number into one." +action_path: "/organizations" diff --git a/docs/adr/0005-filemaker-code-archive-mapping.md b/docs/adr/0005-filemaker-code-archive-mapping.md index 8f5ea1b3b5..ea7c096496 100644 --- a/docs/adr/0005-filemaker-code-archive-mapping.md +++ b/docs/adr/0005-filemaker-code-archive-mapping.md @@ -10,7 +10,7 @@ source of truth across the board: staff edited records in both FileMaker and the portal, so for some data the portal is right and for some FileMaker is. The raw export lives in the read-only `fm_*` archive tables (one per FileMaker table, keyed by `fm_id`, every other field in a `data` JSON column; see `lib/tasks/fm_import.rake`). -Six app tables carry a `filemaker_code` column, read as a comma-separated set of +Eight app tables carry a `filemaker_code` column, read as a comma-separated set of codes (`FilemakerCoded`). The code is an archive `fm_id`, but nothing recorded *which* archive table each column's codes belong to. The dev seed (`db/seeds/dev/fm_archive.rb`) had to choose, so this records the choice. @@ -25,10 +25,12 @@ drive the archive pages' links. |---|---|---|---| | `people.filemaker_code` | `fm_rolodexes` | `ID`, e.g. `00006` | `filemaker_code_links` on person edit + admin details | | `organizations.filemaker_code` | `fm_projects`, then `fm_organizations` | `ProjectID`, e.g. `P2775`; `OrgID`, e.g. `ORG07206` | `filemaker_code_links` on org form; `filemaker_project_links` / `filemaker_organization_links` on org edit | -| `affiliations.filemaker_code` | `fm_projects`, then `fm_personnels` | `ProjectID`, e.g. `P2775`; rarely `PrsnlRecID`, e.g. `PS9999` | `filemaker_code_links` on affiliation edit | +| `affiliations.filemaker_code` | `fm_projects`, then `fm_personnels`, then `fm_people_to_organizations` | `ProjectID`, e.g. `P2775`; `PrsnlRecID`, e.g. `PS9999`, and `RecordID`, e.g. `112`, from the ADR-0008 import | `filemaker_code_links` on affiliation edit | | `payments.filemaker_code` | `fm_payments` | `RecordID`, e.g. `D29186` | `filemaker_code_hint` on payment edit | | `workshops.filemaker_code` | `fm_workshops` | `WorkshopID` | `filemaker_code_hint` on workshop form | | `resources.filemaker_code` | none | `RecordID` of the source quotation, e.g. `Q10215` | resource form (editable, no archive link) | +| `addresses.filemaker_code` | `fm_addresses` | `AddrsID`, e.g. `1234` | `fm_archives#show` links back to the address's person or org | +| `contact_methods.filemaker_code` | `fm_phone_numbers` | `PhoneID`, e.g. `5678` | `fm_archives#show` links back to the phone's person or org | - **Organization → Project, or → Organization.** An app `Organization` is usually one art program site, which is a FileMaker *Project*. FileMaker's own Organization row is @@ -64,7 +66,7 @@ when a new export lands. | Archive table | FileMaker key | Id pattern | Range in prod | Rows | Portal column | |---|---|---|---|---|---| | `fm_activities` | `ActivityLogID` | 5 digits | `00011`–`30061` | 29,185 | | -| `fm_addresses` | `AddrsID` | 1–5 digits, unpadded | `1`–`30808` | 28,556 | | +| `fm_addresses` | `AddrsID` | 1–5 digits, unpadded | `1`–`30808` | 28,556 | `addresses` | | `fm_allocations` | `AllocRecID` | 4 digits | `0001`–`3508` | 3,098 | | | `fm_ancillary_data` | `RecordID` | 1–3 digits, unpadded | `1`–`559` | 556 | | | `fm_art_purchases` | `PurchaseID` | 4 digits | `0001`–`0226` | 226 | | @@ -95,7 +97,7 @@ when a new export lands. | `fm_payments` | `RecordID` | `D` + 5 digits | `D00001`–`D29263` | 27,220 | `payments` | | `fm_people_to_organizations` | `RecordID` | 1–4 digits, unpadded | `1`–`1128` | 1,128 | | | `fm_personnels` | `PrsnlRecID` | `PS` + 4–5 digits | `PS0028`–`PS10667` | 9,037 | `affiliations` | -| `fm_phone_numbers` | `PhoneID` | 2–5 digits, unpadded | `22`–`35946` | 34,040 | | +| `fm_phone_numbers` | `PhoneID` | 2–5 digits, unpadded | `22`–`35946` | 34,040 | `contact_methods` | | `fm_postal_codes` | `Zipcode` | 5-digit ZIP | `00501`–`99950` | 41,685 | | | `fm_program_sponsorships` | `RecordID` | `PSP` + 4 digits | `PSP0035`–`PSP0901` | 608 | | | `fm_projects` | `ProjectID` | `P` + 4 digits | `P0001`–`P2806` | 2,007 | `organizations` | diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 0a55b214ab..5580d2d3ef 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -19,6 +19,8 @@ The archive tables involved (row counts from the export): | `fm_projects` | 1,976 | One art program run at an organization (`ProjectID`, `OrgID`) | | `fm_personnels` | 8,907 | A person assigned to a project, with role and agreement status (`PersonID`, `ProjectID`) | | `fm_people_to_organizations` | 1,127 | A person's job at an organization, with title (`PersonID`, `OrgID`) | +| `fm_rolodexes` | 39,608 | The person's own card: their organization (`OrgID`, `Organization`), `Title`, worksite (`WorksiteAddrsID`); or, flagged `isAgency__c`, an agency's card | +| `fm_addresses`, `fm_phone_numbers`, `fm_email_addresses` | 28,398 / 33,915 / 29 | Contact rows, filed under a Rolodex card (`RolodexID`), an organization (`OrgID`), or both | Constraints already decided elsewhere: @@ -37,20 +39,89 @@ Constraints already decided elsewhere: - **Order:** people (#2630) → organizations → projects → affiliations. Each pass finds the records the previous one created by `filemaker_code`, so each is idempotent on its own. + `fm:import_organizations` runs organizations + projects + (`OrganizationServices::ImportFromFilemaker`), then affiliations + (`AffiliationServices::ImportFromFilemaker`). - **Organizations and projects:** an `fm_projects` row becomes (or links to) an app `Organization`; its `fm_organizations` row becomes (or links to) that org's parent. Match an existing org by any shared `filemaker_code` first, then by exact name; append the code on a match, as the Rolodex pass does for people. + - **An organization with exactly one project is one org**, carrying both codes — a parent + with a single child would be the same place twice. Only an organization with several + projects (or none) gets its own org with the programs nested under it. When the portal + already keeps the two apart, it stays that way. + - **A name match only reaches an org carrying no code from the same table.** One already + holding another project's code is a namesake; the new org is named `"Name (P1234)"`, + since org names are unique. + - **Organization type** comes from the FileMaker organization's `OrgType`, else the + project's `AgencyType`, first match winning: "501" or "nonprofit" → 501c3/nonprofit; + "for-profit" or "LLC" → For-profit; "government" or "govt" → Government agency; "Other" + → Other. Any other word ("School", "Educational Institution", "Individual", "Grassroots") + is Other with the word as its description. A value naming two types, as FileMaker's + multi-select stored them, takes the first. + - Every FileMaker organization comes over, whether or not a project or person points at it + (decided 2026-10-24). +- **Agencies:** a Rolodex card flagged `isAgency__c` is the agency's own card. Its website, + mission and addresses fill the org its `OrgID` names (or the one its `Organization` name + matches, created when nothing does); its email too, unless the card names a contact + person, whose email it then is. A card naming a contact person also becomes that person, + and every address on it stays with the org. `fm:import_people` skipped every agency card, + so `fm:import_organizations` runs the Rolodex import over agency cards alone as its first + step — the people already imported and cleaned up are left as they are. - **Personnel → facilitator affiliation:** an `fm_personnels` row becomes an `Affiliation` between the person (by Rolodex code) and the project's org, carrying `PrsnlRecID` as its - `filemaker_code` so each row imports once. `Role` → `title`; `Status` / dates → - `start_date` / `end_date` / `inactive`. `event_registration_id` stays NULL — these are + `filemaker_code` so each row imports once. `event_registration_id` stays NULL — these are historical, not training-minted. + - **Role:** matches how prod already records the old app's roles. The January 2026 + migration (`ConvertProjectUserPositionToTitle`, in #735, deleted once applied) turned that + app's position enum (liaison/leader/assistant) into titles, and #2321 folded Leader into + Facilitator + primary contact. Prod, profiled 2026-10-24: 1,829 Facilitators are primary + contacts (402 aren't), all 53 Liaisons are, and 111 rows are still titled "Leader" (which + the exact-match Facilitator scope doesn't count, and which the import matches as a + Facilitator rather than duplicating). Of those, 109 sit beside an undated, primary + Facilitator twin the January conversion made, and 2 have none. The Data health check + "Affiliations still titled with an old-app role" lists every Leader, Liaison, Assistant or + "default" row with the person's Facilitator rows at that org and links to their edit page, + where an admin settles it by hand; nothing changes them automatically. The import matches + any of those titles as a Facilitator. + - Every role becomes a **Facilitator** affiliation, with a "FileMaker import" comment + recording the role FileMaker gave. No prod row is titled Assistant, so the old app's + assistants likely became the 402 Facilitators who aren't primary contacts. + - Only a role FileMaker names **Leader** (8,112 of 8,907 rows) or **Liaison** (611) is + `primary_contact`. Matching an existing Facilitator row turns its primary contact on — + the one value the import overwrites rather than fills — and the comment says so. + - **Assistant** (232), a blank role (80) or a stray word isn't a primary contact, and + leaves a matched row's flag alone. + - **One row, one stretch.** A person can facilitate at one org over several separate + stretches (2020–2022, 2024–Oct 2025, Feb 2026–now), so a Personnel row links an existing + Facilitator row only when their dates agree — each the same or blank on one side, and + still start-before-end once the blanks fill. A row with no dates on either side updates + the other (taking its dates, or the code alone). Any other dates are a separate stretch + and come in as a new affiliation (decided 2026-10-25), leaving the portal's untouched. + - **A row naming a project but no person** (29 of the blank-role rows) still records that + the program ran. Once every other row is read, an organization nothing else affiliates + anyone with gets one Facilitator affiliation for the placeholder person **AWBW + Placeholder** (found by name, created when missing — distinct from the existing AWBW + Staff), with a comment saying so. + - **Dates:** Personnel rows carry none of their own, so the assignment starts on + `DateEntered` and ends on the project's `CloseDate` (else `EndDate`). Only `Status` + "Active" (1,774) stays open; Inactive (7,220), Pending, Suspended, blank, and values + typed over each other ("ActiveInactive") end it now, undated (`inactive`). The run tallies + every Status and Role value and how it was read. + - An existing same-title affiliation at that org is linked (takes the code, fills blank + dates) unless it already records another Personnel row — a second stint stays its own row. - **People-to-organizations → job affiliation:** an `fm_people_to_organizations` row becomes - an `Affiliation` to the umbrella org with its `Title`, carrying `RecordID` as its code. -- **Work addresses:** the Rolodex `WorksiteAddrsID` address and any address typed Work - become an `Address` on the organization, and the person's affiliation there points at it - (`affiliations.organization_address_id`). + an `Affiliation` to the org carrying its `OrgID` code with its `Title`, carrying `RecordID` + as its code. Undated, per ADR-0003 D1. +- **The Rolodex card → job affiliation:** the card's `OrgID` (or its typed `Organization`, via + `NameMatcher`) and `Title` become a job affiliation there — the field `FmRolodex::PORTAL_FIELDS` + already mapped to `affiliations.title`. No code: the card's ID is the person's. +- **Work addresses:** the Rolodex `WorksiteAddrsID` address, any address typed Work, and + every address on an agency's card become an `Address` on the organization, and the + person's affiliations there that name no address point at it + (`affiliations.organization_address_id`), the worksite first. The organization is the one + FileMaker filed the address under (its `OrgID`), else the card's, else the person's only + one. A work address with no organization to land on stays with the person, typed work. - **Locality and districts come from the project, not derived.** `fm_projects` is the one place FileMaker records a `Locality`, and its vocabulary is where `Address::LOCALITIES` came from — so a project's `Locality`, `District`, `SPA` and `LASupervisorialDistrict` @@ -62,22 +133,41 @@ Constraints already decided elsewhere: - **Addresses are built by `FmAddressAttributes`** (ADR-0009), as the Rolodex pass does, so state, country, type and locality are stored as the dropdowns offer; a project's city/state goes through `AddressTranslation` the same way. -- **Conflicts:** same rule as people — fill blanks, keep the portal value where the archive - disagrees, and report each difference as a comment plus in the task output. +- **Conflicts:** a field on a record — an organization's website, mission, type — fills blanks + and keeps the portal value where the archive disagrees, reported as a comment plus in the + task output. A record itself comes in new instead: an affiliation whose dates differ, an + address the organization doesn't hold, and a phone number it doesn't hold — a phone + record (`contact_methods`) on the organization, as a person's are (decided 2026-10-25). + Each address and phone carries its archive row's ID as its own `filemaker_code` (ADR-0005). - **Out of scope:** event registrations (`fm_participants`, 68,497 rows), payments, funding, notes and activities — each its own pass. ## Open questions -- **Which org does a job affiliation attach to** — the umbrella (`fm_organizations`) or the - person's project at that org, when there is exactly one? -- **Reconcile vs. history.** The reconcile page ends facilitator affiliations no attended - training backs, hand-entered ones included. Imported personnel rows would be swept up - unless they're exempted (e.g. by their `PS…` code) — exempt them, or let reconcile end - them? -- **Which `Status` / `AgreementStatus` values mean active**, and is a closed project - (`CloseDate`, `EndDate`) an end date for every affiliation under it? -- **Merged projects** (`MergerWithID`): fold the merged-away project into the survivor's - org, or skip it? -- **Unmatched people:** a personnel row whose `PersonID` no person carries — skip and - report, or create the person from the Rolodex row on the spot? +The first draft of the import (#2788) answers these provisionally; confirm or change before +the real run. + +- **Which org does a job affiliation attach to** — the one carrying the `OrgID` code. For an + organization with one project that's the program's own org, so the question only remains + for organizations with several projects, where it's the umbrella. +- **Reconcile vs. history.** Still open. The reconcile page ends facilitator affiliations no + attended training backs, hand-entered ones included, and imported personnel rows are + hand-entered by that test. Exempt rows carrying a `PS…` code, or let reconcile end them? +- **Which `Status` values mean active** — only "Active", from the production values profiled + 2026-10-24 (see Personnel above). A closed project's `CloseDate` (else `EndDate`) ends every + affiliation under it. `AgreementStatus` isn't read: 8,376 of 8,907 rows leave it blank. +- **Is an Assistant a facilitator?** Yes — a Facilitator who isn't a primary contact + (decided 2026-10-24). +- **Merged projects** (`MergerWithID`): folded — the merged-away project's code joins the + survivor's org, so its personnel land there. One the portal already keeps as its own org + is reported for a hand merge rather than merged automatically. +- **Unmatched people:** skipped and reported. A row naming no person at all — or only an + agency's own Rolodex card — gets AWBW Placeholder instead, when its organization has no + other affiliations (decided 2026-10-24). +- **Personnel with no project** (27 rows in prod, none with a project on their agreement): + the row lands at the organization on the person's Rolodex card — its `OrgID`, else its + typed name, created as a new organization when nothing matches. A card naming no + organization is skipped and reported (decided 2026-10-25). +- **A full accounting of what didn't come over** — every FileMaker person and organization no + portal record carries a code for — is a later Data health check, read through + `filemaker_code`. diff --git a/lib/tasks/fm_import_organizations.rake b/lib/tasks/fm_import_organizations.rake new file mode 100644 index 0000000000..59cc0725e4 --- /dev/null +++ b/lib/tasks/fm_import_organizations.rake @@ -0,0 +1,119 @@ +# frozen_string_literal: true + +# Bring FileMaker's organizations, projects and the ties between them and people into the +# portal (ADR-0008). Run after fm:import_people, which makes the people; this pass finds them +# by their Rolodex code. Three steps, in order: +# 1. PersonServices::ImportFromRolodex over agency cards only — the contact person an agency +# card names, which fm:import_people skipped +# 2. OrganizationServices::ImportFromFilemaker — organizations, projects (as programs nested +# under them), their addresses, phones, emails, and agencies' own Rolodex cards +# 3. AffiliationServices::ImportFromFilemaker — personnel, people-to-organizations, and each +# Rolodex card's organization, title and work addresses +# Each is idempotent: a row whose code a record already carries is left alone. +# +# Dry-run by default. Set DRY_RUN=false to write changes; VERBOSE=true lists every row. +# A real run commits each batch as it goes, so stopping partway keeps what it finished and +# the rerun skips it. The full report is also saved as an ActiveStorage blob. Every record a +# run creates or fills is logged under the run's name, which fm:undo_import reverses: +# bin/rails fm:import_organizations # preview, writes nothing +# DRY_RUN=false bin/rails fm:import_organizations # create + link +# DRY_RUN=false bin/rails fm:undo_import RUN=fm-import-organizations-20261022134055 +namespace :fm do + desc "Create organizations and affiliations from the FileMaker archive (after fm:import_people). Idempotent. DRY_RUN=false to execute." + task import_organizations: :environment do + unless Setting.import_user + abort "No import user chosen — pick one on /admin/settings (Imports); it's credited as created_by/updated_by when FileMaker names no staffer we can resolve." + end + + dry_run = ENV["DRY_RUN"] != "false" + verbose = ENV["VERBOSE"] == "true" + report = [] + say = ->(line = "") { puts line; report << line } + $stdout.sync = true + + begin + run = "fm-import-organizations-#{Time.current.strftime("%Y%m%d%H%M%S")}" + Current.source = "fm_import_organizations" + say.("Run #{run}#{" (dry run)" if dry_run}") + started = Time.current + # The people pass skipped every agency card, so the contact person one names comes in + # here — only those cards, leaving everyone fm:import_people already brought over alone. + steps = { + "People named on agency cards" => ->(progress) { PersonServices::ImportFromRolodex.new(scope: FmRolodex.agency_cards, progress:, run:) }, + "Organizations and projects" => ->(progress) { OrganizationServices::ImportFromFilemaker.new(progress:, run:) }, + "Affiliations and work addresses" => ->(progress) { AffiliationServices::ImportFromFilemaker.new(progress:, run:) } + } + imports = {} + + run_steps = lambda do + steps.each do |title, build| + say.("\n#{title}…") + progress = ->(read) { puts " #{read} rows read — #{(Time.current - started).round}s" } + imports[title] = build.call(progress).call + end + end + + # A dry run rolls back as a whole; a real one lets each batch commit as it goes. + if dry_run + ActiveRecord::Base.transaction do + run_steps.call + raise ActiveRecord::Rollback + end + else + run_steps.call + end + + imports.each do |title, import| + say.("\n== #{title} ==") + import.results.each do |outcome, messages| + next unless (verbose || %i[failed skipped].include?(outcome)) && messages.any? + + say.("\n#{outcome.to_s.humanize} (#{messages.size}):") + messages.each { |message| say.(" #{message}") } + end + + if import.try(:differences).present? + say.("\nKept the portal's value where FileMaker differs (#{import.differences.size}, also left as a comment):") + import.differences.each { |difference| say.(" #{difference}") } + end + + tallies = { + "Organization types read as Other, kept as its description" => import.try(:unmatched_types), + "Personnel Status values and how they were read" => import.try(:statuses), + "Personnel roles" => import.try(:roles), + "Address countries nothing translates (left blank)" => import.try(:unmatched_addresses)&.fetch(:country, nil), + "Address states nothing translates (kept as typed)" => import.try(:unmatched_addresses)&.fetch(:state, nil) + } + tallies.each do |heading, counts| + next if counts.blank? + + say.("\n#{heading}:") + counts.sort_by { |name, count| [ -count, name ] }.each { |name, count| say.(" #{count.to_s.rjust(5)} #{name}") } + end + + say.() + import.results.each { |outcome, messages| say.("#{outcome.to_s.humanize.ljust(17)} #{messages.size}") } + end + + say.("\nTook #{(Time.current - started).round}s.") + say.(dry_run ? "\nDRY RUN — rolled back, nothing written. Set DRY_RUN=false to execute." : "\nDone.") + say.("To reverse this run: DRY_RUN=false bin/rails fm:undo_import RUN=#{run}") unless dry_run + rescue StandardError => e + say.("\nStopped by #{e.class}: #{e.message}#{" — the batches before this one are saved; rerun to continue." unless dry_run}") + raise + ensure + begin + blob = ActiveStorage::Blob.create_and_upload!( + io: StringIO.new(report.join("\n")), content_type: "text/plain", + filename: "fm-import-organizations-#{Time.current.strftime("%Y%m%d-%H%M%S")}#{"-dry-run" if dry_run}.txt" + ) + puts "\nReport saved. Read it back with: bin/rails runner 'puts ActiveStorage::Blob.find(#{blob.id}).download'" + rescue StandardError => e + puts "\nCould not save the report (#{e.class}: #{e.message}) — copy the output above." + end + end + end + + desc "Reverse any FileMaker import run (RUN=fm-import-…) from the activity it logged. Dry-run by default; DRY_RUN=false to execute." + task undo_import: :undo_import_people +end diff --git a/lib/tasks/fm_import_people.rake b/lib/tasks/fm_import_people.rake index 4db19c862c..e4d33c70aa 100644 --- a/lib/tasks/fm_import_people.rake +++ b/lib/tasks/fm_import_people.rake @@ -107,7 +107,7 @@ namespace :fm do desc "Reverse one fm:import_people run (RUN=fm-import-people-…) from the activity it logged. Dry-run by default; DRY_RUN=false to execute." task undo_import_people: :environment do - run = ENV["RUN"].presence || abort("Name the run to reverse: RUN=fm-import-people-… (the import prints it).") + run = ENV["RUN"].presence || abort("Name the run to reverse: RUN=fm-import-… (the import prints it).") dry_run = ENV["DRY_RUN"] != "false" Current.source = "fm_undo_import_people" undo = nil diff --git a/spec/models/concerns/filemaker_coded_spec.rb b/spec/models/concerns/filemaker_coded_spec.rb index 4c1d7bd465..aa83ad1ac8 100644 --- a/spec/models/concerns/filemaker_coded_spec.rb +++ b/spec/models/concerns/filemaker_coded_spec.rb @@ -65,7 +65,7 @@ it "names the archive tables each coded model points into" do expect(Person.filemaker_archives).to eq([ FmRolodex ]) expect(Organization.filemaker_archives).to eq([ FmProject, FmOrganization ]) - expect(Affiliation.filemaker_archives).to eq([ FmProject, FmPersonnel ]) + expect(Affiliation.filemaker_archives).to eq([ FmProject, FmPersonnel, FmPeopleToOrganization ]) end end end diff --git a/spec/models/organization_spec.rb b/spec/models/organization_spec.rb index ac4791c451..f1c9c359bc 100644 --- a/spec/models/organization_spec.rb +++ b/spec/models/organization_spec.rb @@ -13,6 +13,26 @@ it { should have_many(:users).through(:people) } it { should have_many(:reports) } it { should have_many(:workshop_logs) } + it { should have_many(:contact_methods).dependent(:destroy) } + end + + describe "#phone_number" do + let(:organization) { create(:organization) } + + it "prefers the primary active phone" do + organization.contact_methods.create!(kind: "phone", value: "555-0001") + organization.contact_methods.create!(kind: "phone", value: "555-0002", primary: true) + organization.contact_methods.create!(kind: "phone", value: "555-0003", primary: true, inactive: true) + + expect(organization.reload.phone_number).to eq("555-0002") + end + + it "falls back to a phone an active address holds" do + create(:address, addressable: organization, phone: "555-0004", inactive: true) + create(:address, addressable: organization, phone: "555-0005") + + expect(organization.reload.phone_number).to eq("555-0005") + end end describe 'validations' do diff --git a/spec/requests/admin/data_health_spec.rb b/spec/requests/admin/data_health_spec.rb index 6efc97da77..f0ad514be2 100644 --- a/spec/requests/admin/data_health_spec.rb +++ b/spec/requests/admin/data_health_spec.rb @@ -182,6 +182,33 @@ def stories_without_workshop(number) expect(frame.text).to include("11–12 of 12") end + it "lists old-app role rows with their Facilitator twin, each opening the person's edit page" do + person = create(:person) + twin = create(:affiliation, person:, organization:, title: "Facilitator") + leader = create(:affiliation, person:, organization:, title: "Leader", filemaker_code: "P0001") + + get admin_data_health_rows_path(check: "legacy_role_affiliations"), + headers: { "Turbo-Frame" => "data_health_rows_legacy_role_affiliations" } + + expect(response).to have_http_status(:ok) + frame = Nokogiri::HTML(response.body).at_css("turbo-frame#data_health_rows_legacy_role_affiliations") + expect(frame.text).to include("Has Facilitator row here", "Facilitator ##{twin.id}", "same dates", "FileMaker P0001") + expect(frame.to_html).to include(CGI.escapeHTML(edit_person_path(person, return_to: "data_health", check: "legacy_role_affiliations", + anchor: "affiliation_#{leader.id}"))) + end + + it "says when an old-app role row's person has no Facilitator row at its organization" do + person = create(:person) + create(:affiliation, person:, organization:, title: "Liaison") + create(:affiliation, person:, title: "Facilitator") + + get admin_data_health_rows_path(check: "legacy_role_affiliations"), + headers: { "Turbo-Frame" => "data_health_rows_legacy_role_affiliations" } + + frame = Nokogiri::HTML(response.body).at_css("turbo-frame#data_health_rows_legacy_role_affiliations") + expect(frame.text).to include("No Facilitator row here", "Also a Facilitator at 1 other organization") + end + it "is a 404 for an unknown check" do get admin_data_health_rows_path(check: "no_such_check") diff --git a/spec/requests/dedupable_spec.rb b/spec/requests/dedupable_spec.rb index fd2c1f4dde..ecbf3acb81 100644 --- a/spec/requests/dedupable_spec.rb +++ b/spec/requests/dedupable_spec.rb @@ -415,6 +415,20 @@ expect(Organization.exists?(delete_rec.id)).to be true end + it "moves the duplicate's phones onto the keeper, folding a shared number into one" do + keep.contact_methods.create!(kind: "phone", value: "555-0100", primary: true, filemaker_code: "10") + delete_rec.contact_methods.create!(kind: "phone", value: "555-0100", primary: true, filemaker_code: "11") + delete_rec.contact_methods.create!(kind: "phone", value: "555-0199") + + post dedupe_perform_organizations_path, params: { + organization_to_delete_id: delete_rec.id, + organization_to_keep_id: keep.id + } + + expect(keep.contact_methods.order(:id).pluck(:value, :primary, :filemaker_code)) + .to eq([ [ "555-0100", true, "10, 11" ], [ "555-0199", false, nil ] ]) + end + it "combines both organizations' FileMaker codes onto the keeper" do keep.update!(filemaker_code: "FM-100") delete_rec.update!(filemaker_code: "FM-273") diff --git a/spec/requests/fm_archives_spec.rb b/spec/requests/fm_archives_spec.rb index 27486889fe..c17b4323db 100644 --- a/spec/requests/fm_archives_spec.rb +++ b/spec/requests/fm_archives_spec.rb @@ -335,6 +335,28 @@ expect(response.body).to include(ERB::Util.html_escape(workshop.title)) end + it "links an archived address back to the person holding it" do + FmAddress.create!(fm_id: "123") + person = create(:person) + create(:address, addressable: person, street_address: "1 Main St", filemaker_code: "123") + + get fm_archive_path("123", table: "addresses") + + expect(response.body).to include(person_path(person)) + expect(response.body).to include(ERB::Util.html_escape("#{person.name}: 1 Main St")) + end + + it "links an archived phone number back to the person holding it" do + FmPhoneNumber.create!(fm_id: "456") + person = create(:person) + person.contact_methods.create!(kind: "phone", value: "555-0100", filemaker_code: "456") + + get fm_archive_path("456", table: "phone_numbers") + + expect(response.body).to include(person_path(person)) + expect(response.body).to include(ERB::Util.html_escape("#{person.name}: 555-0100")) + end + it "omits the portal section when nothing carries the code" do FmRolodex.create!(fm_id: "00006") diff --git a/spec/requests/organizations_spec.rb b/spec/requests/organizations_spec.rb index 3db97c173d..d97c433bcb 100644 --- a/spec/requests/organizations_spec.rb +++ b/spec/requests/organizations_spec.rb @@ -271,18 +271,30 @@ end describe "GET /edit" do - it "lists address phones in the details, numbered by address when there are several" do + it "lists address phones under the phone editor, numbered by address when there are several" do organization = create(:organization) create(:address, addressable: organization, phone: "123-123-1234") create(:address, addressable: organization, phone: "123-332-1234") get edit_organization_url(organization) - phones = Capybara.string(response.body).find("ol", text: "123-123-1234") + phones = Capybara.string(response.body).find("#phones") + expect(phones).to have_text("On addresses") expect(phones).to have_css("li", text: "1. 123-123-1234") expect(phones).to have_css("li", text: "2. 123-332-1234") end + it "edits the organization's own phone numbers beside them" do + organization = create(:organization) + organization.contact_methods.create!(kind: "phone", value: "555-0100") + + get edit_organization_url(organization) + + phones = Capybara.string(response.body).find("#phones") + expect(phones).to have_field(with: "555-0100") + expect(phones).to have_link("➕ Add phone") + end + it "leaves focus alone so in-page links keep their scroll position" do get edit_organization_url(create(:organization)) @@ -477,6 +489,14 @@ expect(organization.reload.high_profile).to be(true) end + it "saves the organization's phones" do + organization = Organization.create!(valid_attributes) + patch organization_url(organization), params: { organization: { contact_methods_attributes: { + "0" => { kind: "phone", contact_type: "work", value: "555-0100", primary: "1" } + } } } + expect(organization.contact_methods.sole).to have_attributes(value: "555-0100", contact_type: "work", primary: true) + end + it "nests the organization under a parent" do parent = Organization.create!(valid_attributes.merge(name: "Roof Org")) organization = Organization.create!(valid_attributes) diff --git a/spec/services/address_deduper_spec.rb b/spec/services/address_deduper_spec.rb index 55110d858a..7a5bba855b 100644 --- a/spec/services/address_deduper_spec.rb +++ b/spec/services/address_deduper_spec.rb @@ -151,6 +151,15 @@ def address(primary: false, **overrides) expect(keep.reload.phone).to eq("555-9000") end + it "keeps every FileMaker code the merged addresses carried" do + keep = address(filemaker_code: "12") + address(filemaker_code: "7, 30") + + described_class.new(person).call + + expect(keep.reload.filemaker_code).to eq("12, 30, 7") + end + it "leaves the survivor's own phone alone when the duplicate also has one" do keep = address(phone: "555-1000") address(phone: "555-9000") diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb new file mode 100644 index 0000000000..f1fd15cd97 --- /dev/null +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -0,0 +1,357 @@ +# frozen_string_literal: true + +require "rails_helper" + +RSpec.describe AffiliationServices::ImportFromFilemaker do + let!(:person) { create(:person, filemaker_code: "00006") } + let!(:program) { create(:organization, filemaker_code: "P0001") } + let!(:umbrella) { create(:organization, filemaker_code: "ORG00001") } + + before { FmProject.create!(fm_id: "P0001", data: { "OrgID" => "ORG00001", "CloseDate" => "6/30/2019" }) } + + def personnel(fm_id, **data) + FmPersonnel.create!(fm_id:, data: { "PersonID" => "00006", "ProjectID" => "P0001", "Role" => "Leader", + "Status" => "Active", "DateEntered" => "3/1/2015" }.merge(data.stringify_keys)) + end + + def rolodex(**data) + FmRolodex.create!(fm_id: "00006", data: { "FirstName" => "Ada", "LastName" => "Lovelace" }.merge(data.stringify_keys)) + end + + def import + described_class.new.call + end + + describe "personnel" do + it "makes a Leader a primary-contact Facilitator dated from entry to the project's close" do + personnel("PS0001") + + expect(import.results[:created].size).to eq(1) + expect(person.affiliations.sole).to have_attributes(organization: program, title: "Facilitator", filemaker_code: "PS0001", primary_contact: true, + start_date: Date.new(2015, 3, 1), end_date: Date.new(2019, 6, 30), + event_registration_id: nil) + end + + it "ends an assignment whose status isn't active, and tallies the statuses it read" do + FmProject.find_by!(fm_id: "P0001").update!(data: { "OrgID" => "ORG00001" }) + personnel("PS0001", Status: "Dropped") + + import = described_class.new.call + + expect(person.affiliations.sole).to have_attributes(end_date: nil, inactive: true) + expect(import.statuses).to eq("Dropped → ended" => 1) + end + + it "makes a Liaison a primary-contact Facilitator, noting the role FileMaker gave" do + personnel("PS0001", Role: "Liaison") + + import = described_class.new.call + + affiliation = person.affiliations.sole + expect(affiliation).to have_attributes(title: "Facilitator", primary_contact: true, start_date: Date.new(2015, 3, 1)) + expect(affiliation.comments.sole).to have_attributes(topic: "FileMaker import", body: a_string_including("Liaison", "PS0001")) + expect(import.roles).to eq("Liaison → Facilitator, primary contact" => 1) + expect(affiliation.comments.sole.body).not_to include("Primary contact turned on") + end + + it "notes a Leader's role too, but not a blank one" do + personnel("PS0001") + personnel("PS0002", Role: "", DateEntered: "1/5/2018") + + import + + expect(Comment.where(commentable: person.affiliations).pluck(:body)).to contain_exactly(a_string_including("Leader", "PS0001")) + end + + it "makes an Assistant a Facilitator who isn't a primary contact, noting the role" do + personnel("PS0002", Role: "assistant") + + import = described_class.new.call + + affiliation = person.affiliations.sole + expect(affiliation).to have_attributes(title: "Facilitator", primary_contact: false, start_date: Date.new(2015, 3, 1)) + expect(affiliation.comments.sole.body).to include("assistant", "PS0002") + expect(import.roles).to eq("assistant → Facilitator" => 1) + end + + it "turns on primary contact for a Leader's existing Facilitator row, and says so" do + existing = create(:affiliation, person:, organization: program, title: "Facilitator", primary_contact: false) + personnel("PS0001") + + import + + expect(existing.reload).to have_attributes(primary_contact: true, filemaker_code: "PS0001") + expect(existing.comments.sole.body).to include("Leader", "Primary contact turned on") + end + + it "leaves an Assistant's existing Facilitator row's primary contact as it is" do + existing = create(:affiliation, person:, organization: program, title: "Facilitator", primary_contact: true) + personnel("PS0001", Role: "Assistant") + + import + + expect(existing.reload.primary_contact).to be(true) + expect(existing.comments.sole.body).not_to include("Primary contact turned on") + end + + it "makes a primary contact only where FileMaker's role says Leader or Liaison" do + personnel("PS0001", Role: "Leaders") + personnel("PS0002", Role: "", DateEntered: "1/5/2018") + personnel("PS0003", Role: "Something else", DateEntered: "1/5/2017") + + import = described_class.new.call + + expect(person.affiliations.order(:filemaker_code).pluck(:filemaker_code, :title, :primary_contact)) + .to eq([ [ "PS0001", "Facilitator", true ], [ "PS0002", "Facilitator", false ], [ "PS0003", "Facilitator", false ] ]) + expect(import.roles).to eq("Leaders → Facilitator, primary contact" => 1, "(blank) → Facilitator" => 1, + "Something else → Facilitator" => 1) + end + + describe "matching an existing Facilitator row by its dates" do + before { FmProject.find_by!(fm_id: "P0001").update!(data: { "OrgID" => "ORG00001" }) } + + def stint(starts, ends = nil) + create(:affiliation, person:, organization: program, title: "Facilitator", start_date: starts, end_date: ends) + end + + it "links the row whose dates agree, filling the one it leaves blank" do + stint(Date.new(2020, 1, 1), Date.new(2022, 12, 31)) + same = stint(Date.new(2024, 3, 1)) + personnel("PS0001", DateEntered: "3/1/2024") + + import + + expect(same.reload.filemaker_code).to eq("PS0001") + expect(person.affiliations.count).to eq(2) + end + + it "imports a row whose dates differ as its own affiliation, leaving the portal's alone" do + existing = stint(Date.new(2024, 1, 1), Date.new(2025, 10, 31)) + personnel("PS0001", DateEntered: "3/1/2024") + + import + + expect(existing.reload).to have_attributes(filemaker_code: nil, start_date: Date.new(2024, 1, 1)) + expect(person.affiliations.find_by(filemaker_code: "PS0001")).to have_attributes(start_date: Date.new(2024, 3, 1)) + expect(Comment.where(commentable: existing)).to be_empty + end + + it "doesn't link a row whose blank start would put its end before the archive's start" do + ended = stint(nil, Date.new(2012, 12, 31)) + personnel("PS0001", DateEntered: "3/1/2015") + + import + + expect(ended.reload.filemaker_code).to be_nil + expect(person.affiliations.count).to eq(2) + end + + it "links the portal's dated row when the archive's row has no dates, keeping its dates" do + dated = stint(Date.new(2019, 1, 1), Date.new(2021, 6, 30)) + personnel("PS0001", DateEntered: "") + + import + + expect(dated.reload).to have_attributes(filemaker_code: "PS0001", start_date: Date.new(2019, 1, 1), end_date: Date.new(2021, 6, 30)) + expect(person.affiliations.count).to eq(1) + end + + it "links a row with no dates at all, taking the archive's" do + undated = stint(nil) + personnel("PS0001", DateEntered: "3/1/2024") + + import + + expect(undated.reload).to have_attributes(filemaker_code: "PS0001", start_date: Date.new(2024, 3, 1)) + end + end + + it "links the Facilitator affiliation the person already has there, filling its blank start" do + existing = create(:affiliation, person:, organization: program, title: "Facilitator", filemaker_code: "P0001", start_date: nil) + personnel("PS0001") + + expect(import.results[:linked].size).to eq(1) + expect(existing.reload).to have_attributes(filemaker_code: "P0001, PS0001", start_date: Date.new(2015, 3, 1)) + expect(person.affiliations.count).to eq(1) + end + + it "links a row the old app titled Liaison instead of adding a Facilitator beside it" do + existing = create(:affiliation, person:, organization: program, title: "Liaison", primary_contact: true) + personnel("PS0001", Role: "Liaison") + + import + + expect(person.affiliations.sole).to eq(existing) + expect(existing.reload.filemaker_code).to eq("PS0001") + end + + it "links a row the old app titled Leader instead of adding a Facilitator beside it" do + existing = create(:affiliation, person:, organization: program, title: "Leader", primary_contact: true) + personnel("PS0001") + + import + + expect(person.affiliations.sole).to eq(existing) + expect(existing.reload).to have_attributes(title: "Leader", filemaker_code: "PS0001") + end + + it "keeps a second stint on the same project as its own affiliation" do + personnel("PS0001") + personnel("PS0002", DateEntered: "1/5/2018") + + import + + expect(person.affiliations.pluck(:filemaker_code)).to contain_exactly("PS0001", "PS0002") + end + + describe "a row naming a project but no person" do + it "puts AWBW Placeholder on an organization nothing else affiliates anyone with" do + personnel("PS0001", PersonID: "", Role: "") + personnel("PS0002", PersonID: "", Role: "", DateEntered: "1/5/2018") + + import = described_class.new.call + + placeholder = Person.find_by!(first_name: "AWBW", last_name: "Placeholder") + affiliation = placeholder.affiliations.sole + expect(affiliation).to have_attributes(organization: program, title: "Facilitator", primary_contact: false, filemaker_code: "PS0001") + expect(affiliation.comments.sole.body).to include("PS0001", "names no person") + expect(import.results[:skipped]).to contain_exactly(a_string_including("PS0002", "already has affiliations")) + end + + it "leaves an organization that already has affiliations alone, even from a later row" do + personnel("PS0001", PersonID: "") + personnel("PS0002") + + import + + expect(Person.where(first_name: "AWBW", last_name: "Placeholder")).to be_empty + expect(program.affiliations.sole.person).to eq(person) + end + + it "reuses the placeholder person already in the portal" do + existing = create(:person, first_name: "AWBW", last_name: "Placeholder", email: nil, user: nil) + personnel("PS0001", PersonID: "") + + import + + expect(existing.affiliations.count).to eq(1) + end + end + + describe "a row whose project wasn't imported" do + it "lands at the organization the person's Rolodex card names" do + rolodex(OrgID: "ORG00001") + personnel("PS0001", ProjectID: "") + + import + + expect(person.affiliations.sole).to have_attributes(organization: umbrella, title: "Facilitator", filemaker_code: "PS0001") + end + + it "creates the organization the card only names, and reports it" do + rolodex(Organization: "Foothill Family Services") + personnel("PS0001", ProjectID: "") + + import = described_class.new.call + + created = Organization.find_by!(name: "Foothill Family Services") + expect(person.affiliations.sole.organization).to eq(created) + expect(import.results[:created]).to include(a_string_including("organization ##{created.id}", "Rolodex card")) + end + + it "gives an agency's own card AWBW Placeholder at the agency" do + FmRolodex.create!(fm_id: "00777", data: { "isAgency__c" => "1", "OrgID" => "ORG00001" }) + personnel("PS0001", PersonID: "00777", ProjectID: "") + + import + + expect(umbrella.affiliations.sole.person).to have_attributes(first_name: "AWBW", last_name: "Placeholder") + end + + it "is skipped, saying why, when the card names no organization either" do + rolodex + personnel("PS0001", ProjectID: "") + + expect(import.results[:skipped].sole).to include("PS0001", "no project", "Rolodex card names no organization") + end + end + + it "skips a row whose person or project wasn't imported" do + personnel("PS0001", PersonID: "99999") + personnel("PS0002", ProjectID: "P9999") + + expect(import.results[:skipped]).to contain_exactly(a_string_including("PS0001", "99999"), a_string_including("PS0002", "P9999")) + expect(Affiliation.count).to eq(0) + end + end + + it "makes a job affiliation from a people-to-organizations row" do + FmPeopleToOrganization.create!(fm_id: "12", data: { "PersonID" => "00006", "OrgID" => "ORG00001", "Title" => "Board Chair" }) + + import + + expect(person.affiliations.sole).to have_attributes(organization: umbrella, title: "Board Chair", filemaker_code: "12", start_date: nil) + end + + describe "the Rolodex card" do + it "affiliates the person with their organization and points them at its worksite" do + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", "AddrsState" => "CA" }) + FmAddress.create!(fm_id: "41", data: { "RolodexID" => "00006", "AddrsLine1" => "2 Home Rd", "AddrsCity" => "Pasadena", "AddrsState" => "CA", + "AddrsType" => "Home" }) + rolodex(OrgID: "ORG00001", Title: "Counselor", WorksiteAddrsID: "40", PrimaryAddrsID: "41") + + import + + affiliation = person.affiliations.sole + expect(affiliation).to have_attributes(organization: umbrella, title: "Counselor", filemaker_code: nil) + expect(affiliation.organization_address).to have_attributes(addressable: umbrella, street_address: "1 Main St", address_type: "work", + filemaker_code: "40") + expect(person.addresses).to be_empty + end + + it "adds the archive row's code to a work address the organization already has" do + held = create(:address, addressable: umbrella, street_address: "1 Main Street", city: "Pasadena", state: "CA") + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", + "AddrsState" => "CA", "AddrsType" => "Work" }) + rolodex(OrgID: "ORG00001") + + import + + expect(umbrella.addresses.sole).to eq(held) + expect(held.reload.filemaker_code).to eq("40") + expect(person.affiliations.sole.organization_address).to eq(held) + end + + it "places a work address on the person's only organization when the card names none" do + create(:affiliation, person:, organization: program, title: "Facilitator") + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", + "AddrsState" => "CA", "AddrsType" => "Work" }) + rolodex + + import + + expect(person.affiliations.sole.organization_address).to have_attributes(addressable: program, street_address: "1 Main St") + end + + it "keeps a work address with the person when no organization can take it" do + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", + "AddrsState" => "CA", "AddrsType" => "Work" }) + rolodex + + import + + expect(person.addresses.sole).to have_attributes(street_address: "1 Main St", address_type: "work", filemaker_code: "40") + end + end + + it "changes nothing when it runs again" do + personnel("PS0001") + personnel("PS0002", Role: "Liaison") + FmPeopleToOrganization.create!(fm_id: "12", data: { "PersonID" => "00006", "OrgID" => "ORG00001", "Title" => "Board Chair" }) + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", "AddrsState" => "CA" }) + rolodex(OrgID: "ORG00001", Title: "Counselor", WorksiteAddrsID: "40") + import + + expect { import }.not_to change { [ Affiliation.count, Address.count, Comment.count, Affiliation.maximum(:updated_at) ] } + end +end diff --git a/spec/services/person_services/reconcile_contact_methods_spec.rb b/spec/services/contact_method_deduper_spec.rb similarity index 90% rename from spec/services/person_services/reconcile_contact_methods_spec.rb rename to spec/services/contact_method_deduper_spec.rb index 601bc8228c..72e55f773c 100644 --- a/spec/services/person_services/reconcile_contact_methods_spec.rb +++ b/spec/services/contact_method_deduper_spec.rb @@ -2,7 +2,7 @@ require "rails_helper" -RSpec.describe PersonServices::ReconcileContactMethods do +RSpec.describe ContactMethodDeduper do let(:person) { create(:person) } def contact_method(value: "555-1000", kind: "phone", **overrides) @@ -93,4 +93,13 @@ def address expect(keep.reload.address_id).to eq(kept_address.id) end + + it "keeps every FileMaker code the merged contact methods carried" do + keep = contact_method(filemaker_code: "22") + contact_method(filemaker_code: "35") + + described_class.new(person).call + + expect(keep.reload.filemaker_code).to eq("22, 35") + end end diff --git a/spec/services/data_health/checks_spec.rb b/spec/services/data_health/checks_spec.rb index c5f4627fd3..ea8936823e 100644 --- a/spec/services/data_health/checks_spec.rb +++ b/spec/services/data_health/checks_spec.rb @@ -123,6 +123,51 @@ def affiliation_from(facilitator_training:, title: "Facilitator") end end +RSpec.describe DataHealth::LegacyRoleAffiliations do + let(:person) { create(:person) } + let(:organization) { create(:organization) } + + def affiliation(title, **attributes) = create(:affiliation, person:, organization:, title:, **attributes) + + it "lists affiliations titled with any old-app role, and reports only" do + titled = %w[Leader liaison Assistant default].map { |title| affiliation(title, start_date: Date.new(2019, 1, title.length)) } + affiliation("Facilitator", start_date: Date.new(2019, 1, 1)) + affiliation("Program Director") + + expect(described_class.new.scope).to match_array(titled) + expect(described_class.new).not_to be_repairable + end + + it "labels each Facilitator row by how its stretch meets the Leader row's" do + check = described_class.new + leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) + same = affiliation("Facilitator", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) + earlier = affiliation("Facilitator", start_date: Date.new(2010, 1, 1), end_date: Date.new(2012, 1, 1)) + + expect(check.stretch_label(same, leader, same)).to eq("same dates") + expect(check.stretch_label(earlier, leader, same)).to eq("a different stretch") + end + + describe "#twin_for" do + it "finds the Facilitator row with the same dates first" do + affiliation("Facilitator", start_date: Date.new(2015, 1, 1)) + same = affiliation("Facilitator", start_date: Date.new(2019, 1, 1)) + leader = affiliation("Leader", start_date: Date.new(2019, 1, 1)) + + expect(described_class.new.twin_for(leader)).to eq(same) + end + + it "falls back to one overlapping with different dates, and is nil when none covers the stretch" do + overlapping = affiliation("Facilitator", start_date: Date.new(2018, 1, 1), end_date: Date.new(2022, 12, 31)) + leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) + later = affiliation("Leader", start_date: Date.new(2024, 1, 1), end_date: Date.new(2025, 1, 1)) + + expect(described_class.new.twin_for(leader)).to eq(overlapping) + expect(described_class.new.twin_for(later)).to be_nil + end + end +end + RSpec.describe DataHealth::LegacyOrganizationStatusDrift do let!(:active_status) { OrganizationStatus.find_or_create_by!(name: "Active") } let!(:inactive_status) { OrganizationStatus.find_or_create_by!(name: "Inactive") } diff --git a/spec/services/organization_services/import_from_filemaker_spec.rb b/spec/services/organization_services/import_from_filemaker_spec.rb new file mode 100644 index 0000000000..8254112062 --- /dev/null +++ b/spec/services/organization_services/import_from_filemaker_spec.rb @@ -0,0 +1,213 @@ +# frozen_string_literal: true + +require "rails_helper" + +RSpec.describe OrganizationServices::ImportFromFilemaker do + def fm_organization(fm_id, **data) + FmOrganization.create!(fm_id:, data: { "OrgName" => "Harbor House" }.merge(data.stringify_keys)) + end + + def project(fm_id, org_id, **data) + FmProject.create!(fm_id:, data: { "OrgID" => org_id, "ProjectName" => "Harbor House Art Program" }.merge(data.stringify_keys)) + end + + def import + described_class.new.call + end + + it "keeps an organization with one project as one organization carrying both codes" do + fm_organization("ORG00001", Website: "harbor.example.org", Mission: "Shelter for all", OrgType: "Nonprofit") + project("P0001", "ORG00001", City: "Pasadena", State: "CA", County: "Los Angeles", Locality: "la county", + SPA: "SPA 3", LASupervisorialDistrict: "5", District: "Foothills") + FmAddress.create!(fm_id: "1", data: { "OrgID" => "ORG00001", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", + "AddrsState" => "CA", "AddrsPostalCode" => "91101" }) + FmPhoneNumber.create!(fm_id: "2", data: { "OrgID" => "ORG00001", "PhoneNumber" => "555-0100" }) + FmEmailAddress.create!(fm_id: "3", data: { "OrgID" => "ORG00001", "EmailAddress" => "Attn: info@harbor.example.org" }) + + expect(import.results[:created].size).to eq(1) + + organization = Organization.sole + expect(organization).to have_attributes(name: "Harbor House", filemaker_code: "ORG00001, P0001", parent_id: nil, + website_url: "harbor.example.org", mission_vision_values: "Shelter for all", + organization_type: "501c3/nonprofit", email: "info@harbor.example.org") + expect(organization.addresses.sole).to have_attributes(street_address: "1 Main St", city: "Pasadena", zip_code: "91101", + address_type: "work", primary: true, filemaker_code: "1", + locality: "LA County", district: "Foothills", + la_service_planning_area: 3, la_supervisorial_district: 5) + expect(organization.contact_methods.sole).to have_attributes(kind: "phone", value: "555-0100", primary: true, filemaker_code: "2") + end + + it "adds the archive row's code to an address the organization already has" do + existing = create(:organization, filemaker_code: "ORG00001") + address = create(:address, addressable: existing, street_address: "1 Main St", city: "Pasadena", state: "CA") + fm_organization("ORG00001") + FmAddress.create!(fm_id: "1", data: { "OrgID" => "ORG00001", "AddrsLine1" => "1 Main Street", "AddrsCity" => "Pasadena", + "AddrsState" => "CA" }) + + import + + expect(existing.addresses.sole).to eq(address) + expect(address.reload.filemaker_code).to eq("1") + end + + describe "organization type" do + { + "501c3/Nonprofit" => [ "501c3/nonprofit", nil ], + "Non-Profit Organization" => [ "501c3/nonprofit", nil ], + "501c3/Nonprofit Government Agency" => [ "501c3/nonprofit", nil ], + "For-Profit" => [ "For-profit", nil ], + "LLC" => [ "For-profit", nil ], + "County Government Agency" => [ "Government agency", nil ], + "Govt agency" => [ "Government agency", nil ], + "Other" => [ "Other", nil ], + "School" => [ "Other", "School" ] + }.each do |fm_type, (type, other)| + it "reads #{fm_type.inspect} as #{type}#{" (#{other})" if other}" do + fm_organization("ORG00001", OrgType: fm_type) + + import + + expect(Organization.sole).to have_attributes(organization_type: type, organization_type_other: other) + end + end + + it "takes the organization's type over its single project's" do + fm_organization("ORG00001", OrgType: "Government Agency") + project("P0001", "ORG00001", AgencyType: "School") + + import + + expect(Organization.sole).to have_attributes(organization_type: "Government agency", organization_type_other: nil) + end + + it "leaves the description off when the portal's type isn't Other" do + existing = create(:organization, filemaker_code: "ORG00001", organization_type: "For-profit") + fm_organization("ORG00001", OrgType: "School") + + import + + expect(existing.reload).to have_attributes(organization_type: "For-profit", organization_type_other: nil) + end + end + + describe "phones" do + def phone(fm_id, number, **data) + FmPhoneNumber.create!(fm_id:, data: { "OrgID" => "ORG00001", "PhoneNumber" => number }.merge(data.stringify_keys)) + end + + it "brings each number in as a phone carrying its code, the primary first" do + fm_organization("ORG00001") + phone("11", "555-0102") + phone("10", "555-0101", Primacy: "1", PhoneType: "Work") + + import + + expect(Organization.sole.contact_methods.order(:id).pluck(:value, :contact_type, :primary, :filemaker_code)) + .to eq([ [ "555-0101", "work", true, "10" ], [ "555-0102", "unknown", false, "11" ] ]) + end + + it "adds the code to a phone the organization already has, keeping its primary" do + existing = create(:organization, filemaker_code: "ORG00001") + held = existing.contact_methods.create!(kind: "phone", value: "(555) 0100", primary: true) + fm_organization("ORG00001") + phone("10", "555-0100") + phone("11", "555-0199", Primacy: "1") + + import + + expect(existing.contact_methods.order(:id).pluck(:value, :primary, :filemaker_code)) + .to eq([ [ "(555) 0100", true, "10" ], [ "555-0199", false, "11" ] ]) + expect(held.reload.filemaker_code).to eq("10") + end + end + + it "nests each of an organization's several projects under it" do + fm_organization("ORG00002", OrgName: "Valley Services") + project("P0002", "ORG00002", ProjectName: "Valley Adult Program", City: "Fresno", State: "CA", Locality: "Central CA") + project("P0003", "ORG00002", ProjectName: "Valley Youth Program") + + expect(import.results[:created].size).to eq(3) + + umbrella = Organization.find_by!(name: "Valley Services") + expect(umbrella.filemaker_code).to eq("ORG00002") + expect(umbrella.children.pluck(:name, :filemaker_code)).to contain_exactly([ "Valley Adult Program", "P0002" ], + [ "Valley Youth Program", "P0003" ]) + expect(Organization.find_by!(name: "Valley Adult Program").addresses.sole) + .to have_attributes(street_address: "", city: "Fresno", locality: "Central CA") + end + + it "fills an organization already carrying the code, keeping and noting what it holds" do + existing = create(:organization, name: "Harbor House Inc", filemaker_code: "P0001", website_url: "old.example.org") + fm_organization("ORG00001", Website: "harbor.example.org", Mission: "Shelter for all") + project("P0001", "ORG00001") + + import = described_class.new.call + + expect(import.results[:linked].size).to eq(1) + expect(existing.reload).to have_attributes(name: "Harbor House Inc", filemaker_code: "ORG00001, P0001", + website_url: "old.example.org", mission_vision_values: "Shelter for all") + expect(existing.comments.sole.body).to include("ORG00001", "old.example.org", "harbor.example.org") + expect(import.differences.sole).to include("##{existing.id}") + end + + it "matches by exact name only an organization with no code from the same table" do + taken = create(:organization, name: "Harbor House Art Program", filemaker_code: "P0009") + project("P0009", nil, ProjectName: "Somewhere else") + fm_organization("ORG00001", OrgName: "Harbor House") + project("P0001", "ORG00001", ProjectName: "Harbor House Art Program") + project("P0004", "ORG00001", ProjectName: "Harbor House Youth") + + import + + expect(taken.reload.filemaker_code).to eq("P0009") + expect(Organization.find_by!(name: "Harbor House Art Program (P0001)").parent.name).to eq("Harbor House") + end + + it "still brings in the projects of a nameless organization, unnested" do + fm_organization("ORG00003", OrgName: "") + project("P0006", "ORG00003", ProjectName: "Lone Program") + + expect(import.results[:skipped].sole).to include("ORG00003", "no name") + expect(Organization.sole).to have_attributes(name: "Lone Program", filemaker_code: "P0006", parent_id: nil) + end + + it "folds a merged-away project's code into the program it merged into" do + fm_organization("ORG00001") + project("P0001", "ORG00001") + project("P0005", "ORG00001", ProjectName: "Old Harbor Program", MergerWithID: "P0001") + + results = import.results + + expect(results[:linked].sole).to include("P0005", "merged into P0001") + expect(Organization.sole.filemaker_code).to eq("ORG00001, P0001, P0005") + end + + it "reads an agency's own Rolodex card onto its organization" do + fm_organization("ORG00001") + FmAddress.create!(fm_id: "7", data: { "RolodexID" => "00090", "AddrsLine1" => "9 Elm St", "AddrsCity" => "Ojai", + "AddrsState" => "CA", "AddrsType" => "Home" }) + FmRolodex.create!(fm_id: "00090", data: { "isAgency__c" => "1", "OrgID" => "ORG00001", "Organization" => "Harbor House", + "WebSite" => "harbor.example.org", "EMail" => "front@harbor.example.org", + "FirstName" => "Ada", "LastName" => "Lovelace" }) + FmRolodex.create!(fm_id: "00091", data: { "isAgency__c" => "1", "Organization" => "Elm Street Clinic", + "EMail" => "hello@elm.example.org" }) + + import + + harbor = Organization.find_by!(filemaker_code: "ORG00001") + expect(harbor).to have_attributes(website_url: "harbor.example.org", email: nil) + expect(harbor.addresses.sole).to have_attributes(street_address: "9 Elm St", address_type: "work") + expect(Organization.find_by!(name: "Elm Street Clinic")).to have_attributes(email: "hello@elm.example.org", filemaker_code: nil) + end + + it "changes nothing when it runs again" do + fm_organization("ORG00002", OrgName: "Valley Services") + project("P0002", "ORG00002", ProjectName: "Valley Adult Program", City: "Fresno", State: "CA") + project("P0003", "ORG00002", ProjectName: "Valley Youth Program") + FmAddress.create!(fm_id: "1", data: { "OrgID" => "ORG00002", "AddrsLine1" => "1 Main St", "AddrsCity" => "Fresno", "AddrsState" => "CA" }) + import + + expect { expect(import.results[:already_imported].size).to eq(3) } + .not_to change { [ Organization.count, Address.count, Comment.count ] } + end +end diff --git a/spec/services/person_services/import_from_rolodex_spec.rb b/spec/services/person_services/import_from_rolodex_spec.rb index c90cf7d3c6..a919cc7473 100644 --- a/spec/services/person_services/import_from_rolodex_spec.rb +++ b/spec/services/person_services/import_from_rolodex_spec.rb @@ -436,10 +436,22 @@ def results results person = Person.find_by!(filemaker_code: "00060") - expect(person.addresses.order(:id).pluck(:street_address, :primary)) - .to eq([ [ "1 Main St", true ], [ "2 Second St", false ] ]) - expect(person.contact_methods.order(:id).pluck(:value, :primary)) - .to eq([ [ "555-0100", true ], [ "555-0200", false ] ]) + expect(person.addresses.order(:id).pluck(:street_address, :primary, :filemaker_code)) + .to eq([ [ "1 Main St", true, "A21" ], [ "2 Second St", false, "A20" ] ]) + expect(person.contact_methods.order(:id).pluck(:value, :primary, :filemaker_code)) + .to eq([ [ "555-0100", true, "P21" ], [ "555-0200", false, "P20" ] ]) + end + + it "folds two archive rows for one address into one address carrying both codes" do + FmAddress.create!(fm_id: "A22", data: { "RolodexID" => "00061", "AddrsLine1" => "1 Main St", + "AddrsCity" => "Los Angeles", "AddrsState" => "CA" }) + FmAddress.create!(fm_id: "A23", data: { "RolodexID" => "00061", "AddrsLine1" => "1 Main Street", + "AddrsCity" => "Los Angeles", "AddrsState" => "CA" }) + rolodex("00061", EMail: "ada@example.com") + + results + + expect(Person.find_by!(filemaker_code: "00061").addresses.sole.filemaker_code).to eq("A22, A23") end it "reads the address and phone tables once, not once per row" do @@ -706,6 +718,22 @@ def run_events(run) = Ahoy::Event.where("JSON_UNQUOTE(JSON_EXTRACT(properties, ' .to eq([ [ "1 Main St", true ], [ "2 Second St", false ] ]) expect(person.contact_methods.pluck(:value, :primary)).to eq([ [ "555-0100", true ] ]) end + + it "adds the archive's codes to the address and phone the person already holds" do + address = create(:address, addressable: person, street_address: "1 Main St", city: "Los Angeles", state: "CA") + phone = person.contact_methods.create!(kind: "phone", value: "(555) 0100") + FmAddress.create!(fm_id: "A72", data: { "RolodexID" => "00074", "AddrsLine1" => "1 Main Street", + "AddrsCity" => "Los Angeles", "AddrsState" => "CA" }) + FmPhoneNumber.create!(fm_id: "P72", data: { "RolodexID" => "00074", "PhoneNumber" => "555-0100" }) + rolodex("00074", EMail: "ada@example.com") + + described_class.new.call + + expect(person.addresses.sole).to eq(address) + expect(address.reload.filemaker_code).to eq("A72") + expect(person.contact_methods.sole).to eq(phone) + expect(phone.reload.filemaker_code).to eq("P72") + end end it "links an existing person with the same name and email instead of creating one" do @@ -843,15 +871,23 @@ def run_events(run) = Ahoy::Event.where("JSON_UNQUOTE(JSON_EXTRACT(properties, ' end end - it "skips agencies, deleted rows, and nameless rows" do - rolodex("00013", isAgency__c: "1") + it "skips deleted rows, nameless rows, and agency cards naming no contact person" do + rolodex("00013", isAgency__c: "1", FirstName: "", LastName: "", Organization: "Harbor House") rolodex("00014", Delete: "Yes") rolodex("00015", FirstName: "", LastName: "") - expect(results[:skipped].size).to eq(3) + expect(results[:skipped]).to contain_exactly(a_string_including("00013", "agency"), a_string_including("00014"), a_string_including("00015")) expect(Person.where(filemaker_code: %w[00013 00014 00015])).to be_empty end + it "creates the contact person an agency card names, leaving the agency's addresses to its organization" do + FmAddress.create!(fm_id: "A1", data: { "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", "AddrsState" => "CA", "AddrsType" => "Home" }) + rolodex("00012", isAgency__c: "1", Organization: "Harbor House", EMail: "ada@harbor.example.org", PrimaryAddrsID: "A1") + + expect(results[:created].size).to eq(1) + expect(Person.find_by!(filemaker_code: "00012").addresses).to be_empty + end + it "treats a false-y flag as unset" do rolodex("00016", isAgency__c: "0", Delete: "No") diff --git a/spec/tasks/fm_import_organizations_spec.rb b/spec/tasks/fm_import_organizations_spec.rb new file mode 100644 index 0000000000..2136ff7d9b --- /dev/null +++ b/spec/tasks/fm_import_organizations_spec.rb @@ -0,0 +1,97 @@ +require "rails_helper" +require "rake" + +RSpec.describe "fm:import_organizations" do + before(:all) do + Rails.application.load_tasks unless Rake::Task.task_defined?("fm:import_organizations") + end + + after { Current.reset } + + let!(:person) { create(:person, filemaker_code: "00200") } + + before do + FmOrganization.create!(fm_id: "ORG00001", data: { "OrgName" => "Harbor House" }) + FmProject.create!(fm_id: "P0001", data: { "OrgID" => "ORG00001", "ProjectName" => "Harbor House Art Program" }) + FmPersonnel.create!(fm_id: "PS0001", data: { "PersonID" => "00200", "ProjectID" => "P0001", "Role" => "Leader", "Status" => "Active" }) + end + + def run_task(dry_run: true, task: "fm:import_organizations", run: nil) + ENV["DRY_RUN"] = dry_run ? "true" : "false" + ENV["RUN"] = run + output = StringIO.new + original_stdout = $stdout + $stdout = output + Rake::Task[task].reenable + Rake::Task[task].prerequisite_tasks.each(&:reenable) + Rake::Task[task].invoke + output.string + rescue SystemExit => e + raise "#{task} aborted: #{e.message}" + ensure + $stdout = original_stdout + ENV.delete("DRY_RUN") + ENV.delete("RUN") + end + + it "stops before reading a row when no import user is chosen" do + expect { run_task(dry_run: false) }.to raise_error(/aborted: No import user chosen/) + expect(Organization.count).to eq(0) + end + + context "with an import user chosen" do + before { Setting.create!(import_user: create(:user)) } + + it "rolls a dry run back and saves its report" do + expect { run_task }.to change(ActiveStorage::Blob, :count).by(1) + + expect(Organization.count).to eq(0) + expect(ActiveStorage::Blob.last.download).to include("Organizations and projects", "Affiliations and work addresses", "DRY RUN") + end + + it "makes the organization, then the affiliation, on a real run" do + run_task(dry_run: false) + + organization = Organization.sole + expect(organization.filemaker_code).to eq("ORG00001, P0001") + expect(person.affiliations.sole).to have_attributes(organization:, title: "Facilitator", filemaker_code: "PS0001") + end + + it "turns an agency card into an organization with its work address, and points the person named on it there" do + contact = create(:person, filemaker_code: "00300") + FmAddress.create!(fm_id: "9", data: { "RolodexID" => "00300", "AddrsLine1" => "9 Elm St", "AddrsCity" => "Ojai", + "AddrsState" => "CA", "AddrsType" => "Home" }) + FmRolodex.create!(fm_id: "00300", data: { "isAgency__c" => "1", "Organization" => "Elm Street Clinic", "Title" => "Director", + "FirstName" => "Ada", "LastName" => "Lovelace" }) + + run_task(dry_run: false) + + clinic = Organization.find_by!(name: "Elm Street Clinic") + address = clinic.addresses.sole + expect(address).to have_attributes(street_address: "9 Elm St", address_type: "work") + expect(contact.affiliations.sole).to have_attributes(organization: clinic, title: "Director", organization_address: address) + expect(contact.addresses).to be_empty + end + + it "brings in the person an agency card names, which the people import skipped, and affiliates them there" do + FmRolodex.create!(fm_id: "00301", data: { "isAgency__c" => "1", "Organization" => "Harbor Clinic", "Title" => "Director", + "FirstName" => "Grace", "LastName" => "Hopper", "EMail" => "grace@harbor.example.org" }) + FmRolodex.create!(fm_id: "00302", data: { "FirstName" => "Ada", "LastName" => "Lovelace" }) + + run_task(dry_run: false) + + contact = Person.find_by!(filemaker_code: "00301") + expect(contact.affiliations.sole).to have_attributes(organization: Organization.find_by!(name: "Harbor Clinic"), title: "Director") + expect(Person.where(filemaker_code: "00302")).to be_empty + end + + it "names the run so fm:undo_import can reverse it" do + run = run_task(dry_run: false)[/ RUN=(fm-import-organizations-\d+)/, 1] + + run_task(task: "fm:undo_import", run:, dry_run: false) + + expect(Organization.count).to eq(0) + expect(Affiliation.count).to eq(0) + end + end +end