From 9fb9090fa37b4913f22eaedb857047805ecc79df Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Thu, 22 Oct 2026 14:12:29 -0400 Subject: [PATCH 01/25] Start the FileMaker organizations and affiliations import Co-Authored-By: Claude Opus 5.5 (1M context) From 1d64a95df41180f874c77b8faedee3182bcf7ab6 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Thu, 22 Oct 2026 14:24:27 -0400 Subject: [PATCH 02/25] Import FileMaker organizations, programs, and affiliations The Rolodex pass brought people over but deliberately left out everything tying a person to an organization, so imported people have no jobs, no facilitator history, and no work addresses. This is the second pass ADR-0008 describes: organizations and projects become (nested) orgs, personnel and people-to-organizations rows become affiliations, and each Rolodex card's org, title and work addresses land where they belong. Agency cards were skipped wholesale by the people pass, losing the contact person many of them name; those people now come through, while the card's addresses and website stay with the agency's org. Raw-value parsing and address building move out of ImportFromRolodex so all three passes read the archive the same way. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 10 +- app/models/affiliation.rb | 2 +- app/models/fm_rolodex.rb | 15 + .../import_from_filemaker.rb | 319 +++++++++++++ app/services/filemaker_code_index.rb | 30 ++ app/services/filemaker_values.rb | 75 ++++ .../import_from_filemaker.rb | 422 ++++++++++++++++++ .../person_services/import_from_rolodex.rb | 82 +--- .../person_services/undo_rolodex_import.rb | 6 +- ...lemaker-organizations-and-affiliations.yml | 23 + .../0005-filemaker-code-archive-mapping.md | 2 +- ...maker-organization-and-personnel-import.md | 77 +++- lib/tasks/fm_import_organizations.rake | 112 +++++ lib/tasks/fm_import_people.rake | 2 +- .../import_from_filemaker_spec.rb | 135 ++++++ .../import_from_filemaker_spec.rb | 128 ++++++ .../import_from_rolodex_spec.rb | 14 +- spec/tasks/fm_import_organizations_spec.rb | 69 +++ 18 files changed, 1422 insertions(+), 101 deletions(-) create mode 100644 app/services/affiliation_services/import_from_filemaker.rb create mode 100644 app/services/filemaker_code_index.rb create mode 100644 app/services/filemaker_values.rb create mode 100644 app/services/organization_services/import_from_filemaker.rb create mode 100644 db/seeds/features/2026-10-22-filemaker-organizations-and-affiliations.yml create mode 100644 lib/tasks/fm_import_organizations.rake create mode 100644 spec/services/affiliation_services/import_from_filemaker_spec.rb create mode 100644 spec/services/organization_services/import_from_filemaker_spec.rb create mode 100644 spec/tasks/fm_import_organizations_spec.rb diff --git a/AGENTS.md b/AGENTS.md index 050419f5a8..762002fc7a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -304,6 +304,9 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `EventRegistrationServices::CalloutFormSubmission` — Records a registrant's answers to one of the forms a registration-ticket callout delivers inline (`call(registration:, callout:, form:, form_params:)`, returns the service instance with `#submission`/`#profile_changes`). Find-or-creates one `FormSubmission` per `(registrant, form, event)` carrying the form's own role — a re-submit edits the answers in place via `FormSubmission#persist_answer`. That a callout collected it is recorded separately in `FormSubmission#metadata` (`collected_via: "callout"`, `collected_via_callout_id`) via `#record_callout_collection!` — the flag (`#collected_via_callout?`), not the role, tells a callout-collected submission apart. There is no survey-role flag — the extra side effects self-gate on the form's own fields, so they no-op on a form that doesn't ask those questions: fields carrying the `anonymous_contributions` / `display_name_preference` identifiers write through to the `Person` via `PersonServices::SyncSharingPreferences` (the same write-through the registration form runs), exposing `#profile_changes` so the controller Ahoy-tracks real edits; and submitting any form on the **`scholarship`** callout stamps a recipient's `post_survey_completed_at`. On an **anonymous** form (`Form#anonymous?`) the registrant is credited only when the answers give first name, last name, and email (`Form#identified_by?`); otherwise the answers are saved with no person (a credited submission edited down to a partial identity is detached), no profile write-through runs, a `FormCompletion` records that the registrant finished, and a later submit is refused (`#already_completed?`) — as is one missing a required answer (`#incomplete?`). Drives `Events::CalloutsController#callout`/`#submit_callout` (the person-first `registration/:slug/forms/:callout_id` page and its `.../:form_id` submit) - `TagTranslation` — Turns a legacy word for who someone serves into tags: `#resolve` answers with both the sector ids and the age-group ids a term means, since the vocabularies cross (`Elder Abuse` is a life experience that means the Elders *age range*). A tag's own name resolves only while it's **published** — a retired tag is unpublished, not deleted — but a recorded `TagAlias` may point at a retired one deliberately (FileMaker's `Poverty`). Splits a repeating field on line breaks and semicolons — **not commas**, which the archive's own wording contains — and drops a trailing parenthetical. Built once and reused (it reads the whole vocabulary up front); a term nothing answers to lands in `#unmatched` rather than a guess. Primacy is the caller's call, since it follows from the field a term sat in - `TagAliasSeeder` — Records what the FileMaker archive, the older registration/scholarship forms, Story Share and the portal's earlier sector list each called a tag the portal now names its own way (**ADR-0007**) as `TagAlias` rows, so the translation is data an admin can extend and a record of what a tag used to be called. Terms AWBW decided are recorded as-is; semantic reads of the archive's hand-typed free text are marked **`proposed`** for a pass with staff (they still resolve, so no import waits on it), and text naming an organization or a facilitator's own circumstances is left alone. Idempotent on (term, tag) and never touches a term added by hand; a term whose tag isn't published is reported, not created. Shared with `data:seed_tag_aliases` +- `FilemakerValues` — How every FileMaker archive import reads a raw field: `parse_date` (the archive's mixed formats, two-digit years given a century), `flag?`, `contact_type` (archive type words → the slice a model accepts), `email_addresses` (the valid addresses inside a free-text field), and `clip` to a varchar. Module functions, also included by the importers +- `FmAddressAttributes` — An `FmAddress` row's fields → `Address` attributes (street with house number, ZIP or its lookup, county from the archive's ZIP table read once, locality via `AddressLocality`), nil when it lacks street/city/state. Shared by the Rolodex, organizations and affiliations imports +- `FilemakerCodeIndex` — Which records of a `FilemakerCoded` model carry each code, read once so a bulk import looks codes up in memory; `#add` records codes the import writes as it goes - `StaffTranslation` — Turns a FileMaker `EnteredBy`/`ModifiedBy` value into a user id through the recorded `StaffAlias` rows (`#user_id_for`), case- and spacing-insensitive. Built once and reused (reads every alias up front); a name nothing answers to lands in `#unmatched` unless it's recorded as ignored - `StaffAliasLoader` — Records `StaffAlias` rows from a CSV/TSV staff list (`staff`, `user_ids_by_email`, `portal_user_email`, optional `ignored` columns), resolving the user by id then email. Idempotent: a name already recorded is left alone; a name with no user yet is reported in `#unresolved`, not recorded. Shared with `data:load_staff_aliases` - `ProfileCategoriesToSectors` — Gives people an additional sector for each profile-specific category tag (the retired workshop settings), reading each category name through `TagTranslation` so the tag aliases decide what it means. Idempotent, keeps the category tags and any existing sector's primary flag; organization tags are only counted (their sectors roll up from facilitators). Drives `data:convert_profile_categories_to_sectors` (DRY_RUN) @@ -314,8 +317,8 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `OrganizationServices::NameMatcher` — Resolves a typed organization name tolerating spreadsheet spelling drift: exact (case-insensitive), then `DuplicateFinder.normalized_name` (punctuation), then `DuplicateFinder.suffixless_name` (Inc/LLC/…); a loose match counts only when it fits one org. Indexes names once per instance. Used by the registration CSV import - `OrganizationServices::CaptureFromSubmission` — The shared org-capture for **both** the event registration (`PublicRegistration`) and standalone (`PublicFormSubmission`) flows: resolves the submitted organization by name (`FormField::ORGANIZATION_NAME_FIELD_IDENTIFIER`, case-insensitive per the column collation; never creates one — an unmatched name is left for an admin), syncs its website/type (`SyncProfile`, latest-wins) and work address (`UpsertAddress`, which adds another work address rather than rewriting a different one). `AffiliationServices::CreateFromRegistration` then mints the person's job affiliation from the typed position, plus the standing Facilitator one when `facilitator_training:` — which only an event supplies (`@event.facilitator_training`), so a standalone form gets the job affiliation alone, the same as registering for a non-training event. **Agreement-role submissions never come through here** — they run `LinkSubmittedOrganization`, which confers the scenario's Facilitator affiliation. Returns the org + the autofill changes, which the event flow records on its registration↔org link. `answers:` (identifier → value) stands in for form + params and `organization:` for the name lookup when the caller resolved them (the CSV import); `overwrite: false` only fills the org's blanks - `PersonServices::CaptureFromSubmission` — The shared person-capture for **both** the event registration (`PublicRegistration`) and standalone (`PublicFormSubmission`) flows, run for a newly-created **and** an existing person, so the person smart fields on the Smart form settings page behave identically across both forms. Fills identity extras (pronouns, name `pronunciation`, legal first name, primary email type, secondary email + type), the profile (`racial_ethnic_identity` + `SyncSharingPreferences`), the mailing address, the phone contact method, and sector/age tags (on the person only — organizations roll theirs up from their active facilitators). A non-blank answer overwrites what's on file (newest submission wins); a blank never clobbers. Sector and age tags follow one rule: submitting both a primary and additional set **overwrites** the person's tags with the latest (`replace: true`), submitting only a primary **reassigns** the primary and leaves the rest (additive). Self-gates on each field identifier, so a form that doesn't ask a question no-ops for it. `answers:` (identifier → value) stands in for form + params (the CSV import); `overwrite: false` keeps the person's name, primary address/phone, and primary tags, only filling blanks and adding new entries beside them -- `PersonServices::ImportFromRolodex` — Creates people from FileMaker Rolodex archive rows (`FmRolodex`), carrying the row's ID as `filemaker_code` plus every field `FmRolodex::PORTAL_FIELDS` maps onto a person (profile fields, email type — a precedence word like "Preferred" records none, since the email's own slot already says that — facilitator start → `member_since`, service areas, `Settings`, `LifeExperiences` and `PrimaryAgeGroup` → sectors and age groups translated through `TagTranslation` (ADR-0007; primary fields → primary, the rest additional; one primary sector and one primary age group per person — each kind takes it from the field that owns it, and an answer naming several lets the other field answer instead) via `tag_sectors` / `tag_age_groups`, and the archive's own entry/modification dates as `created_at`/`updated_at`, plus `EnteredBy`/`ModifiedBy` as `created_by`/`updated_by` through `StaffTranslation`, falling back to `Setting.import_user` (picked on `/admin/settings` → Imports) when the name is blank, a shared login, or unresolved, so every person carries both stamps; a linked person keeps any stamps they already have; a name no `StaffAlias` answers to is reported in `#unmatched_staff`). Plus every address and phone the row answers to (`FmRolodex#addresses` / `#phone_numbers`, primary first; both tables are read once via `FmRolodex.contact_index` + `#preload_contacts`), except work addresses (the worksite one or any typed Work), which the affiliations import will bring in. Links an existing person through `PersonMatcher`, trying each address the row holds in `EMail` (`EmailPrimary` is a flag, never an address: `1` → the first address becomes `email`, a second one the person doesn't already hold `email_2`; `2` → the address and `EmailType` land in `email_2`/`email_2_type`; for a row with no address at all, a same-name person with no email) by adding the row's ID to their `filemaker_code` list — every match when several answer, and alongside any code they already carry — and fills them from the row: blank fields take the archive's value, missing addresses/phones/tags are added (an existing primary stays primary), and a field the archive disagrees on keeps the portal value and is noted as a `Comment` on the person and in `#differences`; creates a new person when nobody matches, even when the email belongs to a differently named person; skips agencies, soft-deleted, and nameless rows. A match that doesn't validate is reported under `failed` and left out, so the row's other matches still link; `relink: true` reads named rows past the idempotence guard to pick one up once the person is valid. Exposes `TagTranslation#unmatched` as `#unmatched_tags`. Idempotent on `filemaker_code`. Driven by `fm:import_people` -- `PersonServices::UndoRolodexImport` — Reverses one `fm:import_people` run from the Ahoy lifecycle events `ImportFromRolodex` tagged with its `import_run`: newest first, deletes each record the run created and reverts each field it filled, unless the record was edited since (its `updated_at` is newer than the event) or the field has changed — those are kept and reported. Runs under `no_touching` so deleting a person's address doesn't make the person look edited. Driven by `fm:undo_import_people` +- `PersonServices::ImportFromRolodex` — Creates people from FileMaker Rolodex archive rows (`FmRolodex`), carrying the row's ID as `filemaker_code` plus every field `FmRolodex::PORTAL_FIELDS` maps onto a person (profile fields, email type — a precedence word like "Preferred" records none, since the email's own slot already says that — facilitator start → `member_since`, service areas, `Settings`, `LifeExperiences` and `PrimaryAgeGroup` → sectors and age groups translated through `TagTranslation` (ADR-0007; primary fields → primary, the rest additional; one primary sector and one primary age group per person — each kind takes it from the field that owns it, and an answer naming several lets the other field answer instead) via `tag_sectors` / `tag_age_groups`, and the archive's own entry/modification dates as `created_at`/`updated_at`, plus `EnteredBy`/`ModifiedBy` as `created_by`/`updated_by` through `StaffTranslation`, falling back to `Setting.import_user` (picked on `/admin/settings` → Imports) when the name is blank, a shared login, or unresolved, so every person carries both stamps; a linked person keeps any stamps they already have; a name no `StaffAlias` answers to is reported in `#unmatched_staff`). Plus every address and phone the row answers to (`FmRolodex#addresses` / `#phone_numbers`, primary first; both tables are read once via `FmRolodex.contact_index` + `#preload_contacts`), except organization addresses (`FmRolodex#organization_address?`: the worksite one, any typed Work, and every address on an agency's card), which `AffiliationServices::ImportFromFilemaker` and `OrganizationServices::ImportFromFilemaker` place. Links an existing person through `PersonMatcher`, trying each address the row holds in `EMail` (`EmailPrimary` is a flag, never an address: `1` → the first address becomes `email`, a second one the person doesn't already hold `email_2`; `2` → the address and `EmailType` land in `email_2`/`email_2_type`; for a row with no address at all, a same-name person with no email) by adding the row's ID to their `filemaker_code` list — every match when several answer, and alongside any code they already carry — and fills them from the row: blank fields take the archive's value, missing addresses/phones/tags are added (an existing primary stays primary), and a field the archive disagrees on keeps the portal value and is noted as a `Comment` on the person and in `#differences`; creates a new person when nobody matches, even when the email belongs to a differently named person; an agency card naming a contact person becomes that person; skips soft-deleted and nameless rows (a nameless agency card is the agency itself, which the organizations pass reads). Reads raw values through `FilemakerValues` and addresses through `FmAddressAttributes`. A match that doesn't validate is reported under `failed` and left out, so the row's other matches still link; `relink: true` reads named rows past the idempotence guard to pick one up once the person is valid. Exposes `TagTranslation#unmatched` as `#unmatched_tags`. Idempotent on `filemaker_code`. Driven by `fm:import_people` +- `PersonServices::UndoRolodexImport` — Reverses one FileMaker import run (`fm:import_people` or `fm:import_organizations`) from the Ahoy lifecycle events it tagged with its `import_run`: newest first, deletes each record the run created and reverts each field it filled, unless the record was edited since (its `updated_at` is newer than the event) or the field has changed — those are kept and reported. Runs under `no_touching` so deleting a person's address doesn't make the person look edited. Driven by `fm:undo_import_people` (alias `fm:undo_import`) - `PersonServices::SyncSharingPreferences` — Writes the two content-sharing answers (`anonymous_contributions`, `display_name_preference`) onto the `Person`, mapping each submitted option label back to its stored value; an unanswered or unrecognized answer leaves the profile alone, so a form that doesn't ask the questions is a no-op and every flow can call it unconditionally. The person-side counterpart to `OrganizationServices::SyncProfile`. Shared by `PersonServices::CaptureFromSubmission` (so a submission's answers land in the "what this submission changed" audit via the Ahoy trail) and `CalloutFormSubmission` (whose `#profile_changes` re-reports the same edits for its own Ahoy tracking) - `EventRegistrationServices::TransferContinuingEducation` — Splits/relocates a registrant's CE when they transfer events (issue #1944): a simple forward transfer leaves a paid, zero-hours **stub** on the source (its payments count at the original event) and creates a **live** record on the destination carrying the hours and the outstanding balance; when the reg being transferred out is itself a transfer-in (a collapsing double transfer, or a transfer back to the origin) its live record is relocated forward — merging back into the origin's stub — instead of split again, so no third record appears. Runs inside the transfer transaction, after the destination is saved and before a collapsing middle reg is destroyed - `EventRegistrationServices::ReconcileMergedContinuingEducation` — Run from the event-registration deduper's `after_merge`. A CE registration moved from the deleted registration still points at that (deleted) registrant's license — invalid, since the registration now belongs to the kept registrant — so this repoints each to the kept registrant's equivalent license (same kind + number, found or created), then merges the resulting duplicates into one with `ContinuingEducationDeduper` so payments are preserved @@ -329,6 +332,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Facilitator affiliation on the project's org (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` outside `ACTIVE_STATUSES` ends it undated), plus a job affiliation for a more specific role ("Lead Facilitator", "Supervisor"); people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) @@ -353,6 +357,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Organizations +- `OrganizationServices::ImportFromFilemaker` — The FileMaker organizations pass (ADR-0008): each `FmOrganization` and its `FmProject`s become portal orgs — an org with exactly one project stays one org carrying both codes, otherwise the projects nest under it (`parent`). Matches by code (`FilemakerCodeIndex`), then exact name only to an org carrying no code from that table (a namesake gets `"Name (code)"`). Fills blanks only (website, description, mission, `organization_type` from `OrgType`/`AgencyType` loosely — unmatched words tallied in `#unmatched_types`), noting differences as comments; adds the org's `fm_addresses`, its phone onto the primary address, its `fm_email_addresses`; copies each project's `Locality`/`District`/`SPA`/`LASupervisorialDistrict` onto the program's address in its city (a city-only one when none); folds a merged-away project's code (`MergerWithID`) into the survivor; and reads agencies' own Rolodex cards (website, mission, addresses, and email when no contact person is named) onto their org, creating it by name when nothing answers. Driven by `fm:import_organizations` - `OrganizationServices::AutofillChange` — One thing a registrant's answers wrote onto an org: `field`, human `label`, the `value` that landed in it, `change_type` (`"new"` filled a blank / `"update"` replaced something), `previous_value` on an update, and an optional `scope` naming which work address. `change_type` is derived from `previous_value` so the two can't disagree, but is written to the JSON anyway so the stored row reads on its own. Round-trips through the `form_autofill_changes` column; `#description` renders "Website" or "ZIP on the Austin work address" for a flash, while the linking page shows the value and what it displaced. `#key` is (field, scope), which is what makes a later submission's value replace an earlier one instead of stacking. Drops a stored entry with no `field` rather than raising — it's a display aid, not a ledger. **Not a substitute for the Ahoy audit trail**, which records every before/after on Organization and Address independently; this is the denormalized, registration-scoped copy the linking page can render without correlating events - `OrganizationServices::AddressMatcher` — Given a submitted city/state/street/ZIP, finds the org address it corresponds to so `UpsertAddress`/`ProfileDiff` update the right one instead of duplicating. Tries same city + state (preferring one whose street also matches, so an org with several offices in one city doesn't get the wrong one rewritten), then same city + street (unifies an address with a submission that merely adds the state/country we were missing), then — across cities — same street + same ZIP, which reunites an address whose city was renamed ("Ventura" / "San Buenaventura") without merging two offices that share a street name in different towns. Every comparison goes through `AddressNormalizer`, so known spellings ("St. Louis" / "Saint Louis", "TX" / "Texas", "N Main St" / "North Main Street") match at every step. Module function, used by both address services - `OrganizationServices::UpsertAddress` — Find-or-create an organization's "work" address from a registrant's submitted organization fields (street/city/state/zip/country), matching via `AddressMatcher`; updates the match, else adds a new address; never demotes the org's existing primary (a registrant's address becomes primary only when the org has none yet). `overwrite: true` (default, public flow) replaces the matching address's details; `overwrite: false` (admin org-linking) fills only its blank fields so a conflicting saved address is kept and flagged instead. City and state are always fill-only (never flipped, in either mode), so a cross-city ZIP match keeps the saved city and lets `ProfileDiff` flag the difference; that fill still earns its keep because a legacy row can hold `""` in either column, and the street + ZIP match is the one path that reaches such a row to repair it. Returns a `Result` (`address`, `created`, and `changes` — an `AutofillChange` per field actually written, carrying the value) so callers can report what was really saved. Every field is trimmed before it's stored, so a padded answer isn't later read back as a difference from itself. Tolerates optional answers: a skipped street/ZIP is stored blank (both are `NOT NULL` columns), and `address` is nil — nothing saved — when there's no city, or no state and no existing address to update, rather than failing the whole registration. Shared by `PublicRegistration` and the admin org-linking actions @@ -616,6 +621,7 @@ Located in `lib/tasks/` (18 files): - `backfill_notification_stamps.rake` — Backfill notifications' `created_by_id`/`updated_by_id` from `sender_id`, nil-only (`data:backfill_notification_stamps`) - `fm_import.rake` — Load the FileMaker CSV exports into the `fm_*` archive tables (`fm:import`, reads each CSV from the `FM Archive` resources, batch `insert_all`, skips rows whose key is already stored) - `fm_import_people.rake` — Create people from the FileMaker Rolodex archive (`fm:import_people`, via `PersonServices::ImportFromRolodex`, links existing by name + email, idempotent, DRY_RUN; VERBOSE=true lists every row, FM_IDS=a,b re-runs just those rows past the idempotence guard; commits every 1,000 rows, prints progress, saves the full report as an ActiveStorage blob even on a crash, and aborts when no `TagAlias` rows exist or no `Setting.import_user` is chosen; logs every record it creates or fills as activity under a run name (`Current.source = "fm_import_people"`), which `fm:undo_import_people RUN=…` reverses via `PersonServices::UndoRolodexImport`) +- `fm_import_organizations.rake` — Create organizations and affiliations from the FileMaker archive after `fm:import_people` (`fm:import_organizations`: `OrganizationServices::ImportFromFilemaker` then `AffiliationServices::ImportFromFilemaker`; idempotent, DRY_RUN rolls back as a whole while a real run commits per batch, VERBOSE=true lists every row; prints progress, the skipped/failed rows, differences, unmatched org types and the Personnel status/role tallies, and saves the report as an ActiveStorage blob; aborts with no `Setting.import_user`; logs under a run name that `fm:undo_import RUN=…` reverses) - `resolve_resource_organizations.rake` — Link resources to organizations from the legacy `agency` free-text name (`data:resolve_resource_organizations`, match-or-create, idempotent, DRY_RUN) - `seed_tag_aliases.rake` — Record the legacy sector and age-group terms from ADR-0007 as tag aliases (`data:seed_tag_aliases`, via `TagAliasSeeder`, idempotent; also part of `db:seed`) - `load_staff_aliases.rake` — Record FileMaker staff names → portal users as staff aliases from `FILE=` (`data:load_staff_aliases`, via `StaffAliasLoader`, idempotent) diff --git a/app/models/affiliation.rb b/app/models/affiliation.rb index 0a9a4a1168..fdbe26ff3a 100644 --- a/app/models/affiliation.rb +++ b/app/models/affiliation.rb @@ -1,7 +1,7 @@ class Affiliation < ApplicationRecord include Communicable include FilemakerCoded - filemaker_archived_in FmProject, FmPersonnel + filemaker_archived_in FmProject, FmPersonnel, FmPeopleToOrganization # Standing title given to the "facilitator affiliation" we create on registration # and org linking. Matches the `facilitators` scope / `facilitator?` predicate diff --git a/app/models/fm_rolodex.rb b/app/models/fm_rolodex.rb index 2abdd07784..2d376a334d 100644 --- a/app/models/fm_rolodex.rb +++ b/app/models/fm_rolodex.rb @@ -225,6 +225,21 @@ def addresses @addresses ||= contact_records(FmAddress, primary_ids: address_ids) end + # The row is an agency's own contact card, sometimes naming the person to reach there. + def agency? + FilemakerValues.flag?(data["isAgency__c"]) + end + + def work_address?(address) + address.fm_id == data["WorksiteAddrsID"] || + FilemakerValues.contact_type(address.data["AddrsType"], allowed: Address::CONTACT_TYPES) == "work" + end + + # Where the person works rather than lives — every address on an agency's card is the agency's. + def organization_address?(address) + agency? || work_address?(address) + end + def job_affiliations @job_affiliations ||= FmPeopleToOrganization.find_by_data("PersonID", fm_id).order(:fm_id).to_a end diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb new file mode 100644 index 0000000000..7e5020af06 --- /dev/null +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -0,0 +1,319 @@ +module AffiliationServices + # Turns the FileMaker archive's ties between people and organizations into affiliations + # (ADR-0008), after the Rolodex and organizations passes have made both ends. Reads, in + # order: + # + # 1. Personnel (FmPersonnel) — a person staffing a project's art program. A facilitating + # role becomes a Facilitator affiliation, dated from when the row was entered to when + # the project closed; any more specific role ("Lead Facilitator", "Supervisor") also + # becomes a job affiliation with that title (ADR-0003 D1). Each carries the row's + # PrsnlRecID as its filemaker_code. + # 2. People-to-organizations (FmPeopleToOrganization) — a job at an organization, with + # its title, carrying the row's RecordID. + # 3. The Rolodex card itself (FmRolodex) — the organization its OrgID (or its typed + # organization name) names, with its Title, and its work addresses: each lands on + # the organization it belongs to and the person's affiliations there point at it. + # A work address no organization can be found for stays with the person. + # + # People are found by their Rolodex code, organizations by the project or organization + # code the earlier pass gave them. A row whose person or organization wasn't imported is + # skipped and reported. An existing affiliation of the same person, organization and + # title is linked rather than duplicated: it takes the row's code and fills its blank + # dates, keeping its own where FileMaker disagrees (noted in #differences and as a + # comment on the affiliation). + # + # None of these is minted from a training, so event_registration_id stays NULL and the + # training reconciliation leaves them alone unless an admin includes hand-made rows. + class ImportFromFilemaker + include FilemakerValues + + OUTCOMES = %i[created linked already_imported skipped failed].freeze + BATCH_SIZE = 1_000 + # Personnel Status values that mean the person still staffs the program. Anything else + # reads as ended — over-reporting active programs would skew program status. + ACTIVE_STATUSES = %w[active current].freeze + FACILITATING_ROLE = /facilitat/i + SOURCES = { FmPersonnel => "Personnel", FmPeopleToOrganization => "People-to-organizations", FmRolodex => "Rolodex" }.freeze + + attr_reader :results, :differences, :statuses, :roles + + def initialize(progress: nil, run: nil) + @progress = progress + @run = run + @results = OUTCOMES.index_with { [] } + @differences = [] + @statuses = Hash.new(0) + @roles = Hash.new(0) + @staff = StaffTranslation.new + @import_user_id = Setting.import_user&.id + @address_attributes = FmAddressAttributes.new + @name_matcher = OrganizationServices::NameMatcher.new + end + + def call + @tracker = Ahoy::Tracker.new + @people = FilemakerCodeIndex.new(Person) + @organizations = FilemakerCodeIndex.new(Organization) + @affiliations = FilemakerCodeIndex.new(Affiliation) + @projects = FmProject.all.index_by(&:fm_id) + @personnel_ids = FmPersonnel.pluck(:fm_id).to_set + @job_ids = FmPeopleToOrganization.pluck(:fm_id).to_set + @read = 0 + + each_batch(FmPersonnel.all) { |row| import_personnel(row) } + each_batch(FmPeopleToOrganization.all) { |row| import_job(row) } + addresses = FmRolodex.contact_index(FmAddress) + phones = FmRolodex.contact_index(FmPhoneNumber) + each_batch(FmRolodex.all) { |rolodex| import_rolodex(rolodex.preload_contacts(addresses:, phone_numbers: phones)) } + self + end + + private + + # Each batch commits on its own, so a run that stops partway keeps what it finished. + # Its activity events, tagged with the run, commit with it — UndoRolodexImport reads them. + # Untouched, so a new affiliation doesn't stamp its person as edited now. + def each_batch(scope, &import) + scope.find_in_batches(batch_size: BATCH_SIZE) do |batch| + ActiveRecord::Base.transaction(requires_new: true) do + ActiveRecord::Base.no_touching do + Analytics::LifecycleBuffer.tagging(@run ? { import_run: @run } : {}) { batch.each(&import) } + end + Analytics::LifecycleBuffer.flush_to(@tracker) + end + @progress&.call(@read += batch.size) + end + end + + def import_personnel(row) + data = row.data + label = "#{row.fm_id} #{data["Name"].to_s.squish}".squish + return record(:already_imported, label) if @affiliations.include?(row.fm_id) + + people = @people.records(data["PersonID"]) + organization = organization_for(data["ProjectID"]) + return record(:skipped, "#{label}: person #{data["PersonID"].presence || "(none)"} wasn't imported") if people.empty? + return record(:skipped, "#{label}: project #{data["ProjectID"].presence || "(none)"} wasn't imported") unless organization + + dates = personnel_dates(data, @projects[data["ProjectID"].to_s]) + titles(data["Role"]).each do |title| + attributes = title == Affiliation::FACILITATOR_TITLE ? dates : dates.except(:start_date) + people.each { |person| affiliate(person, organization, row, label, title:, attributes:, codes: @personnel_ids) } + end + end + + # Personnel are the people staffing the art program, so a row with no role is a + # facilitator. A role saying more than that is also a job title (ADR-0003 D1). + def titles(role) + role = clip(role.to_s.squish) + @roles[role.presence || "(blank)"] += 1 + return [ Affiliation::FACILITATOR_TITLE ] if role.blank? || role.casecmp?(Affiliation::FACILITATOR_TITLE) + return [ role ] unless role.match?(FACILITATING_ROLE) + + [ Affiliation::FACILITATOR_TITLE, role ] + end + + # Personnel rows carry no dates of their own: the assignment starts when it was entered + # and ends when its project closed. A status that isn't active ends it now, undated. + def personnel_dates(data, project) + status = data["Status"].to_s.squish + active = ACTIVE_STATUSES.include?(status.downcase) + @statuses["#{status.presence || "(blank)"} → #{active ? "active" : "ended"}"] += 1 + start_date = parse_date(data["DateEntered"]) + end_date = project && (parse_date(project.data["CloseDate"]) || parse_date(project.data["EndDate"])) + end_date = nil if start_date && end_date && end_date < start_date + { start_date:, end_date:, inactive: !active } + end + + def import_job(row) + data = row.data + label = "#{row.fm_id} #{data["PersonID"]} at #{data["OrgID"]}" + return record(:already_imported, label) if @affiliations.include?(row.fm_id) + + people = @people.records(data["PersonID"]) + organization = organization_for(data["OrgID"]) + return record(:skipped, "#{label}: person wasn't imported") if people.empty? + return record(:skipped, "#{label}: organization wasn't imported") unless organization + + people.each { |person| affiliate(person, organization, row, label, title: clip(data["Title"].to_s.squish), attributes: {}, codes: @job_ids) } + end + + # The Rolodex card names the person's organization and their title there, and lists the + # work addresses the people pass left for this one. + def import_rolodex(rolodex) + data = rolodex.data + work = rolodex.addresses.select { |address| rolodex.organization_address?(address) } + return if flag?(data["Delete"]) || (data["OrgID"].blank? && data["Organization"].blank? && work.empty?) + + label = "#{rolodex.fm_id} #{rolodex.name}" + people = @people.records(rolodex.fm_id) + return record(:skipped, "#{label}: person wasn't imported") if people.empty? + + organization = organization_for(data["OrgID"]) || named_organization(data["Organization"]) + if organization.nil? && (data["OrgID"].present? || data["Organization"].present?) + record(:skipped, "#{label}: organization #{data["OrgID"].presence || data["Organization"].squish} wasn't found") + end + title = clip(data["Title"].to_s.squish) + people.each do |person| + next unless organization + next if title.blank? && person.affiliations.exists?(organization:) + + affiliate(person, organization, rolodex, label, title:, attributes: {}, codes: nil) + end + place_work_addresses(rolodex, people, organization, work, label) + end + + def named_organization(name) + name = name.to_s.squish + @name_matcher.call(name)&.organization if name.present? + end + + # The affiliation this row describes: one the person already has at the organization + # with the same title is linked, otherwise one is created. `codes` is the set of IDs + # the row's own table holds — an affiliation already carrying another of them records + # a different row (another stint on the same project), so it's never the match. A row + # with no code of its own (the Rolodex card) only fills in what's missing. + def affiliate(person, organization, row, label, title:, attributes:, codes:) + existing = person.affiliations.where(organization:).to_a + match = existing.find { |affiliation| same_title?(affiliation.title, title) && !records_another_row?(affiliation, codes) } + match ||= existing.find { |affiliation| affiliation.title.blank? } if title.present? && codes.nil? + return link(match, row, label, title:, attributes:, codes:) if match + + create(person, organization, row, label, title:, attributes:, code: codes && row.fm_id) + end + + def records_another_row?(affiliation, codes) + codes && affiliation.filemaker_codes.any? { |code| codes.include?(code) } + end + + def same_title?(held, title) + held.to_s.squish.casecmp?(title.to_s.squish) + end + + def link(affiliation, row, label, title:, attributes:, codes:) + return record(:already_imported, "#{label} → affiliation ##{affiliation.id}") if codes.nil? && affiliation.title.present? + + affiliation.filemaker_code = Affiliation.join_filemaker_codes(affiliation.filemaker_code, row.fm_id) if codes + affiliation.title = title if affiliation.title.blank? + kept = fill_dates(affiliation, attributes) + affiliation.updated_by_id ||= updated_by(row.data) + return failed(affiliation, label) unless affiliation.save + + @affiliations.add(row.fm_id, affiliation.id) if codes + note_differences(affiliation, row, label, kept) if kept.any? + record(:linked, "#{label} → affiliation ##{affiliation.id}") + end + + # A blank date takes the archive's. The archive's status only ends an affiliation the + # portal still holds open with no dates of its own to say otherwise. + def fill_dates(affiliation, attributes) + kept = attributes.slice(:start_date, :end_date).filter_map do |attribute, value| + next if value.nil? || affiliation[attribute] == value + + if affiliation[attribute].nil? + affiliation[attribute] = value + next + end + + [ Affiliation.human_attribute_name(attribute), affiliation[attribute], value ] + end + if attributes[:inactive] && !affiliation.inactive? && affiliation.end_date.nil? + affiliation.inactive = true + affiliation.inactive_supplied = true + end + kept + end + + def create(person, organization, row, label, title:, attributes:, code:) + data = row.data + entered = parse_date(data["DateEntered"]) + affiliation = person.affiliations.build( + organization:, title:, filemaker_code: code, **attributes, + inactive_supplied: attributes[:inactive], + created_at: entered, updated_at: parse_date(data["DateTimeModified"].presence || data["ModificationTimeStamp"]) || entered, + created_by_id: @staff.user_id_for(data["EnteredBy"]) || @import_user_id, updated_by_id: updated_by(data) + ) + return failed(affiliation, label) unless affiliation.save + + @affiliations.add(code, affiliation.id) if code + record(:created, "#{label} → affiliation ##{affiliation.id}#{" (#{title})" if title}") + end + + def updated_by(data) + @staff.user_id_for(data["ModifiedBy"]) || @import_user_id + end + + # Each work address lands on the organization it belongs to — the one FileMaker filed it + # under, else the card's organization, else the person's only one — and the person's + # affiliations there that name no address yet point at it. The worksite FileMaker named + # goes first, so it's the one they point at. With nowhere to put it, it stays with the + # person as their own work address. + def place_work_addresses(rolodex, people, organization, work, label) + work.sort_by { |address| address.fm_id == rolodex.data["WorksiteAddrsID"] ? 0 : 1 }.each do |fm_address| + attributes = @address_attributes.call(fm_address.data, address_type: "work") + next unless attributes + + people.each do |person| + target = organization_for(fm_address.data["OrgID"]) || organization || sole_organization(person) + next keep_with_person(person, attributes, label) unless target + + address = organization_address(target, attributes) + next failed(address, label) unless address.persisted? + + person.affiliations.where(organization: target, organization_address_id: nil).find_each do |affiliation| + affiliation.update(organization_address: address) || failed(affiliation, label) + end + end + end + end + + def sole_organization(person) + ids = person.affiliations.distinct.pluck(:organization_id) + Organization.find_by(id: ids.first) if ids.one? + end + + def organization_address(organization, attributes) + held = organization.addresses.find { |address| address_key(address.street_address, address.city) == address_key(attributes[:street_address], attributes[:city]) } + return held if held + + organization.addresses.create(attributes.merge(primary: organization.addresses.where(primary: true).none?)) + end + + def keep_with_person(person, attributes, label) + return if person.addresses.any? { |address| address_key(address.street_address, address.city) == address_key(attributes[:street_address], attributes[:city]) } + + address = person.addresses.create(attributes.merge(primary: person.addresses.where(primary: true).none?)) + return failed(address, label) unless address.persisted? + + record(:created, "#{label} → work address ##{address.id} kept on person ##{person.id}, no organization to place it on") + end + + def address_key(street, city) + [ AddressNormalizer.street(street), AddressNormalizer.city(city) ] + end + + def organization_for(code) + code = code.to_s.strip + return if code.blank? + + ids = @organizations.ids(code) + Organization.find_by(id: ids.first) if ids.any? + end + + def note_differences(affiliation, row, label, kept) + lines = kept.map { |field, portal, archive| "#{field}: kept “#{portal}”, FileMaker has “#{archive}”" } + Comment.create!(commentable: affiliation, + body: "FileMaker #{SOURCES[row.class]} #{row.fm_id} differs from this affiliation:\n#{lines.join("\n")}") + @differences << "#{label} → affiliation ##{affiliation.id}: #{lines.join('; ')}" + end + + def failed(record, label) + record(:failed, "#{label}: #{record.class.name} #{record.errors.full_messages.to_sentence.presence || "matches one already on file"}") + false + end + + def record(outcome, message) + @results[outcome] << message + end + end +end diff --git a/app/services/filemaker_code_index.rb b/app/services/filemaker_code_index.rb new file mode 100644 index 0000000000..23153999f7 --- /dev/null +++ b/app/services/filemaker_code_index.rb @@ -0,0 +1,30 @@ +# Which records of a FilemakerCoded model carry each code, read once so a bulk import +# looks codes up in memory rather than scanning the table per archive row. Records the +# import codes as it goes are added with #add, so a later row finds them. +class FilemakerCodeIndex + def initialize(model) + @model = model + @ids = Hash.new { |hash, code| hash[code] = [] } + model.where.not(filemaker_code: [ nil, "" ]).pluck(:id, :filemaker_code).each do |id, value| + model.split_filemaker_codes(value).each { |code| @ids[code] << id } + end + end + + def ids(code) + @ids.fetch(@model.strip_filemaker_code(code), []) + end + + def include?(code) + ids(code).any? + end + + def records(code) + ids = ids(code) + ids.empty? ? [] : @model.where(id: ids).order(:id).to_a + end + + def add(code, id) + code = @model.strip_filemaker_code(code) + @ids[code] |= [ id ] if code.present? + end +end diff --git a/app/services/filemaker_values.rb b/app/services/filemaker_values.rb new file mode 100644 index 0000000000..e3503d5a16 --- /dev/null +++ b/app/services/filemaker_values.rb @@ -0,0 +1,75 @@ +# How every FileMaker archive import reads a raw field value: dates in the archive's mixed +# formats, yes/no flags, contact types, and free text bounded to the column it lands in. +# Included by the importers so each reads the archive the same way. +module FilemakerValues + module_function + + STRING_LIMIT = 255 + EMAIL_PATTERN = /[^\s,;<>"']+@[^\s,;<>"']+\.[^\s,;<>"']+/ + FALSE_FLAGS = %w[0 n no false].freeze + DATE_FORMATS = %w[%m/%d/%Y %Y-%m-%d %m-%d-%Y %Y/%m/%d].freeze + # A two-digit year below this is read as 20xx, at or above it as 19xx. + TWO_DIGIT_YEAR_PIVOT = 10 + CONTACT_TYPES = { "home" => "personal", "personal" => "personal", "cell" => "personal", + "mobile" => "personal", "work" => "work", "business" => "work", + "professional" => "work", "mailing" => "mailing" }.freeze + + # Every string copied here lands in a varchar(255) the archive's free text isn't + # bounded by, and an over-long incidental field would otherwise cost the record — + # or, for a column with no length validation behind it, raise mid-import. + def clip(value) + value.presence && value.to_s.truncate(STRING_LIMIT) + end + + # The archive words an email field freely ("Attn: someone", a note, two addresses at + # once), so it's read for the addresses in it the portal's email validation accepts. + def email_addresses(value) + value.to_s.scan(EMAIL_PATTERN).select { |email| valid_email?(email) } + end + + def valid_email?(email) + email.length <= STRING_LIMIT && email.match?(URI::MailTo::EMAIL_REGEXP) + end + + def flag?(value) + value.to_s.strip.presence && !FALSE_FLAGS.include?(value.to_s.strip.downcase) + end + + # The archive's types map onto one set of ours, but each model declares the slice it + # accepts — an address can be the mailing one, a phone number can't. A type we can't + # place is left blank, which both models then record as unknown. + def contact_type(value, allowed:) + CONTACT_TYPES[value.to_s.strip.downcase].presence_in(allowed) + end + + def parse_date(value) + value = value.to_s.strip + return if value.blank? + + DATE_FORMATS.each do |format| + parsed = parse_with(value, format) + return parsed if parsed + end + nil + end + + # Parsing stops where the format ends, so "10-5-1983" read as %Y-%m-%d takes 19 + # as the day and drops the 83 — digits left over mean the format split the value + # in the wrong place, so the next one gets its turn. + def parse_with(value, format) + parts = Date._strptime(value, format) + return if parts.nil? || parts[:leftover].to_s.match?(/\A\d/) + + expand_century(Date.new(parts[:year], parts[:mon], parts[:mday])) + rescue Date::Error, TypeError + nil + end + + # %Y reads a two-digit year as the year itself, so "10/5/83" is 83 AD. The archive + # mixes two- and four-digit years, so a bare pair gets a century: 83 is 1983, 05 is 2005. + def expand_century(date) + return date if date.year >= 100 + + date.change(year: date.year + (date.year < TWO_DIGIT_YEAR_PIVOT ? 2000 : 1900)) + end +end diff --git a/app/services/organization_services/import_from_filemaker.rb b/app/services/organization_services/import_from_filemaker.rb new file mode 100644 index 0000000000..aa083113c1 --- /dev/null +++ b/app/services/organization_services/import_from_filemaker.rb @@ -0,0 +1,422 @@ +module OrganizationServices + # Turns the FileMaker archive's organizations (FmOrganization) and the art programs run at + # them (FmProject) into portal organizations (ADR-0008). Runs after the Rolodex pass and + # before AffiliationServices::ImportFromFilemaker, which finds what this creates by code. + # + # An app organization is usually a FileMaker project; the FileMaker organization above it + # becomes its parent (ADR-0004). An organization with exactly one project is one place, + # not two, so it stays one portal organization carrying both codes. Each archive row is + # matched to an existing organization by any shared filemaker_code first, then by exact + # name — but only to one carrying no code from that row's table yet, since a name taken + # by another FileMaker record is a namesake, not a match. A match takes the row's code. + # + # Every row lands in one outcome bucket of #results: + # * created — a new organization + # * linked — an existing organization took the row's code and was filled + # * already_imported — an organization already carried the row's code + # * skipped — nameless rows, and merged projects with nowhere to fold into + # * failed — the Organization didn't validate + # + # Fields fill blanks only. Where the portal already holds a different value it keeps its + # own, noted in #differences and as a comment on the organization. Also carried: + # * the organization's addresses (fm_addresses by OrgID), its phone onto its main + # address, and its email (fm_email_addresses) + # * each project's City/State with its Locality, District, SPA and supervisorial + # district, onto the program's address in that city + # * a merged-away project's code, onto the program it merged into + # * an agency's own Rolodex card: its website, mission and addresses (and its email, + # when the card names no contact person whose email it would be) + class ImportFromFilemaker + include FilemakerValues + + OUTCOMES = %i[created linked already_imported skipped failed].freeze + BATCH_SIZE = 500 + # FileMaker's OrgType / AgencyType words, matched loosely onto Organization::ORGANIZATION_TYPES. + ORGANIZATION_TYPE_PATTERNS = { + /non.?profit|501|charit/i => "501c3/nonprofit", + /for.?profit|business|corporat|compan/i => "For-profit", + /\b(gov|government|county|city|state|federal|public|municipal)\b/i => "Government agency" + }.freeze + COMMENT_SOURCES = { FmOrganization => "FileMaker organization", FmProject => "FileMaker project", + FmRolodex => "FileMaker Rolodex agency" }.freeze + + attr_reader :results, :differences, :unmatched_types + + def initialize(progress: nil, run: nil) + @progress = progress + @run = run + @results = OUTCOMES.index_with { [] } + @differences = [] + @unmatched_types = Hash.new(0) + @staff = StaffTranslation.new + @import_user_id = Setting.import_user&.id + @address_attributes = FmAddressAttributes.new + @name_matcher = NameMatcher.new + end + + def call + @tracker = Ahoy::Tracker.new + @codes = FilemakerCodeIndex.new(Organization) + @organization_ids = FmOrganization.pluck(:fm_id).to_set + projects = FmProject.all.to_a + @projects = projects.index_by(&:fm_id) + @project_ids = @projects.keys.to_set + @addresses = FmAddress.all.group_by { |address| address.data["OrgID"].to_s } + @phones = FmPhoneNumber.all.group_by { |phone| phone.data["OrgID"].to_s } + @emails = FmEmailAddress.all.group_by { |email| email.data["OrgID"].to_s } + + merged, live = projects.partition { |project| merged_into(project) } + by_organization = live.group_by { |project| project.data["OrgID"].to_s } + read = 0 + FmOrganization.find_in_batches(batch_size: BATCH_SIZE) do |batch| + import_batch(batch) { |organization| import_organization(organization, by_organization.delete(organization.fm_id) || []) } + @progress&.call(read += batch.size) + end + import_batch(by_organization.values.flatten) { |project| import_projects(nil, [ project ]) } + import_batch(merged) { |project| fold_merged(project) } + import_batch(agency_cards) { |rolodex| import_agency(rolodex) } + self + end + + private + + # Each batch commits on its own, so a run that stops partway keeps what it finished. + # Its activity events, tagged with the run, commit with it — UndoRolodexImport reads them. + # Untouched, so an address added to an organization doesn't stamp it as edited now. + def import_batch(rows, &import) + ActiveRecord::Base.transaction(requires_new: true) do + ActiveRecord::Base.no_touching do + Analytics::LifecycleBuffer.tagging(@run ? { import_run: @run } : {}) { rows.each(&import) } + end + Analytics::LifecycleBuffer.flush_to(@tracker) + end + end + + def import_organization(fm_organization, projects) + data = fm_organization.data + name = clip(data["OrgName"].to_s.squish) + label = "#{fm_organization.fm_id} #{name}".squish + if name.blank? + record(:skipped, "#{label}: no name") + return import_projects(nil, projects) + end + + umbrella = coded(fm_organization.fm_id) || named(name, @organization_ids) + return import_single_program(fm_organization, projects.sole, umbrella, label) if projects.one? + + umbrella ||= Organization.new(name: unique_name(name, fm_organization.fm_id)) + return unless save(umbrella, fm_organization, label, organization_attributes(data)) + + import_projects(umbrella, projects) + end + + # One organization running one program is a single place, so it stays one portal + # organization carrying both codes — unless the portal already keeps them apart. + def import_single_program(fm_organization, project, umbrella, label) + program = coded(project.fm_id) || named(project_name(project), @project_ids) + if umbrella && program && umbrella != program + return unless save(umbrella, fm_organization, label, organization_attributes(fm_organization.data)) + + return import_projects(umbrella, [ project ]) + end + + organization = umbrella || program || Organization.new(name: unique_name(clip(fm_organization.data["OrgName"].to_s.squish), project.fm_id)) + attributes = organization_attributes(fm_organization.data).merge(project_attributes(project.data)) { |_key, mine, theirs| mine.presence || theirs } + save(organization, fm_organization, "#{label} + #{project.fm_id}", attributes, also: project) + end + + def import_projects(umbrella, projects) + projects.each do |project| + name = project_name(project) + label = "#{project.fm_id} #{name}".squish + next record(:skipped, "#{label}: no name") if name.blank? + + program = coded(project.fm_id) || named(name, @project_ids) || Organization.new(name: unique_name(name, project.fm_id)) + nest(program, umbrella, label) if umbrella && program != umbrella + save(program, project, label, project_attributes(project.data)) + end + end + + # A program the portal already nests elsewhere stays where it is. + def nest(program, umbrella, label) + return program.parent = umbrella if program.parent_id.nil? + return if program.parent_id == umbrella.id + + @differences << "#{label} → ##{program.id}: kept parent ##{program.parent_id}, FileMaker nests it under ##{umbrella.id}" + end + + # Writes one archive row (and, for a single-program organization, its project too) onto + # the organization: its code, blank fields, addresses, phone and email. + def save(organization, row, label, attributes, also: nil) + rows = [ row, also ].compact + outcome = outcome_for(organization, rows) + organization.filemaker_code = Organization.join_filemaker_codes(organization.filemaker_code, rows.map(&:fm_id)) + kept = fill(organization, attributes) + stamp(organization, row.data) + add_addresses(organization, @addresses.fetch(row.fm_id, []).map(&:data)) + rows.select { |source| source.is_a?(FmProject) }.each { |project| place_project(organization, project.data) } + add_phone(organization, @phones.fetch(row.fm_id, []).map(&:data)) + fill_email(organization, @emails.fetch(row.fm_id, []).map { |email| email.data["EmailAddress"] }) + return failed(organization, label) unless organization.save + + rows.each { |source| @codes.add(source.fm_id, organization.id) } + note_differences(organization, row, label, kept) if kept.any? + record(outcome, "#{label} → ##{organization.id}") + organization + end + + def outcome_for(organization, rows) + return :created if organization.new_record? + + rows.all? { |row| organization.filemaker_codes.include?(row.fm_id) } ? :already_imported : :linked + end + + # FileMaker kept a merged-away project's row, pointing at the one it merged into, so its + # code joins that program and the records still naming it find their way there. + def fold_merged(project) + label = "#{project.fm_id} #{project_name(project)}".squish + survivor = merged_into(project) + program = coded(survivor.fm_id) + own = coded(project.fm_id) + return record(:skipped, "#{label}: merged into #{survivor.fm_id}, which wasn't imported") unless program + return record(:already_imported, "#{label} → ##{program.id}") if own == program + if own + return record(:skipped, "#{label}: merged into #{survivor.fm_id} (##{program.id}) in FileMaker but still its own organization ##{own.id} — merge them by hand") + end + + program.filemaker_code = Organization.join_filemaker_codes(program.filemaker_code, project.fm_id) + return failed(program, label) unless program.save + + @codes.add(project.fm_id, program.id) + record(:linked, "#{label} → ##{program.id} (merged into #{survivor.fm_id})") + end + + # The project a merged-away one ended up in, following a chain of merges to its end. + def merged_into(project) + seen = Set[project.fm_id] + survivor = nil + while (next_id = (survivor || project).data["MergerWithID"].to_s.strip.presence) && seen.add?(next_id) + break unless @projects[next_id] + + survivor = @projects[next_id] + end + survivor + end + + # An agency's own Rolodex card describes the agency: it fills the organization its OrgID + # names, or the one its name means, which is created when nothing answers to it. + def import_agency(rolodex) + data = rolodex.data + name = clip(data["Organization"].to_s.squish) + label = "#{rolodex.fm_id} #{name}".squish + organization = coded(data["OrgID"]) || (@name_matcher.call(name)&.organization if name) + return record(:skipped, "#{label}: agency card with no organization named") if organization.nil? && name.blank? + + organization ||= Organization.new(name:) + kept = fill(organization, website_url: clip(data["WebSite"]), mission_vision_values: clip(data["Mission"])) + stamp(organization, data) + add_addresses(organization, rolodex.addresses.map(&:data)) + add_phone(organization, rolodex.phone_numbers.map(&:data)) + fill_email(organization, [ data["EMail"] ]) unless names_a_person?(data) + outcome = agency_outcome(organization) + return failed(organization, label) unless organization.save + + note_differences(organization, rolodex, label, kept) if kept.any? + record(outcome, "#{label} → ##{organization.id} (agency card)") + end + + def agency_outcome(organization) + return :created if organization.new_record? + + organization.changed_for_autosave? ? :linked : :already_imported + end + + def agency_cards + FmRolodex.where("JSON_UNQUOTE(JSON_EXTRACT(data, '$.isAgency__c')) <> ''").order(:fm_id).select(&:agency?) + .reject { |rolodex| flag?(rolodex.data["Delete"]) } + end + + # A card naming a contact person carries that person's email, not the agency's. + def names_a_person?(data) + data["FirstName"].present? && data["LastName"].present? + end + + def coded(code) + code = code.to_s.strip + return if code.blank? + + ids = @codes.ids(code) + Organization.find_by(id: ids.first) if ids.any? + end + + # The organization by exact name, unless it already carries another code from the same + # archive table — then it's a namesake of a different FileMaker record. + def named(name, table_ids) + return if name.blank? + + organization = Organization.find_by(name:) + organization unless organization&.filemaker_codes&.any? { |code| table_ids.include?(code) } + end + + # Organization names are unique, so a FileMaker record whose name a namesake already + # holds is told apart by its code. + def unique_name(name, code) + Organization.exists?(name:) ? clip("#{name} (#{code})") : name + end + + def project_name(project) + clip(project.data["ProjectName"].to_s.squish) + end + + def organization_attributes(data) + { + website_url: clip(data["Website"]), + description: data["Description"].to_s.strip.presence, + mission_vision_values: clip(data["Mission"]), + organization_type: organization_type(data["OrgType"]), + **archive_timestamps(data["CreationDate"], data["ModificationDateTime"]) + } + end + + def project_attributes(data) + { + description: data["Description"].to_s.strip.presence, + organization_type: organization_type(data["AgencyType"]), + **archive_timestamps(data["DateEntered"], data["DateTimeModified"]) + } + end + + # The archive's own dates, so a created organization dates from when FileMaker first + # knew it. Only a new record takes them; fill passes them over on an existing one. + def archive_timestamps(entered, modified) + entered = parse_date(entered) + { created_at: entered, updated_at: parse_date(modified) || entered } + end + + def organization_type(value) + value = value.to_s.squish + return if value.blank? + + ORGANIZATION_TYPE_PATTERNS.find { |pattern, _type| value.match?(pattern) }&.last.tap do |type| + @unmatched_types[value] += 1 unless type + end + end + + # Staff edited both systems, so neither side is newer by rule: a blank takes the + # archive's value, and a disagreement keeps the portal's and is returned as + # [ label, portal value, archive value ] to note. + def fill(organization, attributes) + timestamps, fields = attributes.partition { |attribute, _value| %i[created_at updated_at].include?(attribute) } + timestamps.each { |attribute, value| organization[attribute] = value if organization.new_record? && value } + fields.filter_map do |attribute, value| + next if value.blank? + + if organization[attribute].blank? + organization[attribute] = value + next + end + next if organization[attribute].to_s.squish.casecmp?(value.to_s.squish) + + [ Organization.human_attribute_name(attribute), organization[attribute].to_s.truncate(80), value.to_s.truncate(80) ] + end + end + + # The staff FileMaker recorded as entering and last changing the row, or the settings' + # import user. An existing organization keeps any stamps it already has. + def stamp(organization, data) + organization.created_by_id ||= @staff.user_id_for(data["CreatedBy"].presence || data["EnteredBy"]) || @import_user_id + organization.updated_by_id ||= @staff.user_id_for(data["ModifiedBy"]) || @import_user_id + end + + # An organization gathers addresses from every source, so it keeps the ones it has and + # takes the archive's others; the first becomes its primary when it has none. + def add_addresses(organization, rows) + rows.each do |fields| + attributes = @address_attributes.call(fields, address_type: "work") + next if attributes.nil? || held_address(organization, attributes) + + primary = organization.addresses.none? { |address| address.primary? && !address.marked_for_destruction? } + organization.addresses.build(attributes.merge(primary:)) + end + end + + def held_address(organization, attributes) + key = address_key(attributes[:street_address], attributes[:city]) + organization.addresses.find { |address| address_key(address.street_address, address.city) == key } + end + + def address_key(street, city) + [ AddressNormalizer.street(street), AddressNormalizer.city(city) ] + end + + # FileMaker records a project's locality and districts, which the portal keeps on an + # address — so they land on the program's address in the project's city, or on a new + # one holding just the city when the program has no address there yet. + def place_project(organization, data) + city = clip(data["City"].to_s.squish) + state = clip(data["State"].to_s.squish) + return if city.blank? + + geography = project_geography(data) + address = organization.addresses.find { |held| AddressNormalizer.city(held.city) == AddressNormalizer.city(city) } + return fill_geography(address, geography) if address + return if state.blank? + + primary = organization.addresses.none?(&:primary?) + locality = geography[:locality] || AddressLocality.for(city:, state:, county: geography[:county]) + organization.addresses.build(street_address: "", zip_code: "", city:, state:, address_type: "work", + **geography, locality:, primary:) + end + + def project_geography(data) + { + locality: Address::LOCALITIES.find { |locality| locality.casecmp?(data["Locality"].to_s.squish) }, + county: clip(data["County"].to_s.squish), + district: clip(data["District"].to_s.squish), + la_service_planning_area: district_number(data["SPA"]), + la_supervisorial_district: district_number(data["LASupervisorialDistrict"]) + }.compact_blank + end + + # A locality worked out from the address's parts is a guess the project's own answer + # replaces when the guess was Unknown. + def fill_geography(address, geography) + geography.each do |attribute, value| + current = address[attribute] + address[attribute] = value if current.blank? || (attribute == :locality && current == Address::UNKNOWN_LOCALITY) + end + end + + def district_number(value) + value.to_s[/\d+/]&.to_i + end + + # The portal keeps an organization's phone on its address, so the archive's main + # number goes onto the primary one when that has none. + def add_phone(organization, rows) + number = rows.sort_by { |fields| flag?(fields["Primacy"]) ? 0 : 1 }.filter_map { |fields| clip(fields["PhoneNumber"].to_s.strip) }.first + address = organization.addresses.find(&:primary?) || organization.addresses.first + address.phone = number if number && address && address.phone.blank? + end + + def fill_email(organization, values) + organization.email = values.flat_map { |value| email_addresses(value) }.first if organization.email.blank? + end + + def note_differences(organization, row, label, kept) + lines = kept.map { |field, portal, archive| "#{field}: kept “#{portal}”, FileMaker has “#{archive}”" } + Comment.create!(commentable: organization, + body: "#{COMMENT_SOURCES[row.class]} #{row.fm_id} differs from this organization:\n#{lines.join("\n")}") + @differences << "#{label} → ##{organization.id}: #{lines.join('; ')}" + end + + def failed(organization, label) + record(:failed, "#{label}: #{"##{organization.id} " if organization.persisted?}#{organization.errors.full_messages.to_sentence}") + nil + end + + def record(outcome, message) + @results[outcome] << message + end + end +end diff --git a/app/services/person_services/import_from_rolodex.rb b/app/services/person_services/import_from_rolodex.rb index a37c60a122..c900f38daf 100644 --- a/app/services/person_services/import_from_rolodex.rb +++ b/app/services/person_services/import_from_rolodex.rb @@ -15,7 +15,8 @@ module PersonServices # field the archive disagrees on keeps the person's value — # noted in #differences and as a comment on the person # * already_imported — a person already carries the row's ID - # * skipped — agency, soft-deleted, or nameless rows + # * skipped — soft-deleted or nameless rows (an agency row naming no contact + # person is the agency itself, which the organizations import reads) # * failed — the Person didn't validate # # An email field holding no address the person's email validation accepts ("ada@@x.org", @@ -35,21 +36,16 @@ module PersonServices # named handful of rows (the task's FM_IDS), since a row whose created person has since # changed their email no longer matches and would be created again. class ImportFromRolodex + include FilemakerValues + OUTCOMES = %i[created linked already_imported skipped failed].freeze - FALSE_FLAGS = %w[0 n no false].freeze - DATE_FORMATS = %w[%m/%d/%Y %Y-%m-%d %m-%d-%Y %Y/%m/%d].freeze - # A two-digit year below this is read as 20xx, at or above it as 19xx. - TWO_DIGIT_YEAR_PIVOT = 10 - CONTACT_TYPES = AddressTranslation::CONTACT_TYPES # Answers to "which kind of email is this?" that aren't a kind at all. PRECEDENCE_TYPES = %w[preferred primary main].freeze - EMAIL_PATTERN = /[^\s,;<>"']+@[^\s,;<>"']+\.[^\s,;<>"']+/ # EmailPrimary holds a flag, never an address, so only EMail is read for one. The flag # says whether EMail is the person's primary email ("1") or their secondary ("2"). EMAIL_FIELDS = %w[EMail].freeze EMAIL_FLAG_FIELD = "EmailPrimary" SECONDARY_EMAIL_FLAG = "2" - STRING_LIMIT = 255 BATCH_SIZE = 1_000 # Whether a translated tag lands as primary or additional follows the field (ADR-0007). @@ -128,8 +124,8 @@ def import(rolodex) label = "#{rolodex.fm_id} #{first_name} #{last_name}".squish return record(:already_imported, label) if !@relink && @imported_codes.include?(rolodex.fm_id) - return record(:skipped, "#{label}: agency") if flag?(data["isAgency__c"]) return record(:skipped, "#{label}: marked deleted") if flag?(data["Delete"]) + return record(:skipped, "#{label}: agency with no contact person — the organizations import reads it") if rolodex.agency? && (first_name.blank? || last_name.blank?) return record(:skipped, "#{label}: missing first or last name") if first_name.blank? || last_name.blank? row_emails = emails(data) @@ -264,7 +260,7 @@ def fill_email_2(person, data) # The archive words an email field freely ("Attn: someone", a note, two addresses at # once), so each is read for the addresses in it; the same one twice counts once. def emails(data) - EMAIL_FIELDS.flat_map { |field| data[field].to_s.scan(EMAIL_PATTERN) }.select { |email| valid_email?(email) }.uniq(&:downcase) + EMAIL_FIELDS.flat_map { |field| email_addresses(data[field]) }.uniq(&:downcase) end def invalid_email_values(data) @@ -277,10 +273,6 @@ def usable_email_value?(value) addresses.any? && addresses.all? { |email| valid_email?(email) } end - def valid_email?(email) - email.length <= STRING_LIMIT && email.match?(URI::MailTo::EMAIL_REGEXP) - end - def same?(portal, archive) portal.to_s.squish.casecmp?(archive.to_s.squish) end @@ -394,22 +386,17 @@ def names(model, ids) # Every address the row answers to, not just the one it names as primary — # FmRolodex#addresses lists the primary first, so the first that can be built is the - # person's primary and the rest ride along. A work address, one the person already - # has, and a primary they already hold are all left as they are. + # person's primary and the rest ride along. An organization's address (a work address, + # or any address on an agency's row), one the person already has, and a primary they + # already hold are all left as they are; the organizations import places the first. def build_addresses(person, rolodex) held = person.addresses.map { |address| address_key(address.street_address, address.city) } - attributes = rolodex.addresses.reject { |address| work_address?(rolodex, address) } + attributes = rolodex.addresses.reject { |address| rolodex.organization_address?(address) } .filter_map { |address| @address_attributes.call(address.data) } .reject { |attrs| held.include?(address_key(attrs[:street_address], attrs[:city])) } build_primary_first(person.addresses, attributes) end - # A work address belongs to the person's organization, so the affiliations import brings it in. - def work_address?(rolodex, address) - address.fm_id == rolodex.data["WorksiteAddrsID"] || - contact_type(address.data["AddrsType"], allowed: Address::CONTACT_TYPES) == "work" - end - def build_phones(person, rolodex) held = person.contact_methods.map { |phone| phone_key(phone.value) } attributes = rolodex.phone_numbers.filter_map { |phone| phone_attributes(phone.data) } @@ -437,55 +424,6 @@ def phone_attributes(fields) { kind: "phone", value: number, contact_type: contact_type(fields["PhoneType"], allowed: ContactMethod::CONTACT_TYPES) } end - # The archive's types map onto one set of ours, but each model declares the slice it - # accepts — an address can be the mailing one, a phone number can't. A type we can't - # place is left blank, which both models then record as unknown. - def contact_type(value, allowed:) - CONTACT_TYPES[value.to_s.strip.downcase].presence_in(allowed) - end - - # Every string copied here lands in a varchar(255) the archive's free text isn't - # bounded by, and an over-long incidental field would otherwise cost the person — - # or, for a column with no length validation behind it, raise mid-import. - def clip(value) - value.presence && value.to_s.truncate(STRING_LIMIT) - end - - def parse_date(value) - value = value.to_s.strip - return if value.blank? - - DATE_FORMATS.each do |format| - parsed = parse_with(value, format) - return parsed if parsed - end - nil - end - - # Parsing stops where the format ends, so "10-5-1983" read as %Y-%m-%d takes 19 - # as the day and drops the 83 — digits left over mean the format split the value - # in the wrong place, so the next one gets its turn. - def parse_with(value, format) - parts = Date._strptime(value, format) - return if parts.nil? || parts[:leftover].to_s.match?(/\A\d/) - - expand_century(Date.new(parts[:year], parts[:mon], parts[:mday])) - rescue Date::Error, TypeError - nil - end - - # %Y reads a two-digit year as the year itself, so "10/5/83" is 83 AD. The archive - # mixes two- and four-digit years, so a bare pair gets a century: 83 is 1983, 05 is 2005. - def expand_century(date) - return date if date.year >= 100 - - date.change(year: date.year + (date.year < TWO_DIGIT_YEAR_PIVOT ? 2000 : 1900)) - end - - def flag?(value) - value.to_s.strip.presence && !FALSE_FLAGS.include?(value.to_s.strip.downcase) - end - def ids(people) people.map { |person| "##{person.id}" }.join(", ") end diff --git a/app/services/person_services/undo_rolodex_import.rb b/app/services/person_services/undo_rolodex_import.rb index 84be389436..89c121a390 100644 --- a/app/services/person_services/undo_rolodex_import.rb +++ b/app/services/person_services/undo_rolodex_import.rb @@ -1,7 +1,7 @@ module PersonServices - # Reverses one fm:import_people run from the activity events ImportFromRolodex logged under - # it (`import_run`). Newest first, so a person's addresses, tags and comments go before the - # person. A record created by the run is deleted, and a field the run filled goes back to + # Reverses one FileMaker import run (fm:import_people, fm:import_organizations) from the + # activity events it logged under it (`import_run`). Newest first, so a record's addresses, + # tags, comments and affiliations go before the record. A record created by the run is deleted, and a field the run filled goes back to # its value before the run — unless someone has changed it since, which is kept and reported. class UndoRolodexImport OUTCOMES = %i[deleted reverted kept gone].freeze diff --git a/db/seeds/features/2026-10-22-filemaker-organizations-and-affiliations.yml b/db/seeds/features/2026-10-22-filemaker-organizations-and-affiliations.yml new file mode 100644 index 0000000000..33644a0db7 --- /dev/null +++ b/db/seeds/features/2026-10-22-filemaker-organizations-and-affiliations.yml @@ -0,0 +1,23 @@ +name: "FileMaker organizations, programs, and affiliations" +area: people +display_status: admin_facing +released_on: 2026-10-22 +pr_number: 2788 +summary: >- + Organizations and art programs from the old FileMaker system can now be + brought into the portal, along with who worked or facilitated where and the + work addresses that go with them. +description: >- +
  • An organization running one art program comes in as one organization. + One running several gets each program nested under it.
  • Each program + carries FileMaker's locality and districts onto its address.
  • FileMaker's + facilitator assignments become Facilitator affiliations, dated from when they + were entered to when the program closed.
  • Job titles from FileMaker + become job affiliations.
  • Work addresses land on the organization, and + the person's affiliation there points at them.
  • Agency contact cards + fill in their organization's website and address, and the contact person they + name now comes in as a person.
+pro_tips: + - "An organization or affiliation already in the portal is linked rather than duplicated, and only its blank fields are filled in." + - "Where FileMaker says something different, the portal's value is kept and a comment on the record lists what FileMaker had." +action_path: "/organizations" diff --git a/docs/adr/0005-filemaker-code-archive-mapping.md b/docs/adr/0005-filemaker-code-archive-mapping.md index 8f5ea1b3b5..a7d8baa687 100644 --- a/docs/adr/0005-filemaker-code-archive-mapping.md +++ b/docs/adr/0005-filemaker-code-archive-mapping.md @@ -25,7 +25,7 @@ drive the archive pages' links. |---|---|---|---| | `people.filemaker_code` | `fm_rolodexes` | `ID`, e.g. `00006` | `filemaker_code_links` on person edit + admin details | | `organizations.filemaker_code` | `fm_projects`, then `fm_organizations` | `ProjectID`, e.g. `P2775`; `OrgID`, e.g. `ORG07206` | `filemaker_code_links` on org form; `filemaker_project_links` / `filemaker_organization_links` on org edit | -| `affiliations.filemaker_code` | `fm_projects`, then `fm_personnels` | `ProjectID`, e.g. `P2775`; rarely `PrsnlRecID`, e.g. `PS9999` | `filemaker_code_links` on affiliation edit | +| `affiliations.filemaker_code` | `fm_projects`, then `fm_personnels`, then `fm_people_to_organizations` | `ProjectID`, e.g. `P2775`; `PrsnlRecID`, e.g. `PS9999`, and `RecordID`, e.g. `112`, from the ADR-0008 import | `filemaker_code_links` on affiliation edit | | `payments.filemaker_code` | `fm_payments` | `RecordID`, e.g. `D29186` | `filemaker_code_hint` on payment edit | | `workshops.filemaker_code` | `fm_workshops` | `WorkshopID` | `filemaker_code_hint` on workshop form | | `resources.filemaker_code` | none | `RecordID` of the source quotation, e.g. `Q10215` | resource form (editable, no archive link) | diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 0a55b214ab..3f290a4866 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -19,6 +19,8 @@ The archive tables involved (row counts from the export): | `fm_projects` | 1,976 | One art program run at an organization (`ProjectID`, `OrgID`) | | `fm_personnels` | 8,907 | A person assigned to a project, with role and agreement status (`PersonID`, `ProjectID`) | | `fm_people_to_organizations` | 1,127 | A person's job at an organization, with title (`PersonID`, `OrgID`) | +| `fm_rolodexes` | 39,608 | The person's own card: their organization (`OrgID`, `Organization`), `Title`, worksite (`WorksiteAddrsID`); or, flagged `isAgency__c`, an agency's card | +| `fm_addresses`, `fm_phone_numbers`, `fm_email_addresses` | 28,398 / 33,915 / 29 | Contact rows, filed under a Rolodex card (`RolodexID`), an organization (`OrgID`), or both | Constraints already decided elsewhere: @@ -37,20 +39,52 @@ Constraints already decided elsewhere: - **Order:** people (#2630) → organizations → projects → affiliations. Each pass finds the records the previous one created by `filemaker_code`, so each is idempotent on its own. + `fm:import_organizations` runs organizations + projects + (`OrganizationServices::ImportFromFilemaker`), then affiliations + (`AffiliationServices::ImportFromFilemaker`). - **Organizations and projects:** an `fm_projects` row becomes (or links to) an app `Organization`; its `fm_organizations` row becomes (or links to) that org's parent. Match an existing org by any shared `filemaker_code` first, then by exact name; append the code on a match, as the Rolodex pass does for people. + - **An organization with exactly one project is one org**, carrying both codes — a parent + with a single child would be the same place twice. Only an organization with several + projects (or none) gets its own org with the programs nested under it. When the portal + already keeps the two apart, it stays that way. + - **A name match only reaches an org carrying no code from the same table.** One already + holding another project's code is a namesake; the new org is named `"Name (P1234)"`, + since org names are unique. + - Every FileMaker organization comes over, whether or not a project or person points at it. +- **Agencies:** a Rolodex card flagged `isAgency__c` is the agency's own card. Its website, + mission and addresses fill the org its `OrgID` names (or the one its `Organization` name + matches, created when nothing does); its email too, unless the card names a contact + person, whose email it then is. A card naming a contact person also becomes that person + in the people pass, and every address on it stays with the org. - **Personnel → facilitator affiliation:** an `fm_personnels` row becomes an `Affiliation` between the person (by Rolodex code) and the project's org, carrying `PrsnlRecID` as its - `filemaker_code` so each row imports once. `Role` → `title`; `Status` / dates → - `start_date` / `end_date` / `inactive`. `event_registration_id` stays NULL — these are + `filemaker_code` so each row imports once. `event_registration_id` stays NULL — these are historical, not training-minted. + - **Role:** a blank role or "Facilitator" is a Facilitator affiliation. A role naming more + ("Lead Facilitator") is a Facilitator affiliation plus a job affiliation with that title, + as registration mints them (ADR-0003 D1); any other role ("Supervisor") is a job + affiliation alone. + - **Dates:** Personnel rows carry none of their own, so the assignment starts on + `DateEntered` and ends on the project's `CloseDate` (else `EndDate`). A `Status` outside + `ACTIVE_STATUSES` ends it now, undated (`inactive`). The run tallies every Status value + and how it was read, so the list can be corrected before the real run. + - An existing same-title affiliation at that org is linked (takes the code, fills blank + dates) unless it already records another Personnel row — a second stint stays its own row. - **People-to-organizations → job affiliation:** an `fm_people_to_organizations` row becomes - an `Affiliation` to the umbrella org with its `Title`, carrying `RecordID` as its code. -- **Work addresses:** the Rolodex `WorksiteAddrsID` address and any address typed Work - become an `Address` on the organization, and the person's affiliation there points at it - (`affiliations.organization_address_id`). + an `Affiliation` to the org carrying its `OrgID` code with its `Title`, carrying `RecordID` + as its code. Undated, per ADR-0003 D1. +- **The Rolodex card → job affiliation:** the card's `OrgID` (or its typed `Organization`, via + `NameMatcher`) and `Title` become a job affiliation there — the field `FmRolodex::PORTAL_FIELDS` + already mapped to `affiliations.title`. No code: the card's ID is the person's. +- **Work addresses:** the Rolodex `WorksiteAddrsID` address, any address typed Work, and + every address on an agency's card become an `Address` on the organization, and the + person's affiliations there that name no address point at it + (`affiliations.organization_address_id`), the worksite first. The organization is the one + FileMaker filed the address under (its `OrgID`), else the card's, else the person's only + one. A work address with no organization to land on stays with the person, typed work. - **Locality and districts come from the project, not derived.** `fm_projects` is the one place FileMaker records a `Locality`, and its vocabulary is where `Address::LOCALITIES` came from — so a project's `Locality`, `District`, `SPA` and `LASupervisorialDistrict` @@ -69,15 +103,22 @@ Constraints already decided elsewhere: ## Open questions -- **Which org does a job affiliation attach to** — the umbrella (`fm_organizations`) or the - person's project at that org, when there is exactly one? -- **Reconcile vs. history.** The reconcile page ends facilitator affiliations no attended - training backs, hand-entered ones included. Imported personnel rows would be swept up - unless they're exempted (e.g. by their `PS…` code) — exempt them, or let reconcile end - them? -- **Which `Status` / `AgreementStatus` values mean active**, and is a closed project - (`CloseDate`, `EndDate`) an end date for every affiliation under it? -- **Merged projects** (`MergerWithID`): fold the merged-away project into the survivor's - org, or skip it? -- **Unmatched people:** a personnel row whose `PersonID` no person carries — skip and - report, or create the person from the Rolodex row on the spot? +The first draft of the import (#2788) answers these provisionally; confirm or change before +the real run. + +- **Which org does a job affiliation attach to** — the one carrying the `OrgID` code. For an + organization with one project that's the program's own org, so the question only remains + for organizations with several projects, where it's the umbrella. +- **Reconcile vs. history.** Still open. The reconcile page ends facilitator affiliations no + attended training backs, hand-entered ones included, and imported personnel rows are + hand-entered by that test. Exempt rows carrying a `PS…` code, or let reconcile end them? +- **Which `Status` values mean active** — provisionally "Active" and "Current" + (`AffiliationServices::ImportFromFilemaker::ACTIVE_STATUSES`); the dry run's tally is the + check. A closed project's `CloseDate` (else `EndDate`) ends every affiliation under it. + `AgreementStatus` isn't read. +- **Merged projects** (`MergerWithID`): folded — the merged-away project's code joins the + survivor's org, so its personnel land there. One the portal already keeps as its own org + is reported for a hand merge rather than merged automatically. +- **Unmatched people:** skipped and reported. +- **Every FileMaker organization, or only the ones something points at?** The draft brings all + of them. diff --git a/lib/tasks/fm_import_organizations.rake b/lib/tasks/fm_import_organizations.rake new file mode 100644 index 0000000000..7338737fc2 --- /dev/null +++ b/lib/tasks/fm_import_organizations.rake @@ -0,0 +1,112 @@ +# frozen_string_literal: true + +# Bring FileMaker's organizations, projects and the ties between them and people into the +# portal (ADR-0008). Run after fm:import_people, which makes the people; this pass finds them +# by their Rolodex code. Two steps, in order: +# 1. OrganizationServices::ImportFromFilemaker — organizations, projects (as programs nested +# under them), their addresses, phones, emails, and agencies' own Rolodex cards +# 2. AffiliationServices::ImportFromFilemaker — personnel, people-to-organizations, and each +# Rolodex card's organization, title and work addresses +# Both are idempotent: a row whose code a record already carries is left alone. +# +# Dry-run by default. Set DRY_RUN=false to write changes; VERBOSE=true lists every row. +# A real run commits each batch as it goes, so stopping partway keeps what it finished and +# the rerun skips it. The full report is also saved as an ActiveStorage blob. Every record a +# run creates or fills is logged under the run's name, which fm:undo_import reverses: +# bin/rails fm:import_organizations # preview, writes nothing +# DRY_RUN=false bin/rails fm:import_organizations # create + link +# DRY_RUN=false bin/rails fm:undo_import RUN=fm-import-organizations-20261022134055 +namespace :fm do + desc "Create organizations and affiliations from the FileMaker archive (after fm:import_people). Idempotent. DRY_RUN=false to execute." + task import_organizations: :environment do + unless Setting.import_user + abort "No import user chosen — pick one on /admin/settings (Imports); it's credited as created_by/updated_by when FileMaker names no staffer we can resolve." + end + + dry_run = ENV["DRY_RUN"] != "false" + verbose = ENV["VERBOSE"] == "true" + report = [] + say = ->(line = "") { puts line; report << line } + $stdout.sync = true + + begin + run = "fm-import-organizations-#{Time.current.strftime("%Y%m%d%H%M%S")}" + Current.source = "fm_import_organizations" + say.("Run #{run}#{" (dry run)" if dry_run}") + started = Time.current + steps = { + "Organizations and projects" => OrganizationServices::ImportFromFilemaker, + "Affiliations and work addresses" => AffiliationServices::ImportFromFilemaker + } + imports = {} + + run_steps = lambda do + steps.each do |title, service| + say.("\n#{title}…") + progress = ->(read) { puts " #{read} rows read — #{(Time.current - started).round}s" } + imports[title] = service.new(progress:, run:).call + end + end + + # A dry run rolls back as a whole; a real one lets each batch commit as it goes. + if dry_run + ActiveRecord::Base.transaction do + run_steps.call + raise ActiveRecord::Rollback + end + else + run_steps.call + end + + imports.each do |title, import| + say.("\n== #{title} ==") + import.results.each do |outcome, messages| + next unless (verbose || %i[failed skipped].include?(outcome)) && messages.any? + + say.("\n#{outcome.to_s.humanize} (#{messages.size}):") + messages.each { |message| say.(" #{message}") } + end + + if import.differences.any? + say.("\nKept the portal's value where FileMaker differs (#{import.differences.size}, also left as a comment):") + import.differences.each { |difference| say.(" #{difference}") } + end + + tallies = { + "Organization types no portal type answers to (left blank)" => import.try(:unmatched_types), + "Personnel Status values and how they were read" => import.try(:statuses), + "Personnel roles" => import.try(:roles) + } + tallies.each do |heading, counts| + next if counts.blank? + + say.("\n#{heading}:") + counts.sort_by { |name, count| [ -count, name ] }.each { |name, count| say.(" #{count.to_s.rjust(5)} #{name}") } + end + + say.() + import.results.each { |outcome, messages| say.("#{outcome.to_s.humanize.ljust(17)} #{messages.size}") } + end + + say.("\nTook #{(Time.current - started).round}s.") + say.(dry_run ? "\nDRY RUN — rolled back, nothing written. Set DRY_RUN=false to execute." : "\nDone.") + say.("To reverse this run: DRY_RUN=false bin/rails fm:undo_import RUN=#{run}") unless dry_run + rescue StandardError => e + say.("\nStopped by #{e.class}: #{e.message}#{" — the batches before this one are saved; rerun to continue." unless dry_run}") + raise + ensure + begin + blob = ActiveStorage::Blob.create_and_upload!( + io: StringIO.new(report.join("\n")), content_type: "text/plain", + filename: "fm-import-organizations-#{Time.current.strftime("%Y%m%d-%H%M%S")}#{"-dry-run" if dry_run}.txt" + ) + puts "\nReport saved. Read it back with: bin/rails runner 'puts ActiveStorage::Blob.find(#{blob.id}).download'" + rescue StandardError => e + puts "\nCould not save the report (#{e.class}: #{e.message}) — copy the output above." + end + end + end + + desc "Reverse any FileMaker import run (RUN=fm-import-…) from the activity it logged. Dry-run by default; DRY_RUN=false to execute." + task undo_import: :undo_import_people +end diff --git a/lib/tasks/fm_import_people.rake b/lib/tasks/fm_import_people.rake index 4db19c862c..e4d33c70aa 100644 --- a/lib/tasks/fm_import_people.rake +++ b/lib/tasks/fm_import_people.rake @@ -107,7 +107,7 @@ namespace :fm do desc "Reverse one fm:import_people run (RUN=fm-import-people-…) from the activity it logged. Dry-run by default; DRY_RUN=false to execute." task undo_import_people: :environment do - run = ENV["RUN"].presence || abort("Name the run to reverse: RUN=fm-import-people-… (the import prints it).") + run = ENV["RUN"].presence || abort("Name the run to reverse: RUN=fm-import-… (the import prints it).") dry_run = ENV["DRY_RUN"] != "false" Current.source = "fm_undo_import_people" undo = nil diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb new file mode 100644 index 0000000000..b7c399639b --- /dev/null +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -0,0 +1,135 @@ +# frozen_string_literal: true + +require "rails_helper" + +RSpec.describe AffiliationServices::ImportFromFilemaker do + let!(:person) { create(:person, filemaker_code: "00006") } + let!(:program) { create(:organization, filemaker_code: "P0001") } + let!(:umbrella) { create(:organization, filemaker_code: "ORG00001") } + + before { FmProject.create!(fm_id: "P0001", data: { "OrgID" => "ORG00001", "CloseDate" => "6/30/2019" }) } + + def personnel(fm_id, **data) + FmPersonnel.create!(fm_id:, data: { "PersonID" => "00006", "ProjectID" => "P0001", "Role" => "Facilitator", + "Status" => "Active", "DateEntered" => "3/1/2015" }.merge(data.stringify_keys)) + end + + def rolodex(**data) + FmRolodex.create!(fm_id: "00006", data: { "FirstName" => "Ada", "LastName" => "Lovelace" }.merge(data.stringify_keys)) + end + + def import + described_class.new.call + end + + describe "personnel" do + it "makes a Facilitator affiliation dated from entry to the project's close" do + personnel("PS0001") + + expect(import.results[:created].size).to eq(1) + expect(person.affiliations.sole).to have_attributes(organization: program, title: "Facilitator", filemaker_code: "PS0001", + start_date: Date.new(2015, 3, 1), end_date: Date.new(2019, 6, 30), + event_registration_id: nil) + end + + it "ends an assignment whose status isn't active, and tallies the statuses it read" do + FmProject.find_by!(fm_id: "P0001").update!(data: { "OrgID" => "ORG00001" }) + personnel("PS0001", Status: "Dropped") + + import = described_class.new.call + + expect(person.affiliations.sole).to have_attributes(end_date: nil, inactive: true) + expect(import.statuses).to eq("Dropped → ended" => 1) + end + + it "also records a more specific facilitating role as a job title" do + personnel("PS0001", Role: "Lead Facilitator") + + import + + expect(person.affiliations.order(:title).pluck(:title, :start_date, :filemaker_code)) + .to eq([ [ "Facilitator", Date.new(2015, 3, 1), "PS0001" ], [ "Lead Facilitator", nil, "PS0001" ] ]) + end + + it "links the Facilitator affiliation the person already has there, filling its blank start" do + existing = create(:affiliation, person:, organization: program, title: "Facilitator", filemaker_code: "P0001", start_date: nil) + personnel("PS0001") + + expect(import.results[:linked].size).to eq(1) + expect(existing.reload).to have_attributes(filemaker_code: "P0001, PS0001", start_date: Date.new(2015, 3, 1)) + expect(person.affiliations.count).to eq(1) + end + + it "keeps a second stint on the same project as its own affiliation" do + personnel("PS0001") + personnel("PS0002", DateEntered: "1/5/2018") + + import + + expect(person.affiliations.pluck(:filemaker_code)).to contain_exactly("PS0001", "PS0002") + end + + it "skips a row whose person or project wasn't imported" do + personnel("PS0001", PersonID: "99999") + personnel("PS0002", ProjectID: "P9999") + + expect(import.results[:skipped]).to contain_exactly(a_string_including("PS0001", "99999"), a_string_including("PS0002", "P9999")) + expect(Affiliation.count).to eq(0) + end + end + + it "makes a job affiliation from a people-to-organizations row" do + FmPeopleToOrganization.create!(fm_id: "12", data: { "PersonID" => "00006", "OrgID" => "ORG00001", "Title" => "Board Chair" }) + + import + + expect(person.affiliations.sole).to have_attributes(organization: umbrella, title: "Board Chair", filemaker_code: "12", start_date: nil) + end + + describe "the Rolodex card" do + it "affiliates the person with their organization and points them at its worksite" do + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", "AddrsState" => "CA" }) + FmAddress.create!(fm_id: "41", data: { "RolodexID" => "00006", "AddrsLine1" => "2 Home Rd", "AddrsCity" => "Pasadena", "AddrsState" => "CA", + "AddrsType" => "Home" }) + rolodex(OrgID: "ORG00001", Title: "Counselor", WorksiteAddrsID: "40", PrimaryAddrsID: "41") + + import + + affiliation = person.affiliations.sole + expect(affiliation).to have_attributes(organization: umbrella, title: "Counselor", filemaker_code: nil) + expect(affiliation.organization_address).to have_attributes(addressable: umbrella, street_address: "1 Main St", address_type: "work") + expect(person.addresses).to be_empty + end + + it "places a work address on the person's only organization when the card names none" do + create(:affiliation, person:, organization: program, title: "Facilitator") + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", + "AddrsState" => "CA", "AddrsType" => "Work" }) + rolodex + + import + + expect(person.affiliations.sole.organization_address).to have_attributes(addressable: program, street_address: "1 Main St") + end + + it "keeps a work address with the person when no organization can take it" do + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", + "AddrsState" => "CA", "AddrsType" => "Work" }) + rolodex + + import + + expect(person.addresses.sole).to have_attributes(street_address: "1 Main St", address_type: "work") + end + end + + it "changes nothing when it runs again" do + personnel("PS0001", Role: "Lead Facilitator") + FmPeopleToOrganization.create!(fm_id: "12", data: { "PersonID" => "00006", "OrgID" => "ORG00001", "Title" => "Board Chair" }) + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", "AddrsState" => "CA" }) + rolodex(OrgID: "ORG00001", Title: "Counselor", WorksiteAddrsID: "40") + import + + expect { import }.not_to change { [ Affiliation.count, Address.count, Comment.count, Affiliation.maximum(:updated_at) ] } + end +end diff --git a/spec/services/organization_services/import_from_filemaker_spec.rb b/spec/services/organization_services/import_from_filemaker_spec.rb new file mode 100644 index 0000000000..5c8ab23004 --- /dev/null +++ b/spec/services/organization_services/import_from_filemaker_spec.rb @@ -0,0 +1,128 @@ +# frozen_string_literal: true + +require "rails_helper" + +RSpec.describe OrganizationServices::ImportFromFilemaker do + def fm_organization(fm_id, **data) + FmOrganization.create!(fm_id:, data: { "OrgName" => "Harbor House" }.merge(data.stringify_keys)) + end + + def project(fm_id, org_id, **data) + FmProject.create!(fm_id:, data: { "OrgID" => org_id, "ProjectName" => "Harbor House Art Program" }.merge(data.stringify_keys)) + end + + def import + described_class.new.call + end + + it "keeps an organization with one project as one organization carrying both codes" do + fm_organization("ORG00001", Website: "harbor.example.org", Mission: "Shelter for all", OrgType: "Nonprofit") + project("P0001", "ORG00001", City: "Pasadena", State: "CA", County: "Los Angeles", Locality: "la county", + SPA: "SPA 3", LASupervisorialDistrict: "5", District: "Foothills") + FmAddress.create!(fm_id: "1", data: { "OrgID" => "ORG00001", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", + "AddrsState" => "CA", "AddrsPostalCode" => "91101" }) + FmPhoneNumber.create!(fm_id: "2", data: { "OrgID" => "ORG00001", "PhoneNumber" => "555-0100" }) + FmEmailAddress.create!(fm_id: "3", data: { "OrgID" => "ORG00001", "EmailAddress" => "Attn: info@harbor.example.org" }) + + expect(import.results[:created].size).to eq(1) + + organization = Organization.sole + expect(organization).to have_attributes(name: "Harbor House", filemaker_code: "ORG00001, P0001", parent_id: nil, + website_url: "harbor.example.org", mission_vision_values: "Shelter for all", + organization_type: "501c3/nonprofit", email: "info@harbor.example.org") + expect(organization.addresses.sole).to have_attributes(street_address: "1 Main St", city: "Pasadena", zip_code: "91101", + address_type: "work", primary: true, phone: "555-0100", + locality: "LA County", district: "Foothills", + la_service_planning_area: 3, la_supervisorial_district: 5) + end + + it "nests each of an organization's several projects under it" do + fm_organization("ORG00002", OrgName: "Valley Services") + project("P0002", "ORG00002", ProjectName: "Valley Adult Program", City: "Fresno", State: "CA", Locality: "Central CA") + project("P0003", "ORG00002", ProjectName: "Valley Youth Program") + + expect(import.results[:created].size).to eq(3) + + umbrella = Organization.find_by!(name: "Valley Services") + expect(umbrella.filemaker_code).to eq("ORG00002") + expect(umbrella.children.pluck(:name, :filemaker_code)).to contain_exactly([ "Valley Adult Program", "P0002" ], + [ "Valley Youth Program", "P0003" ]) + expect(Organization.find_by!(name: "Valley Adult Program").addresses.sole) + .to have_attributes(street_address: "", city: "Fresno", locality: "Central CA") + end + + it "fills an organization already carrying the code, keeping and noting what it holds" do + existing = create(:organization, name: "Harbor House Inc", filemaker_code: "P0001", website_url: "old.example.org") + fm_organization("ORG00001", Website: "harbor.example.org", Mission: "Shelter for all") + project("P0001", "ORG00001") + + import = described_class.new.call + + expect(import.results[:linked].size).to eq(1) + expect(existing.reload).to have_attributes(name: "Harbor House Inc", filemaker_code: "ORG00001, P0001", + website_url: "old.example.org", mission_vision_values: "Shelter for all") + expect(existing.comments.sole.body).to include("ORG00001", "old.example.org", "harbor.example.org") + expect(import.differences.sole).to include("##{existing.id}") + end + + it "matches by exact name only an organization with no code from the same table" do + taken = create(:organization, name: "Harbor House Art Program", filemaker_code: "P0009") + project("P0009", nil, ProjectName: "Somewhere else") + fm_organization("ORG00001", OrgName: "Harbor House") + project("P0001", "ORG00001", ProjectName: "Harbor House Art Program") + project("P0004", "ORG00001", ProjectName: "Harbor House Youth") + + import + + expect(taken.reload.filemaker_code).to eq("P0009") + expect(Organization.find_by!(name: "Harbor House Art Program (P0001)").parent.name).to eq("Harbor House") + end + + it "still brings in the projects of a nameless organization, unnested" do + fm_organization("ORG00003", OrgName: "") + project("P0006", "ORG00003", ProjectName: "Lone Program") + + expect(import.results[:skipped].sole).to include("ORG00003", "no name") + expect(Organization.sole).to have_attributes(name: "Lone Program", filemaker_code: "P0006", parent_id: nil) + end + + it "folds a merged-away project's code into the program it merged into" do + fm_organization("ORG00001") + project("P0001", "ORG00001") + project("P0005", "ORG00001", ProjectName: "Old Harbor Program", MergerWithID: "P0001") + + results = import.results + + expect(results[:linked].sole).to include("P0005", "merged into P0001") + expect(Organization.sole.filemaker_code).to eq("ORG00001, P0001, P0005") + end + + it "reads an agency's own Rolodex card onto its organization" do + fm_organization("ORG00001") + FmAddress.create!(fm_id: "7", data: { "RolodexID" => "00090", "AddrsLine1" => "9 Elm St", "AddrsCity" => "Ojai", + "AddrsState" => "CA", "AddrsType" => "Home" }) + FmRolodex.create!(fm_id: "00090", data: { "isAgency__c" => "1", "OrgID" => "ORG00001", "Organization" => "Harbor House", + "WebSite" => "harbor.example.org", "EMail" => "front@harbor.example.org", + "FirstName" => "Ada", "LastName" => "Lovelace" }) + FmRolodex.create!(fm_id: "00091", data: { "isAgency__c" => "1", "Organization" => "Elm Street Clinic", + "EMail" => "hello@elm.example.org" }) + + import + + harbor = Organization.find_by!(filemaker_code: "ORG00001") + expect(harbor).to have_attributes(website_url: "harbor.example.org", email: nil) + expect(harbor.addresses.sole).to have_attributes(street_address: "9 Elm St", address_type: "work") + expect(Organization.find_by!(name: "Elm Street Clinic")).to have_attributes(email: "hello@elm.example.org", filemaker_code: nil) + end + + it "changes nothing when it runs again" do + fm_organization("ORG00002", OrgName: "Valley Services") + project("P0002", "ORG00002", ProjectName: "Valley Adult Program", City: "Fresno", State: "CA") + project("P0003", "ORG00002", ProjectName: "Valley Youth Program") + FmAddress.create!(fm_id: "1", data: { "OrgID" => "ORG00002", "AddrsLine1" => "1 Main St", "AddrsCity" => "Fresno", "AddrsState" => "CA" }) + import + + expect { expect(import.results[:already_imported].size).to eq(3) } + .not_to change { [ Organization.count, Address.count, Comment.count ] } + end +end diff --git a/spec/services/person_services/import_from_rolodex_spec.rb b/spec/services/person_services/import_from_rolodex_spec.rb index c90cf7d3c6..6850e33d30 100644 --- a/spec/services/person_services/import_from_rolodex_spec.rb +++ b/spec/services/person_services/import_from_rolodex_spec.rb @@ -843,15 +843,23 @@ def run_events(run) = Ahoy::Event.where("JSON_UNQUOTE(JSON_EXTRACT(properties, ' end end - it "skips agencies, deleted rows, and nameless rows" do - rolodex("00013", isAgency__c: "1") + it "skips deleted rows, nameless rows, and agency cards naming no contact person" do + rolodex("00013", isAgency__c: "1", FirstName: "", LastName: "", Organization: "Harbor House") rolodex("00014", Delete: "Yes") rolodex("00015", FirstName: "", LastName: "") - expect(results[:skipped].size).to eq(3) + expect(results[:skipped]).to contain_exactly(a_string_including("00013", "agency"), a_string_including("00014"), a_string_including("00015")) expect(Person.where(filemaker_code: %w[00013 00014 00015])).to be_empty end + it "creates the contact person an agency card names, leaving the agency's addresses to its organization" do + FmAddress.create!(fm_id: "A1", data: { "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", "AddrsState" => "CA", "AddrsType" => "Home" }) + rolodex("00012", isAgency__c: "1", Organization: "Harbor House", EMail: "ada@harbor.example.org", PrimaryAddrsID: "A1") + + expect(results[:created].size).to eq(1) + expect(Person.find_by!(filemaker_code: "00012").addresses).to be_empty + end + it "treats a false-y flag as unset" do rolodex("00016", isAgency__c: "0", Delete: "No") diff --git a/spec/tasks/fm_import_organizations_spec.rb b/spec/tasks/fm_import_organizations_spec.rb new file mode 100644 index 0000000000..f95874497f --- /dev/null +++ b/spec/tasks/fm_import_organizations_spec.rb @@ -0,0 +1,69 @@ +require "rails_helper" +require "rake" + +RSpec.describe "fm:import_organizations" do + before(:all) do + Rails.application.load_tasks unless Rake::Task.task_defined?("fm:import_organizations") + end + + after { Current.reset } + + let!(:person) { create(:person, filemaker_code: "00200") } + + before do + FmOrganization.create!(fm_id: "ORG00001", data: { "OrgName" => "Harbor House" }) + FmProject.create!(fm_id: "P0001", data: { "OrgID" => "ORG00001", "ProjectName" => "Harbor House Art Program" }) + FmPersonnel.create!(fm_id: "PS0001", data: { "PersonID" => "00200", "ProjectID" => "P0001", "Role" => "Facilitator", "Status" => "Active" }) + end + + def run_task(dry_run: true, task: "fm:import_organizations", run: nil) + ENV["DRY_RUN"] = dry_run ? "true" : "false" + ENV["RUN"] = run + output = StringIO.new + original_stdout = $stdout + $stdout = output + Rake::Task[task].reenable + Rake::Task[task].prerequisite_tasks.each(&:reenable) + Rake::Task[task].invoke + output.string + rescue SystemExit => e + raise "#{task} aborted: #{e.message}" + ensure + $stdout = original_stdout + ENV.delete("DRY_RUN") + ENV.delete("RUN") + end + + it "stops before reading a row when no import user is chosen" do + expect { run_task(dry_run: false) }.to raise_error(/aborted: No import user chosen/) + expect(Organization.count).to eq(0) + end + + context "with an import user chosen" do + before { Setting.create!(import_user: create(:user)) } + + it "rolls a dry run back and saves its report" do + expect { run_task }.to change(ActiveStorage::Blob, :count).by(1) + + expect(Organization.count).to eq(0) + expect(ActiveStorage::Blob.last.download).to include("Organizations and projects", "Affiliations and work addresses", "DRY RUN") + end + + it "makes the organization, then the affiliation, on a real run" do + run_task(dry_run: false) + + organization = Organization.sole + expect(organization.filemaker_code).to eq("ORG00001, P0001") + expect(person.affiliations.sole).to have_attributes(organization:, title: "Facilitator", filemaker_code: "PS0001") + end + + it "names the run so fm:undo_import can reverse it" do + run = run_task(dry_run: false)[/ RUN=(fm-import-organizations-\d+)/, 1] + + run_task(task: "fm:undo_import", run:, dry_run: false) + + expect(Organization.count).to eq(0) + expect(Affiliation.count).to eq(0) + end + end +end From d6c943d259366959939c9fdaf1cc6e2002c1d8ef Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 07:53:00 -0400 Subject: [PATCH 03/25] Record that every FileMaker organization comes over Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/adr/0008-filemaker-organization-and-personnel-import.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 3f290a4866..33af8921a6 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -53,7 +53,8 @@ Constraints already decided elsewhere: - **A name match only reaches an org carrying no code from the same table.** One already holding another project's code is a namesake; the new org is named `"Name (P1234)"`, since org names are unique. - - Every FileMaker organization comes over, whether or not a project or person points at it. + - Every FileMaker organization comes over, whether or not a project or person points at it + (decided 2026-10-24). - **Agencies:** a Rolodex card flagged `isAgency__c` is the agency's own card. Its website, mission and addresses fill the org its `OrgID` names (or the one its `Organization` name matches, created when nothing does); its email too, unless the card names a contact @@ -120,5 +121,3 @@ the real run. survivor's org, so its personnel land there. One the portal already keeps as its own org is reported for a hand merge rather than merged automatically. - **Unmatched people:** skipped and reported. -- **Every FileMaker organization, or only the ones something points at?** The draft brings all - of them. From eb8433b5561450a091ab635d0ab33c5044ba8f9e Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 08:15:04 -0400 Subject: [PATCH 04/25] Read Personnel roles and org types as production records them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The archive's real values showed no Personnel role says Facilitator — FileMaker calls a facilitator a Leader — so every assignment would have landed as a job and no program would read as running. Liaisons and assistants support a program without facilitating it, so they stay jobs. Organization types were already close to the portal's own four; a word none of them covers is Other with the word kept, not dropped. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 4 +- .../import_from_filemaker.rb | 46 +++++++++---------- .../import_from_filemaker.rb | 33 ++++++++----- ...maker-organization-and-personnel-import.md | 30 +++++++----- lib/tasks/fm_import_organizations.rake | 2 +- .../import_from_filemaker_spec.rb | 18 +++++--- .../import_from_filemaker_spec.rb | 40 ++++++++++++++++ spec/tasks/fm_import_organizations_spec.rb | 2 +- 8 files changed, 118 insertions(+), 57 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 762002fc7a..2cab4fa071 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -332,7 +332,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Facilitator affiliation on the project's org (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` outside `ACTIVE_STATUSES` ends it undated), plus a job affiliation for a more specific role ("Lead Facilitator", "Supervisor"); people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Leader (or blank/stray role) becomes a Facilitator affiliation on the project's org (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), a Liaison or Assistant (`JOB_ROLES`) an undated job affiliation; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) @@ -357,7 +357,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Organizations -- `OrganizationServices::ImportFromFilemaker` — The FileMaker organizations pass (ADR-0008): each `FmOrganization` and its `FmProject`s become portal orgs — an org with exactly one project stays one org carrying both codes, otherwise the projects nest under it (`parent`). Matches by code (`FilemakerCodeIndex`), then exact name only to an org carrying no code from that table (a namesake gets `"Name (code)"`). Fills blanks only (website, description, mission, `organization_type` from `OrgType`/`AgencyType` loosely — unmatched words tallied in `#unmatched_types`), noting differences as comments; adds the org's `fm_addresses`, its phone onto the primary address, its `fm_email_addresses`; copies each project's `Locality`/`District`/`SPA`/`LASupervisorialDistrict` onto the program's address in its city (a city-only one when none); folds a merged-away project's code (`MergerWithID`) into the survivor; and reads agencies' own Rolodex cards (website, mission, addresses, and email when no contact person is named) onto their org, creating it by name when nothing answers. Driven by `fm:import_organizations` +- `OrganizationServices::ImportFromFilemaker` — The FileMaker organizations pass (ADR-0008): each `FmOrganization` and its `FmProject`s become portal orgs — an org with exactly one project stays one org carrying both codes, otherwise the projects nest under it (`parent`). Matches by code (`FilemakerCodeIndex`), then exact name only to an org carrying no code from that table (a namesake gets `"Name (code)"`). Fills blanks only (website, description, mission, `organization_type` from `OrgType`/`AgencyType` via `ORGANIZATION_TYPE_PATTERNS` — any other word lands as Other with the word in `organization_type_other`, tallied in `#unmatched_types`), noting differences as comments; adds the org's `fm_addresses`, its phone onto the primary address, its `fm_email_addresses`; copies each project's `Locality`/`District`/`SPA`/`LASupervisorialDistrict` onto the program's address in its city (a city-only one when none); folds a merged-away project's code (`MergerWithID`) into the survivor; and reads agencies' own Rolodex cards (website, mission, addresses, and email when no contact person is named) onto their org, creating it by name when nothing answers. Driven by `fm:import_organizations` - `OrganizationServices::AutofillChange` — One thing a registrant's answers wrote onto an org: `field`, human `label`, the `value` that landed in it, `change_type` (`"new"` filled a blank / `"update"` replaced something), `previous_value` on an update, and an optional `scope` naming which work address. `change_type` is derived from `previous_value` so the two can't disagree, but is written to the JSON anyway so the stored row reads on its own. Round-trips through the `form_autofill_changes` column; `#description` renders "Website" or "ZIP on the Austin work address" for a flash, while the linking page shows the value and what it displaced. `#key` is (field, scope), which is what makes a later submission's value replace an earlier one instead of stacking. Drops a stored entry with no `field` rather than raising — it's a display aid, not a ledger. **Not a substitute for the Ahoy audit trail**, which records every before/after on Organization and Address independently; this is the denormalized, registration-scoped copy the linking page can render without correlating events - `OrganizationServices::AddressMatcher` — Given a submitted city/state/street/ZIP, finds the org address it corresponds to so `UpsertAddress`/`ProfileDiff` update the right one instead of duplicating. Tries same city + state (preferring one whose street also matches, so an org with several offices in one city doesn't get the wrong one rewritten), then same city + street (unifies an address with a submission that merely adds the state/country we were missing), then — across cities — same street + same ZIP, which reunites an address whose city was renamed ("Ventura" / "San Buenaventura") without merging two offices that share a street name in different towns. Every comparison goes through `AddressNormalizer`, so known spellings ("St. Louis" / "Saint Louis", "TX" / "Texas", "N Main St" / "North Main Street") match at every step. Module function, used by both address services - `OrganizationServices::UpsertAddress` — Find-or-create an organization's "work" address from a registrant's submitted organization fields (street/city/state/zip/country), matching via `AddressMatcher`; updates the match, else adds a new address; never demotes the org's existing primary (a registrant's address becomes primary only when the org has none yet). `overwrite: true` (default, public flow) replaces the matching address's details; `overwrite: false` (admin org-linking) fills only its blank fields so a conflicting saved address is kept and flagged instead. City and state are always fill-only (never flipped, in either mode), so a cross-city ZIP match keeps the saved city and lets `ProfileDiff` flag the difference; that fill still earns its keep because a legacy row can hold `""` in either column, and the street + ZIP match is the one path that reaches such a row to repair it. Returns a `Result` (`address`, `created`, and `changes` — an `AutofillChange` per field actually written, carrying the value) so callers can report what was really saved. Every field is trimmed before it's stored, so a padded answer isn't later read back as a difference from itself. Tolerates optional answers: a skipped street/ZIP is stored blank (both are `NOT NULL` columns), and `address` is nil — nothing saved — when there's no city, or no state and no existing address to update, rather than failing the whole registration. Shared by `PublicRegistration` and the admin org-linking actions diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index 7e5020af06..0104f259a9 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -3,11 +3,10 @@ module AffiliationServices # (ADR-0008), after the Rolodex and organizations passes have made both ends. Reads, in # order: # - # 1. Personnel (FmPersonnel) — a person staffing a project's art program. A facilitating - # role becomes a Facilitator affiliation, dated from when the row was entered to when - # the project closed; any more specific role ("Lead Facilitator", "Supervisor") also - # becomes a job affiliation with that title (ADR-0003 D1). Each carries the row's - # PrsnlRecID as its filemaker_code. + # 1. Personnel (FmPersonnel) — a person staffing a project's art program. A Leader + # becomes a Facilitator affiliation, dated from when the row was entered to when the + # project closed; a Liaison or Assistant becomes an undated job affiliation with that + # title (ADR-0003 D1). Each carries the row's PrsnlRecID as its filemaker_code. # 2. People-to-organizations (FmPeopleToOrganization) — a job at an organization, with # its title, carrying the row's RecordID. # 3. The Rolodex card itself (FmRolodex) — the organization its OrgID (or its typed @@ -22,17 +21,20 @@ module AffiliationServices # dates, keeping its own where FileMaker disagrees (noted in #differences and as a # comment on the affiliation). # - # None of these is minted from a training, so event_registration_id stays NULL and the - # training reconciliation leaves them alone unless an admin includes hand-made rows. + # None of these is minted from a training, so event_registration_id stays NULL: the + # reconcile page treats them as hand-entered rows. class ImportFromFilemaker include FilemakerValues OUTCOMES = %i[created linked already_imported skipped failed].freeze BATCH_SIZE = 1_000 - # Personnel Status values that mean the person still staffs the program. Anything else - # reads as ended — over-reporting active programs would skew program status. - ACTIVE_STATUSES = %w[active current].freeze - FACILITATING_ROLE = /facilitat/i + # Personnel Status values that mean the person still staffs the program. Anything else — + # Inactive, Pending, Suspended, a blank, or two values typed over each other — reads as + # ended, since over-reporting active programs would skew program status. + ACTIVE_STATUSES = %w[active].freeze + # FileMaker calls a facilitator a Leader. These other roles support the program without + # facilitating it, so they're jobs at the organization rather than facilitation. + JOB_ROLES = { "liaison" => "Liaison", "assistant" => "Assistant" }.freeze SOURCES = { FmPersonnel => "Personnel", FmPeopleToOrganization => "People-to-organizations", FmRolodex => "Rolodex" }.freeze attr_reader :results, :differences, :statuses, :roles @@ -96,21 +98,17 @@ def import_personnel(row) return record(:skipped, "#{label}: project #{data["ProjectID"].presence || "(none)"} wasn't imported") unless organization dates = personnel_dates(data, @projects[data["ProjectID"].to_s]) - titles(data["Role"]).each do |title| - attributes = title == Affiliation::FACILITATOR_TITLE ? dates : dates.except(:start_date) - people.each { |person| affiliate(person, organization, row, label, title:, attributes:, codes: @personnel_ids) } - end + title = title_for(data["Role"]) + attributes = title == Affiliation::FACILITATOR_TITLE ? dates : dates.except(:start_date) + people.each { |person| affiliate(person, organization, row, label, title:, attributes:, codes: @personnel_ids) } end - # Personnel are the people staffing the art program, so a row with no role is a - # facilitator. A role saying more than that is also a job title (ADR-0003 D1). - def titles(role) - role = clip(role.to_s.squish) - @roles[role.presence || "(blank)"] += 1 - return [ Affiliation::FACILITATOR_TITLE ] if role.blank? || role.casecmp?(Affiliation::FACILITATOR_TITLE) - return [ role ] unless role.match?(FACILITATING_ROLE) - - [ Affiliation::FACILITATOR_TITLE, role ] + # Nearly every Personnel row is a Leader, so a blank or unrecognized role is one too. + def title_for(role) + role = role.to_s.squish + title = JOB_ROLES.fetch(role.downcase, Affiliation::FACILITATOR_TITLE) + @roles["#{role.presence || "(blank)"} → #{title}"] += 1 + title end # Personnel rows carry no dates of their own: the assignment starts when it was entered diff --git a/app/services/organization_services/import_from_filemaker.rb b/app/services/organization_services/import_from_filemaker.rb index aa083113c1..944e9c8e7f 100644 --- a/app/services/organization_services/import_from_filemaker.rb +++ b/app/services/organization_services/import_from_filemaker.rb @@ -31,12 +31,17 @@ class ImportFromFilemaker OUTCOMES = %i[created linked already_imported skipped failed].freeze BATCH_SIZE = 500 - # FileMaker's OrgType / AgencyType words, matched loosely onto Organization::ORGANIZATION_TYPES. + # FileMaker's OrgType / AgencyType words onto Organization::ORGANIZATION_TYPES, first + # match winning — so a value naming two types ("501c3/Nonprofit Government Agency"), as + # FileMaker's multi-select stored them, takes the first. Any other word ("School", + # "Grassroots") is Other, with the word kept as its description. ORGANIZATION_TYPE_PATTERNS = { - /non.?profit|501|charit/i => "501c3/nonprofit", - /for.?profit|business|corporat|compan/i => "For-profit", - /\b(gov|government|county|city|state|federal|public|municipal)\b/i => "Government agency" + /501|non.?profit/i => "501c3/nonprofit", + /for.?profit|\bLLC\b/i => "For-profit", + /govt|government/i => "Government agency", + /\Aother\z/i => Organization::ORGANIZATION_TYPE_OTHER }.freeze + TYPE_FIELDS = %i[organization_type organization_type_other].freeze COMMENT_SOURCES = { FmOrganization => "FileMaker organization", FmProject => "FileMaker project", FmRolodex => "FileMaker Rolodex agency" }.freeze @@ -121,7 +126,9 @@ def import_single_program(fm_organization, project, umbrella, label) end organization = umbrella || program || Organization.new(name: unique_name(clip(fm_organization.data["OrgName"].to_s.squish), project.fm_id)) - attributes = organization_attributes(fm_organization.data).merge(project_attributes(project.data)) { |_key, mine, theirs| mine.presence || theirs } + organization_side = organization_attributes(fm_organization.data) + attributes = organization_side.merge(project_attributes(project.data)) { |_key, mine, theirs| mine.presence || theirs } + attributes.merge!(organization_side.slice(*TYPE_FIELDS)) if organization_side[:organization_type] save(organization, fm_organization, "#{label} + #{project.fm_id}", attributes, also: project) end @@ -273,7 +280,7 @@ def organization_attributes(data) website_url: clip(data["Website"]), description: data["Description"].to_s.strip.presence, mission_vision_values: clip(data["Mission"]), - organization_type: organization_type(data["OrgType"]), + **organization_type(data["OrgType"]), **archive_timestamps(data["CreationDate"], data["ModificationDateTime"]) } end @@ -281,7 +288,7 @@ def organization_attributes(data) def project_attributes(data) { description: data["Description"].to_s.strip.presence, - organization_type: organization_type(data["AgencyType"]), + **organization_type(data["AgencyType"]), **archive_timestamps(data["DateEntered"], data["DateTimeModified"]) } end @@ -295,11 +302,13 @@ def archive_timestamps(entered, modified) def organization_type(value) value = value.to_s.squish - return if value.blank? + return {} if value.blank? - ORGANIZATION_TYPE_PATTERNS.find { |pattern, _type| value.match?(pattern) }&.last.tap do |type| - @unmatched_types[value] += 1 unless type - end + type = ORGANIZATION_TYPE_PATTERNS.find { |pattern, _type| value.match?(pattern) }&.last + return { organization_type: type } if type + + @unmatched_types[value] += 1 + { organization_type: Organization::ORGANIZATION_TYPE_OTHER, organization_type_other: clip(value) } end # Staff edited both systems, so neither side is newer by rule: a blank takes the @@ -310,6 +319,8 @@ def fill(organization, attributes) timestamps.each { |attribute, value| organization[attribute] = value if organization.new_record? && value } fields.filter_map do |attribute, value| next if value.blank? + # The description only belongs to an Other type, which the type just settled. + next if attribute == :organization_type_other && organization.organization_type != Organization::ORGANIZATION_TYPE_OTHER if organization[attribute].blank? organization[attribute] = value diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 33af8921a6..a0b66b11c1 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -53,6 +53,12 @@ Constraints already decided elsewhere: - **A name match only reaches an org carrying no code from the same table.** One already holding another project's code is a namesake; the new org is named `"Name (P1234)"`, since org names are unique. + - **Organization type** comes from the FileMaker organization's `OrgType`, else the + project's `AgencyType`, first match winning: "501" or "nonprofit" → 501c3/nonprofit; + "for-profit" or "LLC" → For-profit; "government" or "govt" → Government agency; "Other" + → Other. Any other word ("School", "Educational Institution", "Individual", "Grassroots") + is Other with the word as its description. A value naming two types, as FileMaker's + multi-select stored them, takes the first. - Every FileMaker organization comes over, whether or not a project or person points at it (decided 2026-10-24). - **Agencies:** a Rolodex card flagged `isAgency__c` is the agency's own card. Its website, @@ -64,14 +70,15 @@ Constraints already decided elsewhere: between the person (by Rolodex code) and the project's org, carrying `PrsnlRecID` as its `filemaker_code` so each row imports once. `event_registration_id` stays NULL — these are historical, not training-minted. - - **Role:** a blank role or "Facilitator" is a Facilitator affiliation. A role naming more - ("Lead Facilitator") is a Facilitator affiliation plus a job affiliation with that title, - as registration mints them (ADR-0003 D1); any other role ("Supervisor") is a job - affiliation alone. + - **Role:** FileMaker calls a facilitator a **Leader** (8,112 of 8,907 rows). A Leader is a + Facilitator affiliation. A **Liaison** (611) or **Assistant** (232) supports the program + without facilitating it, so each is an undated job affiliation with that title + (ADR-0003 D1). A blank (80) or stray role reads as a Leader. - **Dates:** Personnel rows carry none of their own, so the assignment starts on - `DateEntered` and ends on the project's `CloseDate` (else `EndDate`). A `Status` outside - `ACTIVE_STATUSES` ends it now, undated (`inactive`). The run tallies every Status value - and how it was read, so the list can be corrected before the real run. + `DateEntered` and ends on the project's `CloseDate` (else `EndDate`). Only `Status` + "Active" (1,774) stays open; Inactive (7,220), Pending, Suspended, blank, and values + typed over each other ("ActiveInactive") end it now, undated (`inactive`). The run tallies + every Status and Role value and how it was read. - An existing same-title affiliation at that org is linked (takes the code, fills blank dates) unless it already records another Personnel row — a second stint stays its own row. - **People-to-organizations → job affiliation:** an `fm_people_to_organizations` row becomes @@ -113,10 +120,11 @@ the real run. - **Reconcile vs. history.** Still open. The reconcile page ends facilitator affiliations no attended training backs, hand-entered ones included, and imported personnel rows are hand-entered by that test. Exempt rows carrying a `PS…` code, or let reconcile end them? -- **Which `Status` values mean active** — provisionally "Active" and "Current" - (`AffiliationServices::ImportFromFilemaker::ACTIVE_STATUSES`); the dry run's tally is the - check. A closed project's `CloseDate` (else `EndDate`) ends every affiliation under it. - `AgreementStatus` isn't read. +- **Which `Status` values mean active** — only "Active", from the production values profiled + 2026-10-24 (see Personnel above). A closed project's `CloseDate` (else `EndDate`) ends every + affiliation under it. `AgreementStatus` isn't read: 8,376 of 8,907 rows leave it blank. +- **Is an Assistant a facilitator?** Read as a job for now, so assistants don't count toward + program status. - **Merged projects** (`MergerWithID`): folded — the merged-away project's code joins the survivor's org, so its personnel land there. One the portal already keeps as its own org is reported for a hand merge rather than merged automatically. diff --git a/lib/tasks/fm_import_organizations.rake b/lib/tasks/fm_import_organizations.rake index 7338737fc2..75a652ff4b 100644 --- a/lib/tasks/fm_import_organizations.rake +++ b/lib/tasks/fm_import_organizations.rake @@ -73,7 +73,7 @@ namespace :fm do end tallies = { - "Organization types no portal type answers to (left blank)" => import.try(:unmatched_types), + "Organization types read as Other, kept as its description" => import.try(:unmatched_types), "Personnel Status values and how they were read" => import.try(:statuses), "Personnel roles" => import.try(:roles) } diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index b7c399639b..2ba092b573 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -10,7 +10,7 @@ before { FmProject.create!(fm_id: "P0001", data: { "OrgID" => "ORG00001", "CloseDate" => "6/30/2019" }) } def personnel(fm_id, **data) - FmPersonnel.create!(fm_id:, data: { "PersonID" => "00006", "ProjectID" => "P0001", "Role" => "Facilitator", + FmPersonnel.create!(fm_id:, data: { "PersonID" => "00006", "ProjectID" => "P0001", "Role" => "Leader", "Status" => "Active", "DateEntered" => "3/1/2015" }.merge(data.stringify_keys)) end @@ -42,13 +42,16 @@ def import expect(import.statuses).to eq("Dropped → ended" => 1) end - it "also records a more specific facilitating role as a job title" do - personnel("PS0001", Role: "Lead Facilitator") + it "reads a Liaison or Assistant as an undated job, and any other role as a Leader" do + personnel("PS0001", Role: "Liaison") + personnel("PS0002", Role: "assistant") + personnel("PS0003", Role: "Leaders") - import + import = described_class.new.call - expect(person.affiliations.order(:title).pluck(:title, :start_date, :filemaker_code)) - .to eq([ [ "Facilitator", Date.new(2015, 3, 1), "PS0001" ], [ "Lead Facilitator", nil, "PS0001" ] ]) + expect(person.affiliations.order(:filemaker_code).pluck(:filemaker_code, :title, :start_date)) + .to eq([ [ "PS0001", "Liaison", nil ], [ "PS0002", "Assistant", nil ], [ "PS0003", "Facilitator", Date.new(2015, 3, 1) ] ]) + expect(import.roles).to eq("Liaison → Liaison" => 1, "assistant → Assistant" => 1, "Leaders → Facilitator" => 1) end it "links the Facilitator affiliation the person already has there, filling its blank start" do @@ -124,7 +127,8 @@ def import end it "changes nothing when it runs again" do - personnel("PS0001", Role: "Lead Facilitator") + personnel("PS0001") + personnel("PS0002", Role: "Liaison") FmPeopleToOrganization.create!(fm_id: "12", data: { "PersonID" => "00006", "OrgID" => "ORG00001", "Title" => "Board Chair" }) FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", "AddrsState" => "CA" }) rolodex(OrgID: "ORG00001", Title: "Counselor", WorksiteAddrsID: "40") diff --git a/spec/services/organization_services/import_from_filemaker_spec.rb b/spec/services/organization_services/import_from_filemaker_spec.rb index 5c8ab23004..0ff1fce4fa 100644 --- a/spec/services/organization_services/import_from_filemaker_spec.rb +++ b/spec/services/organization_services/import_from_filemaker_spec.rb @@ -36,6 +36,46 @@ def import la_service_planning_area: 3, la_supervisorial_district: 5) end + describe "organization type" do + { + "501c3/Nonprofit" => [ "501c3/nonprofit", nil ], + "Non-Profit Organization" => [ "501c3/nonprofit", nil ], + "501c3/Nonprofit Government Agency" => [ "501c3/nonprofit", nil ], + "For-Profit" => [ "For-profit", nil ], + "LLC" => [ "For-profit", nil ], + "County Government Agency" => [ "Government agency", nil ], + "Govt agency" => [ "Government agency", nil ], + "Other" => [ "Other", nil ], + "School" => [ "Other", "School" ] + }.each do |fm_type, (type, other)| + it "reads #{fm_type.inspect} as #{type}#{" (#{other})" if other}" do + fm_organization("ORG00001", OrgType: fm_type) + + import + + expect(Organization.sole).to have_attributes(organization_type: type, organization_type_other: other) + end + end + + it "takes the organization's type over its single project's" do + fm_organization("ORG00001", OrgType: "Government Agency") + project("P0001", "ORG00001", AgencyType: "School") + + import + + expect(Organization.sole).to have_attributes(organization_type: "Government agency", organization_type_other: nil) + end + + it "leaves the description off when the portal's type isn't Other" do + existing = create(:organization, filemaker_code: "ORG00001", organization_type: "For-profit") + fm_organization("ORG00001", OrgType: "School") + + import + + expect(existing.reload).to have_attributes(organization_type: "For-profit", organization_type_other: nil) + end + end + it "nests each of an organization's several projects under it" do fm_organization("ORG00002", OrgName: "Valley Services") project("P0002", "ORG00002", ProjectName: "Valley Adult Program", City: "Fresno", State: "CA", Locality: "Central CA") diff --git a/spec/tasks/fm_import_organizations_spec.rb b/spec/tasks/fm_import_organizations_spec.rb index f95874497f..e887059ab5 100644 --- a/spec/tasks/fm_import_organizations_spec.rb +++ b/spec/tasks/fm_import_organizations_spec.rb @@ -13,7 +13,7 @@ before do FmOrganization.create!(fm_id: "ORG00001", data: { "OrgName" => "Harbor House" }) FmProject.create!(fm_id: "P0001", data: { "OrgID" => "ORG00001", "ProjectName" => "Harbor House Art Program" }) - FmPersonnel.create!(fm_id: "PS0001", data: { "PersonID" => "00200", "ProjectID" => "P0001", "Role" => "Facilitator", "Status" => "Active" }) + FmPersonnel.create!(fm_id: "PS0001", data: { "PersonID" => "00200", "ProjectID" => "P0001", "Role" => "Leader", "Status" => "Active" }) end def run_task(dry_run: true, task: "fm:import_organizations", run: nil) From e299b66eeaaea24d3f1134e4170807b836c0a084 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 08:19:46 -0400 Subject: [PATCH 05/25] Import a FileMaker Leader as a primary-contact Facilitator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prod's affiliations already record the old app's leaders this way — the January 2026 position-to-title conversion and #2321 folded Leader into a Facilitator who is the program's primary contact — so FileMaker's leaders should arrive looking the same. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- .../affiliation_services/import_from_filemaker.rb | 11 ++++++----- ...008-filemaker-organization-and-personnel-import.md | 5 ++++- .../import_from_filemaker_spec.rb | 8 ++++---- 4 files changed, 15 insertions(+), 11 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2cab4fa071..e5d9c9e397 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -332,7 +332,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Leader (or blank/stray role) becomes a Facilitator affiliation on the project's org (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), a Liaison or Assistant (`JOB_ROLES`) an undated job affiliation; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Leader (or blank/stray role) becomes a primary-contact Facilitator affiliation on the project's org (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), a Liaison or Assistant (`JOB_ROLES`) an undated job affiliation; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index 0104f259a9..c798c642f1 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -4,8 +4,8 @@ module AffiliationServices # order: # # 1. Personnel (FmPersonnel) — a person staffing a project's art program. A Leader - # becomes a Facilitator affiliation, dated from when the row was entered to when the - # project closed; a Liaison or Assistant becomes an undated job affiliation with that + # becomes a Facilitator affiliation marked primary contact, dated from when the row + # was entered to when the project closed; a Liaison or Assistant becomes an undated job affiliation with that # title (ADR-0003 D1). Each carries the row's PrsnlRecID as its filemaker_code. # 2. People-to-organizations (FmPeopleToOrganization) — a job at an organization, with # its title, carrying the row's RecordID. @@ -32,8 +32,9 @@ class ImportFromFilemaker # Inactive, Pending, Suspended, a blank, or two values typed over each other — reads as # ended, since over-reporting active programs would skew program status. ACTIVE_STATUSES = %w[active].freeze - # FileMaker calls a facilitator a Leader. These other roles support the program without - # facilitating it, so they're jobs at the organization rather than facilitation. + # FileMaker calls a facilitator a Leader, which the portal records as a Facilitator who is + # the program's primary contact (the pattern prod's January 2026 conversion and #2321 set). + # These other roles support the program without facilitating it, so they're jobs there. JOB_ROLES = { "liaison" => "Liaison", "assistant" => "Assistant" }.freeze SOURCES = { FmPersonnel => "Personnel", FmPeopleToOrganization => "People-to-organizations", FmRolodex => "Rolodex" }.freeze @@ -99,7 +100,7 @@ def import_personnel(row) dates = personnel_dates(data, @projects[data["ProjectID"].to_s]) title = title_for(data["Role"]) - attributes = title == Affiliation::FACILITATOR_TITLE ? dates : dates.except(:start_date) + attributes = title == Affiliation::FACILITATOR_TITLE ? dates.merge(primary_contact: true) : dates.except(:start_date) people.each { |person| affiliate(person, organization, row, label, title:, attributes:, codes: @personnel_ids) } end diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index a0b66b11c1..e54d60d7a1 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -71,7 +71,10 @@ Constraints already decided elsewhere: `filemaker_code` so each row imports once. `event_registration_id` stays NULL — these are historical, not training-minted. - **Role:** FileMaker calls a facilitator a **Leader** (8,112 of 8,907 rows). A Leader is a - Facilitator affiliation. A **Liaison** (611) or **Assistant** (232) supports the program + Facilitator affiliation marked `primary_contact` — the pattern prod's affiliations already + follow: the January 2026 migration (`ConvertProjectUserPositionToTitle`, in #735, deleted + once applied) turned the old app's position enum (liaison/leader/assistant) into titles, + and #2321 folded Leader into Facilitator + primary contact. A **Liaison** (611) or **Assistant** (232) supports the program without facilitating it, so each is an undated job affiliation with that title (ADR-0003 D1). A blank (80) or stray role reads as a Leader. - **Dates:** Personnel rows carry none of their own, so the assignment starts on diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index 2ba092b573..a30af53f89 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -23,11 +23,11 @@ def import end describe "personnel" do - it "makes a Facilitator affiliation dated from entry to the project's close" do + it "makes a Leader a primary-contact Facilitator dated from entry to the project's close" do personnel("PS0001") expect(import.results[:created].size).to eq(1) - expect(person.affiliations.sole).to have_attributes(organization: program, title: "Facilitator", filemaker_code: "PS0001", + expect(person.affiliations.sole).to have_attributes(organization: program, title: "Facilitator", filemaker_code: "PS0001", primary_contact: true, start_date: Date.new(2015, 3, 1), end_date: Date.new(2019, 6, 30), event_registration_id: nil) end @@ -49,8 +49,8 @@ def import import = described_class.new.call - expect(person.affiliations.order(:filemaker_code).pluck(:filemaker_code, :title, :start_date)) - .to eq([ [ "PS0001", "Liaison", nil ], [ "PS0002", "Assistant", nil ], [ "PS0003", "Facilitator", Date.new(2015, 3, 1) ] ]) + expect(person.affiliations.order(:filemaker_code).pluck(:filemaker_code, :title, :start_date, :primary_contact)) + .to eq([ [ "PS0001", "Liaison", nil, false ], [ "PS0002", "Assistant", nil, false ], [ "PS0003", "Facilitator", Date.new(2015, 3, 1), true ] ]) expect(import.roles).to eq("Liaison → Liaison" => 1, "assistant → Assistant" => 1, "Leaders → Facilitator" => 1) end From 80c4e978f6b44c06eb991251dbb9c36d4d2661b0 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 08:23:33 -0400 Subject: [PATCH 06/25] Give FileMaker liaisons and assistants prod's January shape Prod's existing affiliations show every Liaison as a primary contact, and the January 2026 conversion gave each liaison or assistant a Facilitator affiliation over the same dates beside their role. FileMaker's rows now arrive the same way, so the two systems' people read alike. A row prod still titles Leader is the same facilitation, so it links rather than gaining a Facilitator twin. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- .../import_from_filemaker.rb | 47 ++++++++++++++----- ...maker-organization-and-personnel-import.md | 14 ++++-- .../import_from_filemaker_spec.rb | 41 +++++++++++++--- 4 files changed, 79 insertions(+), 25 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e5d9c9e397..f519f20f92 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -332,7 +332,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Leader (or blank/stray role) becomes a primary-contact Facilitator affiliation on the project's org (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), a Liaison or Assistant (`JOB_ROLES`) an undated job affiliation; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Leader (or blank/stray role) becomes a primary-contact Facilitator affiliation on the project's org (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), a Liaison (primary contact) or Assistant (`JOB_ROLES`) an affiliation with that title plus a Facilitator one over the same dates unless one already exists (the January 2026 pattern); an existing "Leader"-titled row matches as a Facilitator; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index c798c642f1..bbc5a70e41 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -5,8 +5,9 @@ module AffiliationServices # # 1. Personnel (FmPersonnel) — a person staffing a project's art program. A Leader # becomes a Facilitator affiliation marked primary contact, dated from when the row - # was entered to when the project closed; a Liaison or Assistant becomes an undated job affiliation with that - # title (ADR-0003 D1). Each carries the row's PrsnlRecID as its filemaker_code. + # was entered to when the project closed. A Liaison (also a primary contact) or Assistant becomes an affiliation + # with that title over the same dates, plus a Facilitator affiliation beside it unless + # one with those dates already exists. Each carries the row's PrsnlRecID as its filemaker_code. # 2. People-to-organizations (FmPeopleToOrganization) — a job at an organization, with # its title, carrying the row's RecordID. # 3. The Rolodex card itself (FmRolodex) — the organization its OrgID (or its typed @@ -33,9 +34,12 @@ class ImportFromFilemaker # ended, since over-reporting active programs would skew program status. ACTIVE_STATUSES = %w[active].freeze # FileMaker calls a facilitator a Leader, which the portal records as a Facilitator who is - # the program's primary contact (the pattern prod's January 2026 conversion and #2321 set). - # These other roles support the program without facilitating it, so they're jobs there. - JOB_ROLES = { "liaison" => "Liaison", "assistant" => "Assistant" }.freeze + # a primary contact; prod's Liaisons are primary contacts too. A Liaison or Assistant + # title isn't itself facilitation, so it never counts toward program status. + JOB_ROLES = { "liaison" => { title: "Liaison", primary_contact: true }, + "assistant" => { title: "Assistant", primary_contact: false } }.freeze + LEADER = { title: Affiliation::FACILITATOR_TITLE, primary_contact: true }.freeze + LEGACY_LEADER_TITLE = "Leader" SOURCES = { FmPersonnel => "Personnel", FmPeopleToOrganization => "People-to-organizations", FmRolodex => "Rolodex" }.freeze attr_reader :results, :differences, :statuses, :roles @@ -99,17 +103,32 @@ def import_personnel(row) return record(:skipped, "#{label}: project #{data["ProjectID"].presence || "(none)"} wasn't imported") unless organization dates = personnel_dates(data, @projects[data["ProjectID"].to_s]) - title = title_for(data["Role"]) - attributes = title == Affiliation::FACILITATOR_TITLE ? dates.merge(primary_contact: true) : dates.except(:start_date) - people.each { |person| affiliate(person, organization, row, label, title:, attributes:, codes: @personnel_ids) } + role = role_for(data["Role"]) + people.each do |person| + affiliate(person, organization, row, label, title: role[:title], attributes: dates.merge(primary_contact: role[:primary_contact]), codes: @personnel_ids) + next if role == LEADER || facilitator_for?(person, organization, dates) + + create(person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, + attributes: dates.merge(primary_contact: false), code: row.fm_id) + end + end + + # A Liaison or Assistant also facilitated over the same stretch, as prod's January 2026 + # conversion recorded them: a Facilitator affiliation beside the role, unless one with the + # same dates is already there. + def facilitator_for?(person, organization, dates) + person.affiliations.where(organization:).any? do |affiliation| + same_title?(affiliation.title, Affiliation::FACILITATOR_TITLE) && + affiliation.start_date == dates[:start_date] && affiliation.end_date == dates[:end_date] + end end # Nearly every Personnel row is a Leader, so a blank or unrecognized role is one too. - def title_for(role) + def role_for(role) role = role.to_s.squish - title = JOB_ROLES.fetch(role.downcase, Affiliation::FACILITATOR_TITLE) - @roles["#{role.presence || "(blank)"} → #{title}"] += 1 - title + mapped = JOB_ROLES.fetch(role.downcase, LEADER) + @roles["#{role.presence || "(blank)"} → #{mapped[:title]}"] += 1 + mapped end # Personnel rows carry no dates of their own: the assignment starts when it was entered @@ -185,8 +204,10 @@ def records_another_row?(affiliation, codes) codes && affiliation.filemaker_codes.any? { |code| codes.include?(code) } end + # Some prod rows still carry the old app's "Leader" title, which is a Facilitator. def same_title?(held, title) - held.to_s.squish.casecmp?(title.to_s.squish) + held = held.to_s.squish + held.casecmp?(title.to_s.squish) || (title == Affiliation::FACILITATOR_TITLE && held.casecmp?(LEGACY_LEADER_TITLE)) end def link(affiliation, row, label, title:, attributes:, codes:) diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index e54d60d7a1..643d40d5c7 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -74,9 +74,13 @@ Constraints already decided elsewhere: Facilitator affiliation marked `primary_contact` — the pattern prod's affiliations already follow: the January 2026 migration (`ConvertProjectUserPositionToTitle`, in #735, deleted once applied) turned the old app's position enum (liaison/leader/assistant) into titles, - and #2321 folded Leader into Facilitator + primary contact. A **Liaison** (611) or **Assistant** (232) supports the program - without facilitating it, so each is an undated job affiliation with that title - (ADR-0003 D1). A blank (80) or stray role reads as a Leader. + and #2321 folded Leader into Facilitator + primary contact. Prod, profiled 2026-10-24: 1,829 + Facilitators are primary contacts (402 aren't), and all 53 Liaisons are; no Assistant rows + exist, so an Assistant has no precedent and isn't one. 111 rows are still titled "Leader", + which the exact-match Facilitator scope doesn't count. A **Liaison** (611) or **Assistant** (232) gets an affiliation with + that title over the same dates (a Liaison is also a primary contact) — the title itself + never counts as facilitation — plus a Facilitator affiliation over those dates unless the + person already has one, as prod's January 2026 conversion did. A blank (80) or stray role reads as a Leader. - **Dates:** Personnel rows carry none of their own, so the assignment starts on `DateEntered` and ends on the project's `CloseDate` (else `EndDate`). Only `Status` "Active" (1,774) stays open; Inactive (7,220), Pending, Suspended, blank, and values @@ -126,8 +130,8 @@ the real run. - **Which `Status` values mean active** — only "Active", from the production values profiled 2026-10-24 (see Personnel above). A closed project's `CloseDate` (else `EndDate`) ends every affiliation under it. `AgreementStatus` isn't read: 8,376 of 8,907 rows leave it blank. -- **Is an Assistant a facilitator?** Read as a job for now, so assistants don't count toward - program status. +- **Is an Assistant a facilitator?** The Assistant title isn't; the Facilitator affiliation + minted beside it is, matching January 2026 (decided 2026-10-24). - **Merged projects** (`MergerWithID`): folded — the merged-away project's code joins the survivor's org, so its personnel land there. One the portal already keeps as its own org is reported for a hand merge rather than merged automatically. diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index a30af53f89..996935b6f3 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -42,16 +42,35 @@ def import expect(import.statuses).to eq("Dropped → ended" => 1) end - it "reads a Liaison or Assistant as an undated job, and any other role as a Leader" do + it "gives a Liaison or Assistant their role plus a Facilitator affiliation over the same dates" do personnel("PS0001", Role: "Liaison") - personnel("PS0002", Role: "assistant") - personnel("PS0003", Role: "Leaders") + personnel("PS0002", Role: "assistant", DateEntered: "1/5/2018") import = described_class.new.call - expect(person.affiliations.order(:filemaker_code).pluck(:filemaker_code, :title, :start_date, :primary_contact)) - .to eq([ [ "PS0001", "Liaison", nil, false ], [ "PS0002", "Assistant", nil, false ], [ "PS0003", "Facilitator", Date.new(2015, 3, 1), true ] ]) - expect(import.roles).to eq("Liaison → Liaison" => 1, "assistant → Assistant" => 1, "Leaders → Facilitator" => 1) + expect(person.affiliations.order(:start_date, :title).pluck(:filemaker_code, :title, :start_date, :primary_contact)) + .to eq([ [ "PS0001", "Facilitator", Date.new(2015, 3, 1), false ], [ "PS0001", "Liaison", Date.new(2015, 3, 1), true ], + [ "PS0002", "Assistant", Date.new(2018, 1, 5), false ], [ "PS0002", "Facilitator", Date.new(2018, 1, 5), false ] ]) + expect(import.roles).to eq("Liaison → Liaison" => 1, "assistant → Assistant" => 1) + end + + it "adds no second Facilitator when one with the same dates is already there" do + create(:affiliation, person:, organization: program, title: "Facilitator", start_date: Date.new(2015, 3, 1), end_date: Date.new(2019, 6, 30)) + personnel("PS0001", Role: "Liaison") + + import + + expect(person.affiliations.pluck(:title)).to contain_exactly("Facilitator", "Liaison") + end + + it "reads a blank or unrecognized role as a Leader" do + personnel("PS0001", Role: "Leaders") + personnel("PS0002", Role: "", DateEntered: "1/5/2018") + + import = described_class.new.call + + expect(person.affiliations.pluck(:title).uniq).to eq([ "Facilitator" ]) + expect(import.roles).to eq("Leaders → Facilitator" => 1, "(blank) → Facilitator" => 1) end it "links the Facilitator affiliation the person already has there, filling its blank start" do @@ -63,6 +82,16 @@ def import expect(person.affiliations.count).to eq(1) end + it "links a row the old app titled Leader instead of adding a Facilitator beside it" do + existing = create(:affiliation, person:, organization: program, title: "Leader", primary_contact: true) + personnel("PS0001") + + import + + expect(person.affiliations.sole).to eq(existing) + expect(existing.reload).to have_attributes(title: "Leader", filemaker_code: "PS0001") + end + it "keeps a second stint on the same project as its own affiliation" do personnel("PS0001") personnel("PS0002", DateEntered: "1/5/2018") From 2c1b67eab8d03b2fd6745a4fe6d71c366647face Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 08:25:14 -0400 Subject: [PATCH 07/25] Import FileMaker liaisons as primary-contact facilitators, noting the role MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prod records a liaison the way it records a leader — a Facilitator who is a primary contact — so FileMaker's liaisons arrive that way too. Folding both into Facilitator would lose the word FileMaker used, so each affiliation gets a comment saying how FileMaker listed the person at import. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- .../import_from_filemaker.rb | 58 ++++++++++++------- ...maker-organization-and-personnel-import.md | 24 ++++---- .../import_from_filemaker_spec.rb | 34 ++++++++--- 4 files changed, 76 insertions(+), 42 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f519f20f92..53b8a71746 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -332,7 +332,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Leader (or blank/stray role) becomes a primary-contact Facilitator affiliation on the project's org (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), a Liaison (primary contact) or Assistant (`JOB_ROLES`) an affiliation with that title plus a Facilitator one over the same dates unless one already exists (the January 2026 pattern); an existing "Leader"-titled row matches as a Facilitator; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Leader or Liaison (or blank/stray role) becomes a primary-contact Facilitator affiliation on the project's org, with a "FileMaker import" comment naming the role (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an Assistant (`JOB_ROLES`) an Assistant affiliation plus a Facilitator one over the same dates unless one already exists (the January 2026 pattern); an existing "Leader"-titled row matches as a Facilitator; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index bbc5a70e41..e7cec74bb6 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -3,11 +3,12 @@ module AffiliationServices # (ADR-0008), after the Rolodex and organizations passes have made both ends. Reads, in # order: # - # 1. Personnel (FmPersonnel) — a person staffing a project's art program. A Leader - # becomes a Facilitator affiliation marked primary contact, dated from when the row - # was entered to when the project closed. A Liaison (also a primary contact) or Assistant becomes an affiliation - # with that title over the same dates, plus a Facilitator affiliation beside it unless - # one with those dates already exists. Each carries the row's PrsnlRecID as its filemaker_code. + # 1. Personnel (FmPersonnel) — a person staffing a project's art program. A Leader or + # Liaison becomes a Facilitator affiliation marked primary contact, dated from when the + # row was entered to when the project closed, with a comment naming the role. An + # Assistant becomes an Assistant affiliation over the same dates, plus a Facilitator + # affiliation beside it unless one with those dates already exists. Each carries the + # row's PrsnlRecID as its filemaker_code. # 2. People-to-organizations (FmPeopleToOrganization) — a job at an organization, with # its title, carrying the row's RecordID. # 3. The Rolodex card itself (FmRolodex) — the organization its OrgID (or its typed @@ -33,13 +34,11 @@ class ImportFromFilemaker # Inactive, Pending, Suspended, a blank, or two values typed over each other — reads as # ended, since over-reporting active programs would skew program status. ACTIVE_STATUSES = %w[active].freeze - # FileMaker calls a facilitator a Leader, which the portal records as a Facilitator who is - # a primary contact; prod's Liaisons are primary contacts too. A Liaison or Assistant - # title isn't itself facilitation, so it never counts toward program status. - JOB_ROLES = { "liaison" => { title: "Liaison", primary_contact: true }, - "assistant" => { title: "Assistant", primary_contact: false } }.freeze - LEADER = { title: Affiliation::FACILITATOR_TITLE, primary_contact: true }.freeze + # A Leader or Liaison is a Facilitator who is a primary contact, the shape prod already + # gives them. An Assistant keeps the title, which isn't itself facilitation. + JOB_ROLES = { "assistant" => "Assistant" }.freeze LEGACY_LEADER_TITLE = "Leader" + ROLE_COMMENT_TOPIC = "FileMaker import" SOURCES = { FmPersonnel => "Personnel", FmPeopleToOrganization => "People-to-organizations", FmRolodex => "Rolodex" }.freeze attr_reader :results, :differences, :statuses, :roles @@ -103,19 +102,33 @@ def import_personnel(row) return record(:skipped, "#{label}: project #{data["ProjectID"].presence || "(none)"} wasn't imported") unless organization dates = personnel_dates(data, @projects[data["ProjectID"].to_s]) - role = role_for(data["Role"]) + role = data["Role"].to_s.squish + title = title_for(role) people.each do |person| - affiliate(person, organization, row, label, title: role[:title], attributes: dates.merge(primary_contact: role[:primary_contact]), codes: @personnel_ids) - next if role == LEADER || facilitator_for?(person, organization, dates) + if title == Affiliation::FACILITATOR_TITLE + affiliation = affiliate(person, organization, row, label, title:, attributes: dates.merge(primary_contact: true), codes: @personnel_ids) + note_role(affiliation, row, role) if affiliation && role.present? + next + end + + affiliate(person, organization, row, label, title:, attributes: dates.merge(primary_contact: false), codes: @personnel_ids) + next if facilitator_for?(person, organization, dates) create(person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, attributes: dates.merge(primary_contact: false), code: row.fm_id) end end - # A Liaison or Assistant also facilitated over the same stretch, as prod's January 2026 - # conversion recorded them: a Facilitator affiliation beside the role, unless one with the - # same dates is already there. + # Folding Leader and Liaison into Facilitator drops the word FileMaker used, so the + # affiliation keeps it. + def note_role(affiliation, row, role) + affiliation.comments.create!(topic: ROLE_COMMENT_TOPIC, + body: "Marked as #{role} in FileMaker (Personnel #{row.fm_id}) at the time of the FileMaker import.") + end + + # An Assistant also facilitated over the same stretch, as prod's January 2026 conversion + # recorded them: a Facilitator affiliation beside the role, unless one with the same + # dates is already there. def facilitator_for?(person, organization, dates) person.affiliations.where(organization:).any? do |affiliation| same_title?(affiliation.title, Affiliation::FACILITATOR_TITLE) && @@ -124,11 +137,10 @@ def facilitator_for?(person, organization, dates) end # Nearly every Personnel row is a Leader, so a blank or unrecognized role is one too. - def role_for(role) - role = role.to_s.squish - mapped = JOB_ROLES.fetch(role.downcase, LEADER) - @roles["#{role.presence || "(blank)"} → #{mapped[:title]}"] += 1 - mapped + def title_for(role) + title = JOB_ROLES.fetch(role.downcase, Affiliation::FACILITATOR_TITLE) + @roles["#{role.presence || "(blank)"} → #{title}"] += 1 + title end # Personnel rows carry no dates of their own: the assignment starts when it was entered @@ -222,6 +234,7 @@ def link(affiliation, row, label, title:, attributes:, codes:) @affiliations.add(row.fm_id, affiliation.id) if codes note_differences(affiliation, row, label, kept) if kept.any? record(:linked, "#{label} → affiliation ##{affiliation.id}") + affiliation end # A blank date takes the archive's. The archive's status only ends an affiliation the @@ -257,6 +270,7 @@ def create(person, organization, row, label, title:, attributes:, code:) @affiliations.add(code, affiliation.id) if code record(:created, "#{label} → affiliation ##{affiliation.id}#{" (#{title})" if title}") + affiliation end def updated_by(data) diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 643d40d5c7..7a60465f0f 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -70,17 +70,19 @@ Constraints already decided elsewhere: between the person (by Rolodex code) and the project's org, carrying `PrsnlRecID` as its `filemaker_code` so each row imports once. `event_registration_id` stays NULL — these are historical, not training-minted. - - **Role:** FileMaker calls a facilitator a **Leader** (8,112 of 8,907 rows). A Leader is a - Facilitator affiliation marked `primary_contact` — the pattern prod's affiliations already - follow: the January 2026 migration (`ConvertProjectUserPositionToTitle`, in #735, deleted - once applied) turned the old app's position enum (liaison/leader/assistant) into titles, - and #2321 folded Leader into Facilitator + primary contact. Prod, profiled 2026-10-24: 1,829 - Facilitators are primary contacts (402 aren't), and all 53 Liaisons are; no Assistant rows - exist, so an Assistant has no precedent and isn't one. 111 rows are still titled "Leader", - which the exact-match Facilitator scope doesn't count. A **Liaison** (611) or **Assistant** (232) gets an affiliation with - that title over the same dates (a Liaison is also a primary contact) — the title itself - never counts as facilitation — plus a Facilitator affiliation over those dates unless the - person already has one, as prod's January 2026 conversion did. A blank (80) or stray role reads as a Leader. + - **Role:** matches how prod already records the old app's roles. The January 2026 + migration (`ConvertProjectUserPositionToTitle`, in #735, deleted once applied) turned that + app's position enum (liaison/leader/assistant) into titles, and #2321 folded Leader into + Facilitator + primary contact. Prod, profiled 2026-10-24: 1,829 Facilitators are primary + contacts (402 aren't), all 53 Liaisons are, and 111 rows are still titled "Leader" (which + the exact-match Facilitator scope doesn't count, and which the import matches as a + Facilitator rather than duplicating). + - **Leader** (8,112 of 8,907 rows) and **Liaison** (611) → a Facilitator affiliation marked + `primary_contact`, with a "FileMaker import" comment recording the role FileMaker gave. + - **Assistant** (232) → an Assistant affiliation over the same dates — the title itself + never counts as facilitation — plus a Facilitator affiliation over those dates unless the + person already has one, as the January 2026 conversion did. + - A blank (80) or stray role reads as a Leader. - **Dates:** Personnel rows carry none of their own, so the assignment starts on `DateEntered` and ends on the project's `CloseDate` (else `EndDate`). Only `Status` "Active" (1,774) stays open; Inactive (7,220), Pending, Suspended, blank, and values diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index 996935b6f3..281e73298f 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -42,25 +42,43 @@ def import expect(import.statuses).to eq("Dropped → ended" => 1) end - it "gives a Liaison or Assistant their role plus a Facilitator affiliation over the same dates" do + it "makes a Liaison a primary-contact Facilitator, noting the role FileMaker gave" do personnel("PS0001", Role: "Liaison") + + import = described_class.new.call + + affiliation = person.affiliations.sole + expect(affiliation).to have_attributes(title: "Facilitator", primary_contact: true, start_date: Date.new(2015, 3, 1)) + expect(affiliation.comments.sole).to have_attributes(topic: "FileMaker import", body: a_string_including("Liaison", "PS0001")) + expect(import.roles).to eq("Liaison → Facilitator" => 1) + end + + it "notes a Leader's role too, but not a blank one" do + personnel("PS0001") + personnel("PS0002", Role: "", DateEntered: "1/5/2018") + + import + + expect(Comment.where(commentable: person.affiliations).pluck(:body)).to contain_exactly(a_string_including("Leader", "PS0001")) + end + + it "gives an Assistant their role plus a Facilitator affiliation over the same dates" do personnel("PS0002", Role: "assistant", DateEntered: "1/5/2018") import = described_class.new.call - expect(person.affiliations.order(:start_date, :title).pluck(:filemaker_code, :title, :start_date, :primary_contact)) - .to eq([ [ "PS0001", "Facilitator", Date.new(2015, 3, 1), false ], [ "PS0001", "Liaison", Date.new(2015, 3, 1), true ], - [ "PS0002", "Assistant", Date.new(2018, 1, 5), false ], [ "PS0002", "Facilitator", Date.new(2018, 1, 5), false ] ]) - expect(import.roles).to eq("Liaison → Liaison" => 1, "assistant → Assistant" => 1) + expect(person.affiliations.order(:title).pluck(:filemaker_code, :title, :start_date, :primary_contact)) + .to eq([ [ "PS0002", "Assistant", Date.new(2018, 1, 5), false ], [ "PS0002", "Facilitator", Date.new(2018, 1, 5), false ] ]) + expect(import.roles).to eq("assistant → Assistant" => 1) end - it "adds no second Facilitator when one with the same dates is already there" do + it "adds no second Facilitator beside an Assistant when one with the same dates is already there" do create(:affiliation, person:, organization: program, title: "Facilitator", start_date: Date.new(2015, 3, 1), end_date: Date.new(2019, 6, 30)) - personnel("PS0001", Role: "Liaison") + personnel("PS0001", Role: "Assistant") import - expect(person.affiliations.pluck(:title)).to contain_exactly("Facilitator", "Liaison") + expect(person.affiliations.pluck(:title)).to contain_exactly("Facilitator", "Assistant") end it "reads a blank or unrecognized role as a Leader" do From e8bf430624eef3f6133fd5d96a21a5054ed6c08a Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 08:31:21 -0400 Subject: [PATCH 08/25] Import every FileMaker Personnel role as a Facilitator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Assistants are an AWBW program role, not a job title, and prod holds no Assistant-titled rows — the old app's assistants became Facilitators who aren't primary contacts. FileMaker's arrive the same way, with the role kept in a comment like leaders' and liaisons'. A leader or liaison matching a Facilitator row the portal already had now turns its primary contact on, saying so in that comment, so the existing row ends up the shape a fresh import would have made. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- .../import_from_filemaker.rb | 59 +++++++------------ ...maker-organization-and-personnel-import.md | 16 ++--- .../import_from_filemaker_spec.rb | 29 ++++++--- 4 files changed, 52 insertions(+), 54 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 53b8a71746..9de54e1745 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -332,7 +332,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → a Leader or Liaison (or blank/stray role) becomes a primary-contact Facilitator affiliation on the project's org, with a "FileMaker import" comment naming the role (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an Assistant (`JOB_ROLES`) an Assistant affiliation plus a Facilitator one over the same dates unless one already exists (the January 2026 pattern); an existing "Leader"-titled row matches as a Facilitator; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — a Leader or Liaison (or blank/stray role) as a primary contact, turning it on for a matched existing row, an Assistant (`NON_PRIMARY_ROLES`) not (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing "Leader"-titled row matches as a Facilitator; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index e7cec74bb6..8c8334d439 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -3,12 +3,11 @@ module AffiliationServices # (ADR-0008), after the Rolodex and organizations passes have made both ends. Reads, in # order: # - # 1. Personnel (FmPersonnel) — a person staffing a project's art program. A Leader or - # Liaison becomes a Facilitator affiliation marked primary contact, dated from when the - # row was entered to when the project closed, with a comment naming the role. An - # Assistant becomes an Assistant affiliation over the same dates, plus a Facilitator - # affiliation beside it unless one with those dates already exists. Each carries the - # row's PrsnlRecID as its filemaker_code. + # 1. Personnel (FmPersonnel) — a person staffing a project's art program becomes a + # Facilitator affiliation, dated from when the row was entered to when the project + # closed, with a comment naming the role FileMaker gave. A Leader or Liaison is a + # primary contact — turned on for a Facilitator row the portal already had — and an + # Assistant isn't. Each carries the row's PrsnlRecID as its filemaker_code. # 2. People-to-organizations (FmPeopleToOrganization) — a job at an organization, with # its title, carrying the row's RecordID. # 3. The Rolodex card itself (FmRolodex) — the organization its OrgID (or its typed @@ -34,9 +33,9 @@ class ImportFromFilemaker # Inactive, Pending, Suspended, a blank, or two values typed over each other — reads as # ended, since over-reporting active programs would skew program status. ACTIVE_STATUSES = %w[active].freeze - # A Leader or Liaison is a Facilitator who is a primary contact, the shape prod already - # gives them. An Assistant keeps the title, which isn't itself facilitation. - JOB_ROLES = { "assistant" => "Assistant" }.freeze + # Every Personnel role is a Facilitator, the shape prod's January 2026 conversion gave the + # old app's roles. A Leader or Liaison is also a primary contact; an Assistant isn't. + NON_PRIMARY_ROLES = %w[assistant].freeze LEGACY_LEADER_TITLE = "Leader" ROLE_COMMENT_TOPIC = "FileMaker import" SOURCES = { FmPersonnel => "Personnel", FmPeopleToOrganization => "People-to-organizations", FmRolodex => "Rolodex" }.freeze @@ -103,44 +102,27 @@ def import_personnel(row) dates = personnel_dates(data, @projects[data["ProjectID"].to_s]) role = data["Role"].to_s.squish - title = title_for(role) + attributes = dates.merge(primary_contact: primary_contact?(role)) people.each do |person| - if title == Affiliation::FACILITATOR_TITLE - affiliation = affiliate(person, organization, row, label, title:, attributes: dates.merge(primary_contact: true), codes: @personnel_ids) - note_role(affiliation, row, role) if affiliation && role.present? - next - end - - affiliate(person, organization, row, label, title:, attributes: dates.merge(primary_contact: false), codes: @personnel_ids) - next if facilitator_for?(person, organization, dates) - - create(person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, - attributes: dates.merge(primary_contact: false), code: row.fm_id) + affiliation = affiliate(person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, attributes:, codes: @personnel_ids) + note_role(affiliation, row, role) if affiliation && role.present? end end - # Folding Leader and Liaison into Facilitator drops the word FileMaker used, so the - # affiliation keeps it. + # Every role FileMaker gave folds into Facilitator, so the affiliation keeps the word — + # and says when the import turned on primary contact for a row the portal already had. def note_role(affiliation, row, role) + flipped = !affiliation.previously_new_record? && affiliation.saved_change_to_primary_contact? affiliation.comments.create!(topic: ROLE_COMMENT_TOPIC, - body: "Marked as #{role} in FileMaker (Personnel #{row.fm_id}) at the time of the FileMaker import.") - end - - # An Assistant also facilitated over the same stretch, as prod's January 2026 conversion - # recorded them: a Facilitator affiliation beside the role, unless one with the same - # dates is already there. - def facilitator_for?(person, organization, dates) - person.affiliations.where(organization:).any? do |affiliation| - same_title?(affiliation.title, Affiliation::FACILITATOR_TITLE) && - affiliation.start_date == dates[:start_date] && affiliation.end_date == dates[:end_date] - end + body: "Marked as #{role} in FileMaker (Personnel #{row.fm_id}) at the time of the FileMaker import." \ + "#{" Primary contact turned on." if flipped}") end # Nearly every Personnel row is a Leader, so a blank or unrecognized role is one too. - def title_for(role) - title = JOB_ROLES.fetch(role.downcase, Affiliation::FACILITATOR_TITLE) - @roles["#{role.presence || "(blank)"} → #{title}"] += 1 - title + def primary_contact?(role) + primary = NON_PRIMARY_ROLES.exclude?(role.downcase) + @roles["#{role.presence || "(blank)"} → Facilitator#{", not primary contact" unless primary}"] += 1 + primary end # Personnel rows carry no dates of their own: the assignment starts when it was entered @@ -227,6 +209,7 @@ def link(affiliation, row, label, title:, attributes:, codes:) affiliation.filemaker_code = Affiliation.join_filemaker_codes(affiliation.filemaker_code, row.fm_id) if codes affiliation.title = title if affiliation.title.blank? + affiliation.primary_contact = true if attributes[:primary_contact] kept = fill_dates(affiliation, attributes) affiliation.updated_by_id ||= updated_by(row.data) return failed(affiliation, label) unless affiliation.save diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 7a60465f0f..2ef52e1aeb 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -77,11 +77,13 @@ Constraints already decided elsewhere: contacts (402 aren't), all 53 Liaisons are, and 111 rows are still titled "Leader" (which the exact-match Facilitator scope doesn't count, and which the import matches as a Facilitator rather than duplicating). - - **Leader** (8,112 of 8,907 rows) and **Liaison** (611) → a Facilitator affiliation marked - `primary_contact`, with a "FileMaker import" comment recording the role FileMaker gave. - - **Assistant** (232) → an Assistant affiliation over the same dates — the title itself - never counts as facilitation — plus a Facilitator affiliation over those dates unless the - person already has one, as the January 2026 conversion did. + - Every role becomes a **Facilitator** affiliation, with a "FileMaker import" comment + recording the role FileMaker gave. No prod row is titled Assistant, so the old app's + assistants likely became the 402 Facilitators who aren't primary contacts. + - **Leader** (8,112 of 8,907 rows) and **Liaison** (611) are `primary_contact`. Matching + an existing Facilitator row turns its primary contact on — the one value the import + overwrites rather than fills — and the comment says so. + - **Assistant** (232) isn't a primary contact, and leaves a matched row's flag alone. - A blank (80) or stray role reads as a Leader. - **Dates:** Personnel rows carry none of their own, so the assignment starts on `DateEntered` and ends on the project's `CloseDate` (else `EndDate`). Only `Status` @@ -132,8 +134,8 @@ the real run. - **Which `Status` values mean active** — only "Active", from the production values profiled 2026-10-24 (see Personnel above). A closed project's `CloseDate` (else `EndDate`) ends every affiliation under it. `AgreementStatus` isn't read: 8,376 of 8,907 rows leave it blank. -- **Is an Assistant a facilitator?** The Assistant title isn't; the Facilitator affiliation - minted beside it is, matching January 2026 (decided 2026-10-24). +- **Is an Assistant a facilitator?** Yes — a Facilitator who isn't a primary contact + (decided 2026-10-24). - **Merged projects** (`MergerWithID`): folded — the merged-away project's code joins the survivor's org, so its personnel land there. One the portal already keeps as its own org is reported for a hand merge rather than merged automatically. diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index 281e73298f..79f67124ed 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -51,6 +51,7 @@ def import expect(affiliation).to have_attributes(title: "Facilitator", primary_contact: true, start_date: Date.new(2015, 3, 1)) expect(affiliation.comments.sole).to have_attributes(topic: "FileMaker import", body: a_string_including("Liaison", "PS0001")) expect(import.roles).to eq("Liaison → Facilitator" => 1) + expect(affiliation.comments.sole.body).not_to include("Primary contact turned on") end it "notes a Leader's role too, but not a blank one" do @@ -62,23 +63,35 @@ def import expect(Comment.where(commentable: person.affiliations).pluck(:body)).to contain_exactly(a_string_including("Leader", "PS0001")) end - it "gives an Assistant their role plus a Facilitator affiliation over the same dates" do - personnel("PS0002", Role: "assistant", DateEntered: "1/5/2018") + it "makes an Assistant a Facilitator who isn't a primary contact, noting the role" do + personnel("PS0002", Role: "assistant") import = described_class.new.call - expect(person.affiliations.order(:title).pluck(:filemaker_code, :title, :start_date, :primary_contact)) - .to eq([ [ "PS0002", "Assistant", Date.new(2018, 1, 5), false ], [ "PS0002", "Facilitator", Date.new(2018, 1, 5), false ] ]) - expect(import.roles).to eq("assistant → Assistant" => 1) + affiliation = person.affiliations.sole + expect(affiliation).to have_attributes(title: "Facilitator", primary_contact: false, start_date: Date.new(2015, 3, 1)) + expect(affiliation.comments.sole.body).to include("assistant", "PS0002") + expect(import.roles).to eq("assistant → Facilitator, not primary contact" => 1) + end + + it "turns on primary contact for a Leader's existing Facilitator row, and says so" do + existing = create(:affiliation, person:, organization: program, title: "Facilitator", primary_contact: false) + personnel("PS0001") + + import + + expect(existing.reload).to have_attributes(primary_contact: true, filemaker_code: "PS0001") + expect(existing.comments.sole.body).to include("Leader", "Primary contact turned on") end - it "adds no second Facilitator beside an Assistant when one with the same dates is already there" do - create(:affiliation, person:, organization: program, title: "Facilitator", start_date: Date.new(2015, 3, 1), end_date: Date.new(2019, 6, 30)) + it "leaves an Assistant's existing Facilitator row's primary contact as it is" do + existing = create(:affiliation, person:, organization: program, title: "Facilitator", primary_contact: true) personnel("PS0001", Role: "Assistant") import - expect(person.affiliations.pluck(:title)).to contain_exactly("Facilitator", "Assistant") + expect(existing.reload.primary_contact).to be(true) + expect(existing.comments.sole.body).not_to include("Primary contact turned on") end it "reads a blank or unrecognized role as a Leader" do From 8d79d40e4eca8c828aa6d7f30d255af5e05b7edd Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 09:21:56 -0400 Subject: [PATCH 09/25] Keep a FileMaker program on record when its Personnel row names no one Some Personnel rows name a project but no person. Skipping them would leave an organization with no trace that its program ran, so when nothing else affiliates anyone there, a placeholder person, AWBW Placeholder, holds that history. It's read last so a real person always wins. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../import_from_filemaker.rb | 47 ++++++++++++++++--- ...maker-organization-and-personnel-import.md | 8 +++- .../import_from_filemaker_spec.rb | 34 ++++++++++++++ 3 files changed, 81 insertions(+), 8 deletions(-) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index 8c8334d439..4ce191e34c 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -38,6 +38,8 @@ class ImportFromFilemaker NON_PRIMARY_ROLES = %w[assistant].freeze LEGACY_LEADER_TITLE = "Leader" ROLE_COMMENT_TOPIC = "FileMaker import" + # Stands in where a Personnel row names a project but no person. + PLACEHOLDER_NAME = { first_name: "AWBW", last_name: "Placeholder" }.freeze SOURCES = { FmPersonnel => "Personnel", FmPeopleToOrganization => "People-to-organizations", FmRolodex => "Rolodex" }.freeze attr_reader :results, :differences, :statuses, :roles @@ -63,6 +65,7 @@ def call @projects = FmProject.all.index_by(&:fm_id) @personnel_ids = FmPersonnel.pluck(:fm_id).to_set @job_ids = FmPeopleToOrganization.pluck(:fm_id).to_set + @unstaffed = [] @read = 0 each_batch(FmPersonnel.all) { |row| import_personnel(row) } @@ -70,6 +73,7 @@ def call addresses = FmRolodex.contact_index(FmAddress) phones = FmRolodex.contact_index(FmPhoneNumber) each_batch(FmRolodex.all) { |rolodex| import_rolodex(rolodex.preload_contacts(addresses:, phone_numbers: phones)) } + in_batch(@unstaffed) { |row, organization, label| import_unstaffed(row, organization, label) } self end @@ -80,16 +84,20 @@ def call # Untouched, so a new affiliation doesn't stamp its person as edited now. def each_batch(scope, &import) scope.find_in_batches(batch_size: BATCH_SIZE) do |batch| - ActiveRecord::Base.transaction(requires_new: true) do - ActiveRecord::Base.no_touching do - Analytics::LifecycleBuffer.tagging(@run ? { import_run: @run } : {}) { batch.each(&import) } - end - Analytics::LifecycleBuffer.flush_to(@tracker) - end + in_batch(batch, &import) @progress&.call(@read += batch.size) end end + def in_batch(rows, &import) + ActiveRecord::Base.transaction(requires_new: true) do + ActiveRecord::Base.no_touching do + Analytics::LifecycleBuffer.tagging(@run ? { import_run: @run } : {}) { rows.each(&import) } + end + Analytics::LifecycleBuffer.flush_to(@tracker) + end + end + def import_personnel(row) data = row.data label = "#{row.fm_id} #{data["Name"].to_s.squish}".squish @@ -97,8 +105,9 @@ def import_personnel(row) people = @people.records(data["PersonID"]) organization = organization_for(data["ProjectID"]) - return record(:skipped, "#{label}: person #{data["PersonID"].presence || "(none)"} wasn't imported") if people.empty? return record(:skipped, "#{label}: project #{data["ProjectID"].presence || "(none)"} wasn't imported") unless organization + return @unstaffed << [ row, organization, label ] if data["PersonID"].blank? + return record(:skipped, "#{label}: person #{data["PersonID"]} wasn't imported") if people.empty? dates = personnel_dates(data, @projects[data["ProjectID"].to_s]) role = data["Role"].to_s.squish @@ -109,6 +118,30 @@ def import_personnel(row) end end + # A Personnel row naming no one still records that the program ran. Read last, once every + # other row has had its say, so the placeholder only stands in at an organization nothing + # else affiliates anyone with — one placeholder there is enough. + def import_unstaffed(row, organization, label) + if organization.affiliations.exists? + return record(:skipped, "#{label}: names no person, and organization ##{organization.id} already has affiliations") + end + + dates = personnel_dates(row.data, @projects[row.data["ProjectID"].to_s]) + affiliation = affiliate(placeholder_person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, + attributes: dates.merge(primary_contact: false), codes: @personnel_ids) + return unless affiliation + + affiliation.comments.create!(topic: ROLE_COMMENT_TOPIC, + body: "FileMaker Personnel #{row.fm_id} names no person, so #{PLACEHOLDER_NAME.values.join(" ")} " \ + "stands in to keep the program's FileMaker history.") + end + + def placeholder_person + @placeholder_person ||= Person.where("LOWER(first_name) = ? AND LOWER(last_name) = ?", *PLACEHOLDER_NAME.values.map(&:downcase)) + .order(:id).first || + Person.create!(**PLACEHOLDER_NAME, created_by_id: @import_user_id, updated_by_id: @import_user_id) + end + # Every role FileMaker gave folds into Facilitator, so the affiliation keeps the word — # and says when the import turned on primary contact for a row the portal already had. def note_role(affiliation, row, role) diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 2ef52e1aeb..2c18939396 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -85,6 +85,11 @@ Constraints already decided elsewhere: overwrites rather than fills — and the comment says so. - **Assistant** (232) isn't a primary contact, and leaves a matched row's flag alone. - A blank (80) or stray role reads as a Leader. + - **A row naming a project but no person** (29 of the blank-role rows) still records that + the program ran. Once every other row is read, an organization nothing else affiliates + anyone with gets one Facilitator affiliation for the placeholder person **AWBW + Placeholder** (found by name, created when missing — distinct from the existing AWBW + Staff), with a comment saying so. - **Dates:** Personnel rows carry none of their own, so the assignment starts on `DateEntered` and ends on the project's `CloseDate` (else `EndDate`). Only `Status` "Active" (1,774) stays open; Inactive (7,220), Pending, Suspended, blank, and values @@ -139,4 +144,5 @@ the real run. - **Merged projects** (`MergerWithID`): folded — the merged-away project's code joins the survivor's org, so its personnel land there. One the portal already keeps as its own org is reported for a hand merge rather than merged automatically. -- **Unmatched people:** skipped and reported. +- **Unmatched people:** skipped and reported. A row naming no person at all gets AWBW + Placeholder instead, when its organization has no other affiliations (decided 2026-10-24). diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index 79f67124ed..e38fe39e6e 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -132,6 +132,40 @@ def import expect(person.affiliations.pluck(:filemaker_code)).to contain_exactly("PS0001", "PS0002") end + describe "a row naming a project but no person" do + it "puts AWBW Placeholder on an organization nothing else affiliates anyone with" do + personnel("PS0001", PersonID: "", Role: "") + personnel("PS0002", PersonID: "", Role: "", DateEntered: "1/5/2018") + + import = described_class.new.call + + placeholder = Person.find_by!(first_name: "AWBW", last_name: "Placeholder") + affiliation = placeholder.affiliations.sole + expect(affiliation).to have_attributes(organization: program, title: "Facilitator", primary_contact: false, filemaker_code: "PS0001") + expect(affiliation.comments.sole.body).to include("PS0001", "names no person") + expect(import.results[:skipped]).to contain_exactly(a_string_including("PS0002", "already has affiliations")) + end + + it "leaves an organization that already has affiliations alone, even from a later row" do + personnel("PS0001", PersonID: "") + personnel("PS0002") + + import + + expect(Person.where(first_name: "AWBW", last_name: "Placeholder")).to be_empty + expect(program.affiliations.sole.person).to eq(person) + end + + it "reuses the placeholder person already in the portal" do + existing = create(:person, first_name: "AWBW", last_name: "Placeholder", email: nil, user: nil) + personnel("PS0001", PersonID: "") + + import + + expect(existing.affiliations.count).to eq(1) + end + end + it "skips a row whose person or project wasn't imported" do personnel("PS0001", PersonID: "99999") personnel("PS0002", ProjectID: "P9999") From 017b00f8ab70cd4ac1d9550164e6045fb22397de Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 09:22:29 -0400 Subject: [PATCH 10/25] Note the AWBW Placeholder fallback in AGENTS.md Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 9de54e1745..40664671d6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -332,7 +332,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — a Leader or Liaison (or blank/stray role) as a primary contact, turning it on for a matched existing row, an Assistant (`NON_PRIMARY_ROLES`) not (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing "Leader"-titled row matches as a Facilitator; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — a Leader or Liaison (or blank/stray role) as a primary contact, turning it on for a matched existing row, an Assistant (`NON_PRIMARY_ROLES`) not (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing "Leader"-titled row matches as a Facilitator; a row naming a project but no person gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) From d4a806d67aa24cab9322bf4de791a8b67bd1cf6c Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 09:25:00 -0400 Subject: [PATCH 11/25] Mark primary contact only where FileMaker says so; match by stretch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A blank or unrecognized role says nothing about who the program's contact was, so only a FileMaker Leader or Liaison is one now. A person can facilitate at one organization over several separate stretches, so a Personnel row links an existing Facilitator row only when their periods overlap — otherwise it would fold a 2015 assignment into a 2024 one. Blank dates read as open-ended, except an ended row with no end date, which stays at its start so it can't reach a later stretch. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- .../import_from_filemaker.rb | 50 +++++++++++---- ...maker-organization-and-personnel-import.md | 16 +++-- .../import_from_filemaker_spec.rb | 64 +++++++++++++++++-- 4 files changed, 109 insertions(+), 23 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 40664671d6..73e9173506 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -332,7 +332,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — a Leader or Liaison (or blank/stray role) as a primary contact, turning it on for a matched existing row, an Assistant (`NON_PRIMARY_ROLES`) not (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing "Leader"-titled row matches as a Facilitator; a row naming a project but no person gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row; an existing Facilitator row matches only for the same stretch (overlapping periods, `#same_stint`), so separate stints stay separate (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing "Leader"-titled row matches as a Facilitator; a row naming a project but no person gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index 4ce191e34c..1ba75849b8 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -5,9 +5,10 @@ module AffiliationServices # # 1. Personnel (FmPersonnel) — a person staffing a project's art program becomes a # Facilitator affiliation, dated from when the row was entered to when the project - # closed, with a comment naming the role FileMaker gave. A Leader or Liaison is a - # primary contact — turned on for a Facilitator row the portal already had — and an - # Assistant isn't. Each carries the row's PrsnlRecID as its filemaker_code. + # closed, with a comment naming the role FileMaker gave. Only a Leader or Liaison is a + # primary contact — turned on for a Facilitator row the portal already had. It links + # an existing Facilitator row only for the same stretch (#same_stint), so separate + # stints stay separate. Each carries the row's PrsnlRecID as its filemaker_code. # 2. People-to-organizations (FmPeopleToOrganization) — a job at an organization, with # its title, carrying the row's RecordID. # 3. The Rolodex card itself (FmRolodex) — the organization its OrgID (or its typed @@ -34,8 +35,8 @@ class ImportFromFilemaker # ended, since over-reporting active programs would skew program status. ACTIVE_STATUSES = %w[active].freeze # Every Personnel role is a Facilitator, the shape prod's January 2026 conversion gave the - # old app's roles. A Leader or Liaison is also a primary contact; an Assistant isn't. - NON_PRIMARY_ROLES = %w[assistant].freeze + # old app's roles. Only a role FileMaker names as Leader or Liaison makes a primary contact. + PRIMARY_ROLES = %w[leader leaders liaison].freeze LEGACY_LEADER_TITLE = "Leader" ROLE_COMMENT_TOPIC = "FileMaker import" # Stands in where a Personnel row names a project but no person. @@ -113,7 +114,7 @@ def import_personnel(row) role = data["Role"].to_s.squish attributes = dates.merge(primary_contact: primary_contact?(role)) people.each do |person| - affiliation = affiliate(person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, attributes:, codes: @personnel_ids) + affiliation = affiliate(person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, attributes:, codes: @personnel_ids, dated: true) note_role(affiliation, row, role) if affiliation && role.present? end end @@ -128,7 +129,7 @@ def import_unstaffed(row, organization, label) dates = personnel_dates(row.data, @projects[row.data["ProjectID"].to_s]) affiliation = affiliate(placeholder_person, organization, row, label, title: Affiliation::FACILITATOR_TITLE, - attributes: dates.merge(primary_contact: false), codes: @personnel_ids) + attributes: dates.merge(primary_contact: false), codes: @personnel_ids, dated: true) return unless affiliation affiliation.comments.create!(topic: ROLE_COMMENT_TOPIC, @@ -151,10 +152,9 @@ def note_role(affiliation, row, role) "#{" Primary contact turned on." if flipped}") end - # Nearly every Personnel row is a Leader, so a blank or unrecognized role is one too. def primary_contact?(role) - primary = NON_PRIMARY_ROLES.exclude?(role.downcase) - @roles["#{role.presence || "(blank)"} → Facilitator#{", not primary contact" unless primary}"] += 1 + primary = PRIMARY_ROLES.include?(role.downcase) + @roles["#{role.presence || "(blank)"} → Facilitator#{", primary contact" if primary}"] += 1 primary end @@ -218,15 +218,41 @@ def named_organization(name) # the row's own table holds — an affiliation already carrying another of them records # a different row (another stint on the same project), so it's never the match. A row # with no code of its own (the Rolodex card) only fills in what's missing. - def affiliate(person, organization, row, label, title:, attributes:, codes:) + def affiliate(person, organization, row, label, title:, attributes:, codes:, dated: false) existing = person.affiliations.where(organization:).to_a - match = existing.find { |affiliation| same_title?(affiliation.title, title) && !records_another_row?(affiliation, codes) } + candidates = existing.select { |affiliation| same_title?(affiliation.title, title) && !records_another_row?(affiliation, codes) } + match = dated ? same_stint(candidates, attributes) : candidates.first match ||= existing.find { |affiliation| affiliation.title.blank? } if title.present? && codes.nil? return link(match, row, label, title:, attributes:, codes:) if match create(person, organization, row, label, title:, attributes:, code: codes && row.fm_id) end + # A person can facilitate at one organization over several separate stretches, so a + # Personnel row links only the affiliation recording the same one: the periods overlap, + # reading a blank start as "since whenever" and a blank end as "still going" — except an + # ended row with no end date, which reads as the moment it started, so it can't reach a + # later stretch. Identical dates win, then the nearest start, then one with dates at all. + def same_stint(candidates, dates) + period = stint(dates[:start_date], dates[:end_date], dates[:inactive]) + candidates.select { |affiliation| overlap?(period, stint(affiliation.start_date, affiliation.end_date, affiliation.inactive?)) } + .min_by { |affiliation| stint_rank(affiliation, dates) } + end + + def stint(start_date, end_date, ended) + [ start_date, end_date || (start_date if ended) ] + end + + def overlap?((starts, ends), (other_starts, other_ends)) + (starts.nil? || other_ends.nil? || starts <= other_ends) && (other_starts.nil? || ends.nil? || other_starts <= ends) + end + + def stint_rank(affiliation, dates) + identical = affiliation.start_date == dates[:start_date] && affiliation.end_date == dates[:end_date] + gap = (affiliation.start_date - dates[:start_date]).abs if affiliation.start_date && dates[:start_date] + [ identical ? 0 : 1, gap || Float::INFINITY ] + end + def records_another_row?(affiliation, codes) codes && affiliation.filemaker_codes.any? { |code| codes.include?(code) } end diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 2c18939396..d04a92ab24 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -80,11 +80,17 @@ Constraints already decided elsewhere: - Every role becomes a **Facilitator** affiliation, with a "FileMaker import" comment recording the role FileMaker gave. No prod row is titled Assistant, so the old app's assistants likely became the 402 Facilitators who aren't primary contacts. - - **Leader** (8,112 of 8,907 rows) and **Liaison** (611) are `primary_contact`. Matching - an existing Facilitator row turns its primary contact on — the one value the import - overwrites rather than fills — and the comment says so. - - **Assistant** (232) isn't a primary contact, and leaves a matched row's flag alone. - - A blank (80) or stray role reads as a Leader. + - Only a role FileMaker names **Leader** (8,112 of 8,907 rows) or **Liaison** (611) is + `primary_contact`. Matching an existing Facilitator row turns its primary contact on — + the one value the import overwrites rather than fills — and the comment says so. + - **Assistant** (232), a blank role (80) or a stray word isn't a primary contact, and + leaves a matched row's flag alone. + - **One row, one stretch.** A person can facilitate at one org over several separate + stretches (2020–2022, 2024–Oct 2025, Feb 2026–now), so a Personnel row links an existing + Facilitator row only when their periods overlap. A blank start reads as "since whenever" + and a blank end as "still going", except an ended row with no end date, which reads as + the moment it started so it can't reach a later stretch. Identical dates win, then the + nearest start, then a row with no dates at all. No overlap creates a new row. - **A row naming a project but no person** (29 of the blank-role rows) still records that the program ran. Once every other row is read, an organization nothing else affiliates anyone with gets one Facilitator affiliation for the placeholder person **AWBW diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index e38fe39e6e..39b74956e5 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -50,7 +50,7 @@ def import affiliation = person.affiliations.sole expect(affiliation).to have_attributes(title: "Facilitator", primary_contact: true, start_date: Date.new(2015, 3, 1)) expect(affiliation.comments.sole).to have_attributes(topic: "FileMaker import", body: a_string_including("Liaison", "PS0001")) - expect(import.roles).to eq("Liaison → Facilitator" => 1) + expect(import.roles).to eq("Liaison → Facilitator, primary contact" => 1) expect(affiliation.comments.sole.body).not_to include("Primary contact turned on") end @@ -71,7 +71,7 @@ def import affiliation = person.affiliations.sole expect(affiliation).to have_attributes(title: "Facilitator", primary_contact: false, start_date: Date.new(2015, 3, 1)) expect(affiliation.comments.sole.body).to include("assistant", "PS0002") - expect(import.roles).to eq("assistant → Facilitator, not primary contact" => 1) + expect(import.roles).to eq("assistant → Facilitator" => 1) end it "turns on primary contact for a Leader's existing Facilitator row, and says so" do @@ -94,14 +94,68 @@ def import expect(existing.comments.sole.body).not_to include("Primary contact turned on") end - it "reads a blank or unrecognized role as a Leader" do + it "makes a primary contact only where FileMaker's role says Leader or Liaison" do personnel("PS0001", Role: "Leaders") personnel("PS0002", Role: "", DateEntered: "1/5/2018") + personnel("PS0003", Role: "Something else", DateEntered: "1/5/2017") import = described_class.new.call - expect(person.affiliations.pluck(:title).uniq).to eq([ "Facilitator" ]) - expect(import.roles).to eq("Leaders → Facilitator" => 1, "(blank) → Facilitator" => 1) + expect(person.affiliations.order(:filemaker_code).pluck(:filemaker_code, :title, :primary_contact)) + .to eq([ [ "PS0001", "Facilitator", true ], [ "PS0002", "Facilitator", false ], [ "PS0003", "Facilitator", false ] ]) + expect(import.roles).to eq("Leaders → Facilitator, primary contact" => 1, "(blank) → Facilitator" => 1, + "Something else → Facilitator" => 1) + end + + describe "matching an existing Facilitator row by its stretch" do + before { FmProject.find_by!(fm_id: "P0001").update!(data: { "OrgID" => "ORG00001" }) } + + def stint(starts, ends = nil) + create(:affiliation, person:, organization: program, title: "Facilitator", start_date: starts, end_date: ends) + end + + it "links the stretch the row falls in and leaves the others alone" do + stint(Date.new(2020, 1, 1), Date.new(2022, 12, 31)) + middle = stint(Date.new(2024, 1, 1), Date.new(2025, 10, 31)) + stint(Date.new(2026, 2, 1)) + personnel("PS0001", DateEntered: "3/1/2024") + + import + + expect(middle.reload.filemaker_code).to eq("PS0001") + expect(person.affiliations.where.not(id: middle.id).pluck(:filemaker_code)).to all(be_nil) + expect(person.affiliations.count).to eq(3) + end + + it "adds a new stretch when none overlaps" do + stint(Date.new(2020, 1, 1), Date.new(2022, 12, 31)) + personnel("PS0001", DateEntered: "3/1/2024") + + import + + expect(person.affiliations.order(:start_date).pluck(:start_date, :filemaker_code)) + .to eq([ [ Date.new(2020, 1, 1), nil ], [ Date.new(2024, 3, 1), "PS0001" ] ]) + end + + it "reads an ended row with no end date as the moment it started" do + stint(Date.new(2014, 1, 1), Date.new(2016, 12, 31)) + later = stint(Date.new(2024, 1, 1)) + personnel("PS0001", DateEntered: "3/1/2015", Status: "Inactive") + + import + + expect(later.reload.filemaker_code).to be_nil + expect(person.affiliations.find_by(start_date: Date.new(2014, 1, 1)).filemaker_code).to eq("PS0001") + end + + it "links a row with no dates at all when nothing better fits" do + undated = stint(nil) + personnel("PS0001", DateEntered: "3/1/2024") + + import + + expect(undated.reload).to have_attributes(filemaker_code: "PS0001", start_date: Date.new(2024, 3, 1)) + end end it "links the Facilitator affiliation the person already has there, filling its blank start" do From d32519cfc53a991ad16b546a9581ea30c8f945cc Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 17:42:52 -0400 Subject: [PATCH 12/25] Expect affiliations to point into people-to-organizations too The import gives job affiliations a FileMaker people-to-organizations code, so the spec that lists each model's archive tables needs it. Co-Authored-By: Claude Opus 5.5 (1M context) --- spec/models/concerns/filemaker_coded_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/models/concerns/filemaker_coded_spec.rb b/spec/models/concerns/filemaker_coded_spec.rb index 4c1d7bd465..aa83ad1ac8 100644 --- a/spec/models/concerns/filemaker_coded_spec.rb +++ b/spec/models/concerns/filemaker_coded_spec.rb @@ -65,7 +65,7 @@ it "names the archive tables each coded model points into" do expect(Person.filemaker_archives).to eq([ FmRolodex ]) expect(Organization.filemaker_archives).to eq([ FmProject, FmOrganization ]) - expect(Affiliation.filemaker_archives).to eq([ FmProject, FmPersonnel ]) + expect(Affiliation.filemaker_archives).to eq([ FmProject, FmPersonnel, FmPeopleToOrganization ]) end end end From 55c457dbb0189f1bcaa5b5fe8363085c53cf92f8 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 18:57:19 -0400 Subject: [PATCH 13/25] Take address handling out of the organizations import; use the shared translator FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829), so every FileMaker import stores the dropdowns' countries and states. This branch needs Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 1 - app/services/affiliation_services/import_from_filemaker.rb | 2 ++ app/services/filemaker_values.rb | 4 +--- .../organization_services/import_from_filemaker.rb | 7 ++++--- lib/tasks/fm_import_organizations.rake | 4 +++- 5 files changed, 10 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 73e9173506..db4cbb6c22 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -305,7 +305,6 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `TagTranslation` — Turns a legacy word for who someone serves into tags: `#resolve` answers with both the sector ids and the age-group ids a term means, since the vocabularies cross (`Elder Abuse` is a life experience that means the Elders *age range*). A tag's own name resolves only while it's **published** — a retired tag is unpublished, not deleted — but a recorded `TagAlias` may point at a retired one deliberately (FileMaker's `Poverty`). Splits a repeating field on line breaks and semicolons — **not commas**, which the archive's own wording contains — and drops a trailing parenthetical. Built once and reused (it reads the whole vocabulary up front); a term nothing answers to lands in `#unmatched` rather than a guess. Primacy is the caller's call, since it follows from the field a term sat in - `TagAliasSeeder` — Records what the FileMaker archive, the older registration/scholarship forms, Story Share and the portal's earlier sector list each called a tag the portal now names its own way (**ADR-0007**) as `TagAlias` rows, so the translation is data an admin can extend and a record of what a tag used to be called. Terms AWBW decided are recorded as-is; semantic reads of the archive's hand-typed free text are marked **`proposed`** for a pass with staff (they still resolve, so no import waits on it), and text naming an organization or a facilitator's own circumstances is left alone. Idempotent on (term, tag) and never touches a term added by hand; a term whose tag isn't published is reported, not created. Shared with `data:seed_tag_aliases` - `FilemakerValues` — How every FileMaker archive import reads a raw field: `parse_date` (the archive's mixed formats, two-digit years given a century), `flag?`, `contact_type` (archive type words → the slice a model accepts), `email_addresses` (the valid addresses inside a free-text field), and `clip` to a varchar. Module functions, also included by the importers -- `FmAddressAttributes` — An `FmAddress` row's fields → `Address` attributes (street with house number, ZIP or its lookup, county from the archive's ZIP table read once, locality via `AddressLocality`), nil when it lacks street/city/state. Shared by the Rolodex, organizations and affiliations imports - `FilemakerCodeIndex` — Which records of a `FilemakerCoded` model carry each code, read once so a bulk import looks codes up in memory; `#add` records codes the import writes as it goes - `StaffTranslation` — Turns a FileMaker `EnteredBy`/`ModifiedBy` value into a user id through the recorded `StaffAlias` rows (`#user_id_for`), case- and spacing-insensitive. Built once and reused (reads every alias up front); a name nothing answers to lands in `#unmatched` unless it's recorded as ignored - `StaffAliasLoader` — Records `StaffAlias` rows from a CSV/TSV staff list (`staff`, `user_ids_by_email`, `portal_user_email`, optional `ignored` columns), resolving the user by id then email. Idempotent: a name already recorded is left alone; a name with no user yet is reported in `#unresolved`, not recorded. Shared with `data:load_staff_aliases` diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index 1ba75849b8..e8492e640c 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -45,6 +45,8 @@ class ImportFromFilemaker attr_reader :results, :differences, :statuses, :roles + def unmatched_addresses = @address_attributes.unmatched + def initialize(progress: nil, run: nil) @progress = progress @run = run diff --git a/app/services/filemaker_values.rb b/app/services/filemaker_values.rb index e3503d5a16..33cd9e2a52 100644 --- a/app/services/filemaker_values.rb +++ b/app/services/filemaker_values.rb @@ -10,9 +10,7 @@ module FilemakerValues DATE_FORMATS = %w[%m/%d/%Y %Y-%m-%d %m-%d-%Y %Y/%m/%d].freeze # A two-digit year below this is read as 20xx, at or above it as 19xx. TWO_DIGIT_YEAR_PIVOT = 10 - CONTACT_TYPES = { "home" => "personal", "personal" => "personal", "cell" => "personal", - "mobile" => "personal", "work" => "work", "business" => "work", - "professional" => "work", "mailing" => "mailing" }.freeze + CONTACT_TYPES = AddressTranslation::CONTACT_TYPES # Every string copied here lands in a varchar(255) the archive's free text isn't # bounded by, and an over-long incidental field would otherwise cost the record — diff --git a/app/services/organization_services/import_from_filemaker.rb b/app/services/organization_services/import_from_filemaker.rb index 944e9c8e7f..ee0135e197 100644 --- a/app/services/organization_services/import_from_filemaker.rb +++ b/app/services/organization_services/import_from_filemaker.rb @@ -47,6 +47,8 @@ class ImportFromFilemaker attr_reader :results, :differences, :unmatched_types + def unmatched_addresses = @address_attributes.unmatched + def initialize(progress: nil, run: nil) @progress = progress @run = run @@ -374,9 +376,8 @@ def place_project(organization, data) return if state.blank? primary = organization.addresses.none?(&:primary?) - locality = geography[:locality] || AddressLocality.for(city:, state:, county: geography[:county]) - organization.addresses.build(street_address: "", zip_code: "", city:, state:, address_type: "work", - **geography, locality:, primary:) + place = @address_attributes.place(city:, state:, county: geography[:county]) + organization.addresses.build(place.merge(geography.compact, address_type: "work", primary:)) end def project_geography(data) diff --git a/lib/tasks/fm_import_organizations.rake b/lib/tasks/fm_import_organizations.rake index 75a652ff4b..5c9e7411c9 100644 --- a/lib/tasks/fm_import_organizations.rake +++ b/lib/tasks/fm_import_organizations.rake @@ -75,7 +75,9 @@ namespace :fm do tallies = { "Organization types read as Other, kept as its description" => import.try(:unmatched_types), "Personnel Status values and how they were read" => import.try(:statuses), - "Personnel roles" => import.try(:roles) + "Personnel roles" => import.try(:roles), + "Address countries nothing translates (left blank)" => import.try(:unmatched_addresses)&.fetch(:country, nil), + "Address states nothing translates (kept as typed)" => import.try(:unmatched_addresses)&.fetch(:state, nil) } tallies.each do |heading, counts| next if counts.blank? From 88fa8639db2ca702dd5882cdd543ca2a82bf776c Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 20:26:36 -0400 Subject: [PATCH 14/25] Fold leftover Leader affiliations into Facilitator from Data health Prod still holds 111 rows titled Leader, which count toward no program status. The January conversion already gave 109 of them a Facilitator twin with the same dates, leaving the Leader row as a redundant copy; the other 2 have nothing beside them. One Data health fix merges each copy into its twin (keeping its FileMaker codes) or renames a lone one, so no duplicate remains, and lists any that overlap a Facilitator row with different dates for a person to settle. The stretch-of-time test the import uses moves into AffiliationStint so both read a person's separate stints the same way. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 4 +- .../fold_legacy_leaders.rb | 63 +++++++++++++++++++ .../import_from_filemaker.rb | 21 ++----- app/services/affiliation_stint.rb | 17 +++++ app/services/data_health.rb | 1 + .../data_health/legacy_leader_affiliations.rb | 34 ++++++++++ .../2026-10-24-fold-leader-affiliations.yml | 14 +++++ ...maker-organization-and-personnel-import.md | 4 +- .../fold_legacy_leaders_spec.rb | 56 +++++++++++++++++ spec/services/data_health/checks_spec.rb | 15 +++++ 10 files changed, 211 insertions(+), 18 deletions(-) create mode 100644 app/services/affiliation_services/fold_legacy_leaders.rb create mode 100644 app/services/affiliation_stint.rb create mode 100644 app/services/data_health/legacy_leader_affiliations.rb create mode 100644 db/seeds/features/2026-10-24-fold-leader-affiliations.yml create mode 100644 spec/services/affiliation_services/fold_legacy_leaders_spec.rb diff --git a/AGENTS.md b/AGENTS.md index db4cbb6c22..aae3a3ea7f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -223,7 +223,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `Analytics::EventReferenceLoader` — Batch-loads the records referenced in a page of Ahoy event properties (association changes, associated records), one query per type, so the activity table's Details column links each to its show page without an N+1 - `Analytics::ContentCreationBreakdown` — Counts content created in the selected time range for the Ahoy charts page's Total / User-generated / Admin-generated cards. Each record is credited by its author's account (super_user → admin), falling back to the creating account when no author is set, so promoted ideas and admin-entered facilitator content land on the user side and the two cards always sum to the total -- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) +- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled Leader (folds via `AffiliationServices::FoldLegacyLeaders`; one overlapping a Facilitator row with different dates stays listed), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) - `DataHealth::Worklists` — Counts for review pages that live elsewhere (possible duplicates per dedupe page via each `DuplicateFinder` / `Dedupable.name_groups`, and author credit divergence sections, each linked to its section via `?section=` which renders the divergence results with the page so the anchor can scroll), rendered in Data health's lazy `data_health_worklists` frame because the finders load records in Ruby - `ResourceServices::ResolveOrganizations` — Links resources to an organization from the legacy free-text `agency` name (case-insensitive exact match, else creates one with status "Unknown"). Idempotent; drives both the `data:resolve_resource_organizations` rake task (dry-run by default) and the Data health repair ### Business Logic @@ -331,6 +331,8 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations +- `AffiliationServices::FoldLegacyLeaders` — Folds the old app's "Leader" affiliations (a Facilitator who is a primary contact) into Facilitator: a Leader row with a same-dates Facilitator twin merges into it via `ModelDeduper` (twin turns primary contact on and takes the Leader row's FileMaker codes and attachments), one with no Facilitator row in its stretch is renamed Facilitator with primary contact on, and one overlapping a Facilitator row with different dates is left; each fold leaves a "Leader cleanup" comment. Run by the `LegacyLeaderAffiliations` Data health fix +- `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Shared by the FileMaker affiliations import and `FoldLegacyLeaders` - `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row; an existing Facilitator row matches only for the same stretch (overlapping periods, `#same_stint`), so separate stints stay separate (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing "Leader"-titled row matches as a Facilitator; a row naming a project but no person gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row diff --git a/app/services/affiliation_services/fold_legacy_leaders.rb b/app/services/affiliation_services/fold_legacy_leaders.rb new file mode 100644 index 0000000000..d434c40de3 --- /dev/null +++ b/app/services/affiliation_services/fold_legacy_leaders.rb @@ -0,0 +1,63 @@ +module AffiliationServices + # Folds the old app's "Leader" affiliations into Facilitator ones. A Leader is a + # Facilitator who is a primary contact — the shape the January 2026 conversion gave most + # of them, beside the Leader row it left behind — but a row titled Leader counts toward + # no program status. Each row: + # * with a Facilitator twin over the same dates merges into it: the twin turns primary + # contact on, takes the Leader row's FileMaker codes and anything attached to it, and + # the Leader row goes + # * with no Facilitator row in its stretch becomes one, primary contact on + # * overlapping a Facilitator row with different dates is left for an admin, since which + # dates are right is a call made per row + # Either fold leaves a comment on the Facilitator row saying it was a Leader. Returns how + # many it folded. + class FoldLegacyLeaders + LEGACY_TITLE = "Leader".freeze + COMMENT_TOPIC = "Leader cleanup".freeze + + def self.legacy = Affiliation.where(title: LEGACY_TITLE) + + def call + self.class.legacy.to_a.count { |leader| fold(leader) } + end + + private + + def fold(leader) + facilitators = Affiliation.facilitators.where(person_id: leader.person_id, organization_id: leader.organization_id).to_a + twin = facilitators.find { |affiliation| affiliation.start_date == leader.start_date && affiliation.end_date == leader.end_date } + return merge(leader, twin) if twin + + stint = AffiliationStint.of(leader) + return false if facilitators.any? { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } + + rename(leader) + rescue ActiveRecord::RecordInvalid, ActiveRecord::RecordNotSaved + false + end + + def merge(leader, twin) + Affiliation.transaction do + twin.update!(primary_contact: true, filemaker_code: Affiliation.join_filemaker_codes(twin.filemaker_code, leader.filemaker_code), + organization_address_id: twin.organization_address_id || leader.organization_address_id) + deduper.merge(twin, leader) + note(twin, "Also recorded as a Leader affiliation (##{leader.id}), merged into this one.") + end + true + end + + def rename(leader) + leader.update!(title: Affiliation::FACILITATOR_TITLE, primary_contact: true) + note(leader, "Titled Leader until renamed Facilitator, primary contact.") + true + end + + def note(affiliation, body) + affiliation.comments.create!(topic: COMMENT_TOPIC, body:) + end + + def deduper + @deduper ||= ModelDeduper.new(model_class: Affiliation, logger: Rails.logger, dry_run: false) + end + end +end diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index e8492e640c..fdd5637664 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -37,7 +37,6 @@ class ImportFromFilemaker # Every Personnel role is a Facilitator, the shape prod's January 2026 conversion gave the # old app's roles. Only a role FileMaker names as Leader or Liaison makes a primary contact. PRIMARY_ROLES = %w[leader leaders liaison].freeze - LEGACY_LEADER_TITLE = "Leader" ROLE_COMMENT_TOPIC = "FileMaker import" # Stands in where a Personnel row names a project but no person. PLACEHOLDER_NAME = { first_name: "AWBW", last_name: "Placeholder" }.freeze @@ -231,24 +230,14 @@ def affiliate(person, organization, row, label, title:, attributes:, codes:, dat end # A person can facilitate at one organization over several separate stretches, so a - # Personnel row links only the affiliation recording the same one: the periods overlap, - # reading a blank start as "since whenever" and a blank end as "still going" — except an - # ended row with no end date, which reads as the moment it started, so it can't reach a - # later stretch. Identical dates win, then the nearest start, then one with dates at all. + # Personnel row links only the affiliation recording the same one (AffiliationStint). + # Identical dates win, then the nearest start, then one with dates at all. def same_stint(candidates, dates) - period = stint(dates[:start_date], dates[:end_date], dates[:inactive]) - candidates.select { |affiliation| overlap?(period, stint(affiliation.start_date, affiliation.end_date, affiliation.inactive?)) } + stint = AffiliationStint.for(start_date: dates[:start_date], end_date: dates[:end_date], ended: dates[:inactive]) + candidates.select { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } .min_by { |affiliation| stint_rank(affiliation, dates) } end - def stint(start_date, end_date, ended) - [ start_date, end_date || (start_date if ended) ] - end - - def overlap?((starts, ends), (other_starts, other_ends)) - (starts.nil? || other_ends.nil? || starts <= other_ends) && (other_starts.nil? || ends.nil? || other_starts <= ends) - end - def stint_rank(affiliation, dates) identical = affiliation.start_date == dates[:start_date] && affiliation.end_date == dates[:end_date] gap = (affiliation.start_date - dates[:start_date]).abs if affiliation.start_date && dates[:start_date] @@ -262,7 +251,7 @@ def records_another_row?(affiliation, codes) # Some prod rows still carry the old app's "Leader" title, which is a Facilitator. def same_title?(held, title) held = held.to_s.squish - held.casecmp?(title.to_s.squish) || (title == Affiliation::FACILITATOR_TITLE && held.casecmp?(LEGACY_LEADER_TITLE)) + held.casecmp?(title.to_s.squish) || (title == Affiliation::FACILITATOR_TITLE && held.casecmp?(FoldLegacyLeaders::LEGACY_TITLE)) end def link(affiliation, row, label, title:, attributes:, codes:) diff --git a/app/services/affiliation_stint.rb b/app/services/affiliation_stint.rb new file mode 100644 index 0000000000..b43457ae9a --- /dev/null +++ b/app/services/affiliation_stint.rb @@ -0,0 +1,17 @@ +# One affiliation's stretch of time, for telling a person's separate stints at one +# organization apart. A blank start reads as "since whenever" and a blank end as "still +# going" — except an ended row with no end date, which reads as the moment it started, so +# it can't reach a later stretch. +AffiliationStint = Data.define(:starts, :ends) do + def self.for(start_date:, end_date:, ended:) + new(starts: start_date, ends: end_date || (start_date if ended)) + end + + def self.of(affiliation) + self.for(start_date: affiliation.start_date, end_date: affiliation.end_date, ended: affiliation.inactive?) + end + + def overlap?(other) + (starts.nil? || other.ends.nil? || starts <= other.ends) && (other.starts.nil? || ends.nil? || other.starts <= ends) + end +end diff --git a/app/services/data_health.rb b/app/services/data_health.rb index fce4f4dc7a..9d7ca51e7e 100644 --- a/app/services/data_health.rb +++ b/app/services/data_health.rb @@ -8,6 +8,7 @@ module DataHealth "Affiliations" => [ FacilitatorAffiliationsFromNonTrainings, MisalignedAffiliationProvenance, + LegacyLeaderAffiliations, LegacyOrganizationStatusDrift ], "People" => [ diff --git a/app/services/data_health/legacy_leader_affiliations.rb b/app/services/data_health/legacy_leader_affiliations.rb new file mode 100644 index 0000000000..edfb4445ba --- /dev/null +++ b/app/services/data_health/legacy_leader_affiliations.rb @@ -0,0 +1,34 @@ +module DataHealth + # Affiliations still titled Leader, the old app's word for a Facilitator who is a + # primary contact. AffiliationServices::FoldLegacyLeaders folds them in; a row it leaves + # is one overlapping a Facilitator row with different dates. + class LegacyLeaderAffiliations < Check + def title = "Affiliations still titled Leader" + + def explanation + "A Leader is a Facilitator who is a primary contact, but a row titled Leader counts toward " \ + "no program status. Fixing merges each into the Facilitator row with the same dates, or " \ + "renames it Facilitator when none covers its stretch, leaving a comment either way. One that " \ + "overlaps a Facilitator row with different dates stays here to settle by hand." + end + + def scope = AffiliationServices::FoldLegacyLeaders.legacy.includes(:person, :organization) + + def repairable? = true + + def repair_label = "Fold into Facilitator" + + def repaired_message(number) + "Folded #{number} Leader #{'affiliation'.pluralize(number)} into Facilitator." + end + + def repair! + AffiliationServices::FoldLegacyLeaders.new.call + end + + def describe(affiliation) + dates = [ affiliation.start_date, affiliation.end_date ].compact.map(&:to_s).join(" – ").presence + "#{affiliation.person&.name} — #{affiliation.organization&.name}#{" (#{dates})" if dates}" + end + end +end diff --git a/db/seeds/features/2026-10-24-fold-leader-affiliations.yml b/db/seeds/features/2026-10-24-fold-leader-affiliations.yml new file mode 100644 index 0000000000..f85d5dc315 --- /dev/null +++ b/db/seeds/features/2026-10-24-fold-leader-affiliations.yml @@ -0,0 +1,14 @@ +name: "Fold leftover Leader affiliations into Facilitator" +area: people +display_status: admin_facing +released_on: 2026-10-24 +pr_number: 2788 +summary: >- + Data health lists affiliations still titled Leader, which don't count toward + program status, and one click folds them into Facilitator affiliations marked + as primary contact. +pro_tips: + - "A Leader row with a Facilitator row for the same dates merges into it, so no duplicate is left behind." + - "A Leader row whose dates overlap a Facilitator row's but don't match stays listed for you to settle by hand." + - "Each folded affiliation gets a comment saying it was a Leader." +action_path: "/admin/data_health" diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index d04a92ab24..5180951556 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -76,7 +76,9 @@ Constraints already decided elsewhere: Facilitator + primary contact. Prod, profiled 2026-10-24: 1,829 Facilitators are primary contacts (402 aren't), all 53 Liaisons are, and 111 rows are still titled "Leader" (which the exact-match Facilitator scope doesn't count, and which the import matches as a - Facilitator rather than duplicating). + Facilitator rather than duplicating). Of those, 109 sit beside an undated, primary + Facilitator twin the January conversion made, and 2 have none; the Data health check + "Affiliations still titled Leader" folds them in (`AffiliationServices::FoldLegacyLeaders`). - Every role becomes a **Facilitator** affiliation, with a "FileMaker import" comment recording the role FileMaker gave. No prod row is titled Assistant, so the old app's assistants likely became the 402 Facilitators who aren't primary contacts. diff --git a/spec/services/affiliation_services/fold_legacy_leaders_spec.rb b/spec/services/affiliation_services/fold_legacy_leaders_spec.rb new file mode 100644 index 0000000000..d9d488b5dd --- /dev/null +++ b/spec/services/affiliation_services/fold_legacy_leaders_spec.rb @@ -0,0 +1,56 @@ +# frozen_string_literal: true + +require "rails_helper" + +RSpec.describe AffiliationServices::FoldLegacyLeaders do + let(:person) { create(:person) } + let(:organization) { create(:organization) } + + def affiliation(title, **attributes) + create(:affiliation, person:, organization:, title:, **attributes) + end + + it "merges a Leader row into its same-dates Facilitator twin, carrying its codes and comments" do + twin = affiliation("Facilitator", primary_contact: false, filemaker_code: "P0001") + leader = affiliation("Leader", primary_contact: true, filemaker_code: "PS0001") + leader.comments.create!(body: "Met at the 2019 training") + + expect(described_class.new.call).to eq(1) + + expect(Affiliation.exists?(leader.id)).to be(false) + expect(twin.reload).to have_attributes(primary_contact: true, filemaker_code: "P0001, PS0001") + expect(twin.comments.pluck(:body)).to contain_exactly("Met at the 2019 training", a_string_including("Leader", "##{leader.id}")) + end + + it "merges two Leader rows sharing one twin into it" do + twin = affiliation("Facilitator", primary_contact: true) + affiliation("Leader", primary_contact: true) + # The model refuses an identical row now; prod's pairs predate that. + affiliation("Placeholder", primary_contact: true).update_column(:title, "Leader") + + described_class.new.call + + expect(person.affiliations.reload).to contain_exactly(twin) + expect(twin.comments.where(topic: described_class::COMMENT_TOPIC).count).to eq(2) + end + + it "renames a Leader row with no Facilitator row in its stretch" do + affiliation("Facilitator", start_date: Date.new(2010, 1, 1), end_date: Date.new(2012, 12, 31)) + leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1), primary_contact: false) + + expect(described_class.new.call).to eq(1) + + expect(leader.reload).to have_attributes(title: "Facilitator", primary_contact: true) + expect(leader.comments.sole).to have_attributes(topic: described_class::COMMENT_TOPIC, body: a_string_including("Titled Leader")) + end + + it "leaves a Leader row overlapping a Facilitator row with different dates for an admin" do + affiliation("Facilitator", start_date: Date.new(2018, 1, 1), end_date: Date.new(2022, 12, 31)) + leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) + + expect(described_class.new.call).to eq(0) + + expect(leader.reload.title).to eq("Leader") + expect(leader.comments).to be_empty + end +end diff --git a/spec/services/data_health/checks_spec.rb b/spec/services/data_health/checks_spec.rb index c5f4627fd3..9e26f5a523 100644 --- a/spec/services/data_health/checks_spec.rb +++ b/spec/services/data_health/checks_spec.rb @@ -123,6 +123,21 @@ def affiliation_from(facilitator_training:, title: "Facilitator") end end +RSpec.describe DataHealth::LegacyLeaderAffiliations do + let(:person) { create(:person) } + let(:organization) { create(:organization) } + + it "finds affiliations still titled Leader, and folds them into Facilitator" do + leader = create(:affiliation, person:, organization:, title: "Leader") + create(:affiliation, person:, organization:, title: "Facilitator") + check = described_class.new + + expect(check.scope).to contain_exactly(leader) + expect(check.repair!).to eq(1) + expect(described_class.new).not_to be_any + end +end + RSpec.describe DataHealth::LegacyOrganizationStatusDrift do let!(:active_status) { OrganizationStatus.find_or_create_by!(name: "Active") } let!(:inactive_status) { OrganizationStatus.find_or_create_by!(name: "Inactive") } From f277b45ce8bafb9d96f10d1b23ab2e04d386e997 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 20:34:39 -0400 Subject: [PATCH 15/25] Review Leader affiliations row by row instead of one fold button One button that merges, renames and deletes 111 rows at once asks an admin to trust changes they can't see. The Leader affiliations page works like attendance reconciliation: each row shows its Facilitator twin and offers merge, rename, rename and end, delete, or leave, with a suggestion preselected, and a preview lists every change before anything is made. Data health now reports the rows and links to the page. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 6 +- .../admin/leader_affiliations_controller.rb | 39 +++++++ .../fold_legacy_leaders.rb | 91 +++++++++++---- .../data_health/legacy_leader_affiliations.rb | 23 +--- .../leader_affiliations/confirm.html.erb | 52 +++++++++ .../admin/leader_affiliations/index.html.erb | 79 +++++++++++++ config/routes.rb | 3 + .../2026-10-24-fold-leader-affiliations.yml | 14 --- .../2026-10-24-review-leader-affiliations.yml | 15 +++ ...maker-organization-and-personnel-import.md | 6 +- .../admin/leader_affiliations_spec.rb | 51 +++++++++ .../admin_leader_affiliations_routing_spec.rb | 9 ++ .../fold_legacy_leaders_spec.rb | 108 +++++++++++++----- spec/services/data_health/checks_spec.rb | 13 +-- spec/views/page_bg_class_alignment_spec.rb | 2 + 15 files changed, 413 insertions(+), 98 deletions(-) create mode 100644 app/controllers/admin/leader_affiliations_controller.rb create mode 100644 app/views/admin/leader_affiliations/confirm.html.erb create mode 100644 app/views/admin/leader_affiliations/index.html.erb delete mode 100644 db/seeds/features/2026-10-24-fold-leader-affiliations.yml create mode 100644 db/seeds/features/2026-10-24-review-leader-affiliations.yml create mode 100644 spec/requests/admin/leader_affiliations_spec.rb create mode 100644 spec/routing/admin_leader_affiliations_routing_spec.rb diff --git a/AGENTS.md b/AGENTS.md index aae3a3ea7f..74501bc836 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,7 +57,7 @@ This codebase (Rails 8.1) | Directory | Purpose | Count | |---|---|---| -| `app/controllers/` | Rails controllers (admin/, events/, home/) | ~93 files | +| `app/controllers/` | Rails controllers (admin/, events/, home/) | ~111 files | | `app/views/` | ERB templates | ~824 files | | `app/decorators/` | Draper decorators for view logic | ~60 files | | `app/policies/` | ActionPolicy authorization rules | ~75 files | @@ -223,7 +223,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `Analytics::EventReferenceLoader` — Batch-loads the records referenced in a page of Ahoy event properties (association changes, associated records), one query per type, so the activity table's Details column links each to its show page without an N+1 - `Analytics::ContentCreationBreakdown` — Counts content created in the selected time range for the Ahoy charts page's Total / User-generated / Admin-generated cards. Each record is credited by its author's account (super_user → admin), falling back to the creating account when no author is set, so promoted ideas and admin-entered facilitator content land on the user side and the two cards always sum to the total -- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled Leader (folds via `AffiliationServices::FoldLegacyLeaders`; one overlapping a Facilitator row with different dates stays listed), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) +- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled Leader (report-only; Review → the Leader affiliations page), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) - `DataHealth::Worklists` — Counts for review pages that live elsewhere (possible duplicates per dedupe page via each `DuplicateFinder` / `Dedupable.name_groups`, and author credit divergence sections, each linked to its section via `?section=` which renders the divergence results with the page so the anchor can scroll), rendered in Data health's lazy `data_health_worklists` frame because the finders load records in Ruby - `ResourceServices::ResolveOrganizations` — Links resources to an organization from the legacy free-text `agency` name (case-insensitive exact match, else creates one with status "Unknown"). Idempotent; drives both the `data:resolve_resource_organizations` rake task (dry-run by default) and the Data health repair ### Business Logic @@ -331,7 +331,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationServices::FoldLegacyLeaders` — Folds the old app's "Leader" affiliations (a Facilitator who is a primary contact) into Facilitator: a Leader row with a same-dates Facilitator twin merges into it via `ModelDeduper` (twin turns primary contact on and takes the Leader row's FileMaker codes and attachments), one with no Facilitator row in its stretch is renamed Facilitator with primary contact on, and one overlapping a Facilitator row with different dates is left; each fold leaves a "Leader cleanup" comment. Run by the `LegacyLeaderAffiliations` Data health fix +- `AffiliationServices::FoldLegacyLeaders` — The review behind the admin **Leader affiliations** page (`Admin::LeaderAffiliationsController`: index → confirm → perform, `/admin/leader_affiliations`, linked from the `LegacyLeaderAffiliations` Data health check). One `Row` per affiliation still titled Leader (the old app's Facilitator who is a primary contact) with its Facilitator twin (same dates, else the nearest overlapping one) and per-row choices: merge into the twin via `ModelDeduper` (twin turns primary contact on and takes the Leader row's FileMaker codes and attachments), rename to Facilitator, rename and end, delete, or keep. Suggests merge for a same-dates twin, rename when none covers its stretch, keep when one overlaps with different dates. `#planned_changes(outcome:)` previews, `#apply(outcome:)` performs; every change but delete leaves a "Leader cleanup" comment - `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Shared by the FileMaker affiliations import and `FoldLegacyLeaders` - `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row; an existing Facilitator row matches only for the same stretch (overlapping periods, `#same_stint`), so separate stints stay separate (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing "Leader"-titled row matches as a Facilitator; a row naming a project but no person gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) diff --git a/app/controllers/admin/leader_affiliations_controller.rb b/app/controllers/admin/leader_affiliations_controller.rb new file mode 100644 index 0000000000..88ada7e937 --- /dev/null +++ b/app/controllers/admin/leader_affiliations_controller.rb @@ -0,0 +1,39 @@ +module Admin + # Settles the old app's "Leader" affiliations one row at a time: index (choose) → + # confirm (preview, no writes) → create (perform). AffiliationServices::FoldLegacyLeaders + # holds the rules. + class LeaderAffiliationsController < ApplicationController + def index + authorize! :data_health + + @rows = AffiliationServices::FoldLegacyLeaders.new.rows + @pre_outcome = outcome_params + end + + def confirm + authorize! :data_health, to: :repair? + + @outcome = outcome_params + @changes = AffiliationServices::FoldLegacyLeaders.new.planned_changes(outcome: @outcome) + redirect_to admin_leader_affiliations_path(outcome: @outcome), notice: "Nothing selected to change." if @changes.empty? + end + + def create + authorize! :data_health, to: :repair? + + changed = AffiliationServices::FoldLegacyLeaders.new.apply(outcome: outcome_params) + redirect_to admin_leader_affiliations_path, notice: "Changed #{changed} Leader #{'affiliation'.pluralize(changed)}." + end + + private + + # Dynamic keys, so read as a plain string hash (never mass-assigned); the service only + # acts on a row's known choices. + def outcome_params + raw = params[:outcome] + return {} unless raw.respond_to?(:each_pair) + + raw.each_pair.to_h { |key, value| [ key.to_s, value.to_s ] } + end + end +end diff --git a/app/services/affiliation_services/fold_legacy_leaders.rb b/app/services/affiliation_services/fold_legacy_leaders.rb index d434c40de3..05201db5e7 100644 --- a/app/services/affiliation_services/fold_legacy_leaders.rb +++ b/app/services/affiliation_services/fold_legacy_leaders.rb @@ -1,55 +1,98 @@ module AffiliationServices - # Folds the old app's "Leader" affiliations into Facilitator ones. A Leader is a - # Facilitator who is a primary contact — the shape the January 2026 conversion gave most - # of them, beside the Leader row it left behind — but a row titled Leader counts toward - # no program status. Each row: - # * with a Facilitator twin over the same dates merges into it: the twin turns primary - # contact on, takes the Leader row's FileMaker codes and anything attached to it, and - # the Leader row goes - # * with no Facilitator row in its stretch becomes one, primary contact on - # * overlapping a Facilitator row with different dates is left for an admin, since which - # dates are right is a call made per row - # Either fold leaves a comment on the Facilitator row saying it was a Leader. Returns how - # many it folded. + # The review behind the Leader affiliations page (index → confirm → perform). A Leader is + # the old app's Facilitator who is a primary contact, but a row titled Leader counts + # toward no program status. Each row offers: + # * merge — into its Facilitator twin: the twin turns primary contact on and takes the + # Leader row's FileMaker codes and anything attached to it, and the Leader + # row goes + # * rename — to Facilitator, primary contact on + # * end — rename it and end it today, kept on the record without counting as active + # * delete — remove the Leader row outright + # * keep — leave it + # The suggestion is merge when a Facilitator row has the same dates, rename when none + # covers its stretch, and keep when one overlaps with different dates — which dates are + # right is the admin's call. Every change but delete leaves a comment saying it was a Leader. class FoldLegacyLeaders LEGACY_TITLE = "Leader".freeze COMMENT_TOPIC = "Leader cleanup".freeze + Row = Data.define(:leader, :twin, :same_dates) do + def key = leader.id.to_s + + def choices = twin ? %w[merge rename end delete keep] : %w[rename end delete keep] + + def suggestion + return "merge" if same_dates + return "keep" if twin + + "rename" + end + end + + Change = Data.define(:row, :action) + def self.legacy = Affiliation.where(title: LEGACY_TITLE) - def call - self.class.legacy.to_a.count { |leader| fold(leader) } + def rows + @rows ||= self.class.legacy.includes(:person, :organization).order(:organization_id, :person_id, :id).map { |leader| row_for(leader) } + end + + def planned_changes(outcome:) + rows.filter_map do |row| + action = outcome.to_h[row.key].presence_in(row.choices) + Change.new(row:, action:) if action && action != "keep" + end + end + + def apply(outcome:) + planned_changes(outcome:).count { |change| perform(change) } end private - def fold(leader) + # The twin is the Facilitator row with the same dates, else the one overlapping the + # Leader row's stretch with the nearest start. + def row_for(leader) facilitators = Affiliation.facilitators.where(person_id: leader.person_id, organization_id: leader.organization_id).to_a - twin = facilitators.find { |affiliation| affiliation.start_date == leader.start_date && affiliation.end_date == leader.end_date } - return merge(leader, twin) if twin - + same = facilitators.find { |affiliation| affiliation.start_date == leader.start_date && affiliation.end_date == leader.end_date } stint = AffiliationStint.of(leader) - return false if facilitators.any? { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } + overlapping = facilitators.select { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } + .min_by { |affiliation| leader.start_date && affiliation.start_date ? (affiliation.start_date - leader.start_date).abs : Float::INFINITY } + Row.new(leader:, twin: same || overlapping, same_dates: same.present?) + end - rename(leader) - rescue ActiveRecord::RecordInvalid, ActiveRecord::RecordNotSaved + def perform(change) + leader = change.row.leader + case change.action + when "merge" then merge(leader, change.row.twin) + when "rename" then rename(leader) + when "end" then end_affiliation(leader) + when "delete" then leader.destroy! + end + true + rescue ActiveRecord::RecordInvalid, ActiveRecord::RecordNotSaved, ActiveRecord::RecordNotDestroyed false end def merge(leader, twin) Affiliation.transaction do - twin.update!(primary_contact: true, filemaker_code: Affiliation.join_filemaker_codes(twin.filemaker_code, leader.filemaker_code), + # Two Leader rows can share a twin, so read it fresh rather than as this row loaded it. + twin.reload.update!(primary_contact: true, filemaker_code: Affiliation.join_filemaker_codes(twin.filemaker_code, leader.filemaker_code), organization_address_id: twin.organization_address_id || leader.organization_address_id) deduper.merge(twin, leader) note(twin, "Also recorded as a Leader affiliation (##{leader.id}), merged into this one.") end - true end def rename(leader) leader.update!(title: Affiliation::FACILITATOR_TITLE, primary_contact: true) note(leader, "Titled Leader until renamed Facilitator, primary contact.") - true + end + + def end_affiliation(leader) + leader.update!(title: Affiliation::FACILITATOR_TITLE, primary_contact: true, end_date: leader.end_date || Date.current, + inactive: true, inactive_supplied: true) + note(leader, "Titled Leader until renamed Facilitator, primary contact, and ended.") end def note(affiliation, body) diff --git a/app/services/data_health/legacy_leader_affiliations.rb b/app/services/data_health/legacy_leader_affiliations.rb index edfb4445ba..b7ea5af961 100644 --- a/app/services/data_health/legacy_leader_affiliations.rb +++ b/app/services/data_health/legacy_leader_affiliations.rb @@ -1,34 +1,21 @@ module DataHealth # Affiliations still titled Leader, the old app's word for a Facilitator who is a - # primary contact. AffiliationServices::FoldLegacyLeaders folds them in; a row it leaves - # is one overlapping a Facilitator row with different dates. + # primary contact. Settled row by row on the Leader affiliations page. class LegacyLeaderAffiliations < Check def title = "Affiliations still titled Leader" def explanation "A Leader is a Facilitator who is a primary contact, but a row titled Leader counts toward " \ - "no program status. Fixing merges each into the Facilitator row with the same dates, or " \ - "renames it Facilitator when none covers its stretch, leaving a comment either way. One that " \ - "overlaps a Facilitator row with different dates stays here to settle by hand." + "no program status. Review them to merge each into its Facilitator row, rename it, end it, " \ + "or delete it — previewing every change before it's made." end def scope = AffiliationServices::FoldLegacyLeaders.legacy.includes(:person, :organization) - def repairable? = true - - def repair_label = "Fold into Facilitator" - - def repaired_message(number) - "Folded #{number} Leader #{'affiliation'.pluralize(number)} into Facilitator." - end - - def repair! - AffiliationServices::FoldLegacyLeaders.new.call - end + def review_path = admin_leader_affiliations_path def describe(affiliation) - dates = [ affiliation.start_date, affiliation.end_date ].compact.map(&:to_s).join(" – ").presence - "#{affiliation.person&.name} — #{affiliation.organization&.name}#{" (#{dates})" if dates}" + "#{affiliation.person&.name} — #{affiliation.organization&.name}" end end end diff --git a/app/views/admin/leader_affiliations/confirm.html.erb b/app/views/admin/leader_affiliations/confirm.html.erb new file mode 100644 index 0000000000..0d5505500c --- /dev/null +++ b/app/views/admin/leader_affiliations/confirm.html.erb @@ -0,0 +1,52 @@ +<% content_for(:page_title, "Confirm Leader affiliation changes") %> +<% content_for(:page_bg_class, "admin-only bg-blue-100") %> +<% sections = { + "merge" => [ "Merge into Facilitator", "bg-blue-50 text-blue-800 border-blue-200", "The Facilitator row turns primary contact on and takes the Leader row's FileMaker codes and anything attached to it. The Leader row is deleted." ], + "rename" => [ "Rename to Facilitator", "bg-blue-50 text-blue-800 border-blue-200", "The row becomes a Facilitator, primary contact on." ], + "end" => [ "Rename and end", "bg-red-50 text-red-800 border-red-200", "The row becomes a Facilitator, primary contact on, ended today (or on its own end date)." ], + "delete" => [ "Delete", "bg-red-100 text-red-900 border-red-300", "The Leader row and its FileMaker codes are permanently deleted." ] + } %> +
+
+ <%= link_to "← Go back to edit", admin_leader_affiliations_path(outcome: @outcome), class: "text-sm #{eyebrow_link_class}" %> +
+ +

Confirm Leader affiliation changes

+

+ Performing will make the <%= @changes.size %> <%= "change".pluralize(@changes.size) %> below. Nothing else is affected. +

+ +
+ <% sections.each do |action, (label, header_class, description)| %> + <% action_changes = @changes.select { |change| change.action == action } %> + <% next if action_changes.empty? %> +
+
+ <%= label %> (<%= action_changes.size %>) +

<%= description %>

+
+
    + <% action_changes.each do |change| %> + <% leader = change.row.leader %> +
  • + <%= leader.person.name %> + — <%= leader.organization.name %> + · <%= leader.decorate.period_label.presence || "No dates" %> + <% if action == "merge" %> + → Facilitator #<%= change.row.twin.id %> + <% end %> +
    • + <% end %> +
+
+ <% end %> +
+ +
+ <%= link_to "← Go back to edit", admin_leader_affiliations_path(outcome: @outcome), class: "text-sm #{eyebrow_link_class}" %> + <%= form_with url: admin_perform_leader_affiliations_path, method: :post do %> + <% @outcome.each do |key, value| %><%= hidden_field_tag "outcome[#{key}]", value %><% end %> + <%= submit_tag "Perform changes", class: button_classes(:primary) %> + <% end %> +
+
diff --git a/app/views/admin/leader_affiliations/index.html.erb b/app/views/admin/leader_affiliations/index.html.erb new file mode 100644 index 0000000000..fda46fbab3 --- /dev/null +++ b/app/views/admin/leader_affiliations/index.html.erb @@ -0,0 +1,79 @@ +<% content_for(:page_title, "Leader affiliations") %> +<% content_for(:page_bg_class, "admin-only bg-blue-100") %> +<% choice_labels = { "merge" => [ "Merge into Facilitator", :blue ], "rename" => [ "Rename to Facilitator", :blue ], + "end" => [ "Rename and end", :red ], "delete" => [ "Delete", :red ], "keep" => [ "Leave as is", :gray ] } %> +<% color_class = { + red: "has-[:checked]:bg-red-50 has-[:checked]:text-red-700 has-[:checked]:border-red-300", + blue: "has-[:checked]:bg-blue-50 has-[:checked]:text-blue-700 has-[:checked]:border-blue-300", + gray: "has-[:checked]:bg-gray-100 has-[:checked]:text-gray-800 has-[:checked]:border-gray-400" + } %> +<% button_base = "inline-flex items-center gap-2 rounded-md border border-gray-300 bg-white px-3 py-1.5 text-xs font-medium text-gray-500 shadow-sm cursor-pointer hover:bg-gray-50" %> +
+
+ <%= link_to "← Data health", data_health_return_path(DataHealth::LegacyLeaderAffiliations.key), class: "text-sm #{eyebrow_link_class}" %> +
+ +

Leader affiliations

+
+

+ The old app's Leader was a Facilitator who is a primary contact, but a row titled Leader counts + toward no program status. Pick what happens to each one — the suggested choice is preselected, and nothing changes + until you confirm on the next page. +

+
    +
  • Merge into Facilitator — the Facilitator row beside it turns primary contact on and takes its FileMaker codes and anything attached to it; the Leader row is deleted.
    • +
  • Rename to Facilitator — the row becomes a Facilitator, primary contact on.
    • +
  • Rename and end — the same, then ended today, so it stays on the record without counting as active.
    • +
  • Delete — the Leader row and its FileMaker codes are removed.
    • +
+

Every change except Delete leaves a comment on the affiliation saying it was a Leader.

+
+ + <% if @rows.empty? %> +
No affiliations are titled Leader.
+ <% else %> + <%= form_with url: admin_leader_affiliations_path, method: :post, data: { turbo: false } do %> +
+ <% @rows.each do |row| %> + <% leader = row.leader %> + <% chosen = @pre_outcome[row.key] || row.suggestion %> +
+
+
+ <%= link_to leader.person.name, edit_person_path(leader.person, anchor: dom_id(leader)), target: "_blank", rel: "noopener", + class: "font-medium text-gray-900 hover:text-blue-700 hover:underline" %> + — <%= leader.organization.name %> +
+
+ Leader · <%= leader.decorate.period_label.presence || "No dates" %><%= " · primary contact" if leader.primary_contact? %> + <% if leader.filemaker_code.present? %> · FileMaker <%= leader.filemaker_code %><% end %> +
+
"> + <% if row.twin %> + Facilitator #<%= row.twin.id %> · <%= row.twin.decorate.period_label.presence || "No dates" %><%= " · primary contact" if row.twin.primary_contact? %> + <%= row.same_dates ? "· same dates" : "· overlaps with different dates — check which dates are right" %> + <% else %> + No Facilitator row covers this stretch. + <% end %> +
+
+
+ <% row.choices.each do |value| %> + <% label, color = choice_labels[value] %> + + <% end %> +
+
+ <% end %> +
+ +
+ <%= link_to "Cancel", data_health_return_path(DataHealth::LegacyLeaderAffiliations.key), class: "text-sm #{eyebrow_link_class}" %> + <%= submit_tag "Preview changes", class: button_classes(:primary) %> +
+ <% end %> + <% end %> +
diff --git a/config/routes.rb b/config/routes.rb index 5fd77b9a13..d2e7298281 100644 --- a/config/routes.rb +++ b/config/routes.rb @@ -69,6 +69,9 @@ get "data_health/worklists", to: "data_health#worklists", as: "data_health_worklists" get "data_health/:check/rows", to: "data_health#rows", as: "data_health_rows" post "data_health/:check/repair", to: "data_health#repair", as: "data_health_repair" + get "leader_affiliations", to: "leader_affiliations#index", as: "leader_affiliations" + post "leader_affiliations", to: "leader_affiliations#confirm" + post "leader_affiliations/perform", to: "leader_affiliations#create", as: "perform_leader_affiliations" get "resource_roles", to: "resource_roles#show", as: "resource_roles" patch "resource_roles", to: "resource_roles#update" get "certificate_settings", to: redirect("/admin/resource_roles") diff --git a/db/seeds/features/2026-10-24-fold-leader-affiliations.yml b/db/seeds/features/2026-10-24-fold-leader-affiliations.yml deleted file mode 100644 index f85d5dc315..0000000000 --- a/db/seeds/features/2026-10-24-fold-leader-affiliations.yml +++ /dev/null @@ -1,14 +0,0 @@ -name: "Fold leftover Leader affiliations into Facilitator" -area: people -display_status: admin_facing -released_on: 2026-10-24 -pr_number: 2788 -summary: >- - Data health lists affiliations still titled Leader, which don't count toward - program status, and one click folds them into Facilitator affiliations marked - as primary contact. -pro_tips: - - "A Leader row with a Facilitator row for the same dates merges into it, so no duplicate is left behind." - - "A Leader row whose dates overlap a Facilitator row's but don't match stays listed for you to settle by hand." - - "Each folded affiliation gets a comment saying it was a Leader." -action_path: "/admin/data_health" diff --git a/db/seeds/features/2026-10-24-review-leader-affiliations.yml b/db/seeds/features/2026-10-24-review-leader-affiliations.yml new file mode 100644 index 0000000000..ad981afed2 --- /dev/null +++ b/db/seeds/features/2026-10-24-review-leader-affiliations.yml @@ -0,0 +1,15 @@ +name: "Review leftover Leader affiliations" +area: people +display_status: admin_facing +released_on: 2026-10-24 +pr_number: 2788 +summary: >- + A review page lists affiliations still titled Leader, which don't count toward + program status, so you can merge, rename, end, or delete each one — previewing + every change before it's made. +pro_tips: + - "Open it from Data health → Affiliations still titled Leader → Review." + - "A Leader row with a Facilitator row for the same dates suggests merging into it, so no duplicate is left behind." + - "A Leader row whose dates overlap a Facilitator row's but don't match is left as is unless you pick a change." + - "Every change except Delete leaves a comment saying the affiliation was a Leader." +action_path: "/admin/leader_affiliations" diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 5180951556..d8fa588024 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -77,8 +77,10 @@ Constraints already decided elsewhere: contacts (402 aren't), all 53 Liaisons are, and 111 rows are still titled "Leader" (which the exact-match Facilitator scope doesn't count, and which the import matches as a Facilitator rather than duplicating). Of those, 109 sit beside an undated, primary - Facilitator twin the January conversion made, and 2 have none; the Data health check - "Affiliations still titled Leader" folds them in (`AffiliationServices::FoldLegacyLeaders`). + Facilitator twin the January conversion made, and 2 have none. The admin Leader + affiliations page (linked from the Data health check "Affiliations still titled Leader") + settles them row by row: merge into the twin, rename, rename and end, delete, or keep, + previewed before anything changes (`AffiliationServices::FoldLegacyLeaders`). - Every role becomes a **Facilitator** affiliation, with a "FileMaker import" comment recording the role FileMaker gave. No prod row is titled Assistant, so the old app's assistants likely became the 402 Facilitators who aren't primary contacts. diff --git a/spec/requests/admin/leader_affiliations_spec.rb b/spec/requests/admin/leader_affiliations_spec.rb new file mode 100644 index 0000000000..b9ff20ca29 --- /dev/null +++ b/spec/requests/admin/leader_affiliations_spec.rb @@ -0,0 +1,51 @@ +require "rails_helper" + +RSpec.describe "Admin::LeaderAffiliations", type: :request do + let(:admin) { create(:user, :admin) } + let(:person) { create(:person) } + let(:organization) { create(:organization) } + let!(:twin) { create(:affiliation, person:, organization:, title: "Facilitator") } + let!(:leader) { create(:affiliation, person:, organization:, title: "Leader") } + + it "keeps non-admins out" do + sign_in create(:user) + + get admin_leader_affiliations_path + + expect(response).to redirect_to(root_path) + end + + describe "as an admin" do + before { sign_in admin } + + it "lists each Leader row with its suggested choice selected" do + get admin_leader_affiliations_path + + expect(response).to have_http_status(:ok) + expect(response.body).to include("Facilitator ##{twin.id}") + expect(response.body).to match(/id="outcome_#{leader.id}_merge"[^>]*checked|checked[^>]*id="outcome_#{leader.id}_merge"/) + end + + it "previews the chosen changes without making them" do + post admin_leader_affiliations_path, params: { outcome: { leader.id.to_s => "merge" } } + + expect(response).to have_http_status(:ok) + expect(response.body).to include("Merge into Facilitator (1)") + expect(Affiliation.exists?(leader.id)).to be(true) + end + + it "sends an empty choice back to the page" do + post admin_leader_affiliations_path, params: { outcome: { leader.id.to_s => "keep" } } + + expect(response).to redirect_to(admin_leader_affiliations_path(outcome: { leader.id.to_s => "keep" })) + end + + it "performs the confirmed changes" do + post admin_perform_leader_affiliations_path, params: { outcome: { leader.id.to_s => "merge" } } + + expect(response).to redirect_to(admin_leader_affiliations_path) + expect(Affiliation.exists?(leader.id)).to be(false) + expect(twin.reload.primary_contact).to be(true) + end + end +end diff --git a/spec/routing/admin_leader_affiliations_routing_spec.rb b/spec/routing/admin_leader_affiliations_routing_spec.rb new file mode 100644 index 0000000000..3ce7d61907 --- /dev/null +++ b/spec/routing/admin_leader_affiliations_routing_spec.rb @@ -0,0 +1,9 @@ +require "rails_helper" + +RSpec.describe Admin::LeaderAffiliationsController, type: :routing do + it "routes the review, its preview, and performing it" do + expect(get: "/admin/leader_affiliations").to route_to("admin/leader_affiliations#index") + expect(post: "/admin/leader_affiliations").to route_to("admin/leader_affiliations#confirm") + expect(post: "/admin/leader_affiliations/perform").to route_to("admin/leader_affiliations#create") + end +end diff --git a/spec/services/affiliation_services/fold_legacy_leaders_spec.rb b/spec/services/affiliation_services/fold_legacy_leaders_spec.rb index d9d488b5dd..eabbfe39bc 100644 --- a/spec/services/affiliation_services/fold_legacy_leaders_spec.rb +++ b/spec/services/affiliation_services/fold_legacy_leaders_spec.rb @@ -10,47 +10,97 @@ def affiliation(title, **attributes) create(:affiliation, person:, organization:, title:, **attributes) end - it "merges a Leader row into its same-dates Facilitator twin, carrying its codes and comments" do - twin = affiliation("Facilitator", primary_contact: false, filemaker_code: "P0001") - leader = affiliation("Leader", primary_contact: true, filemaker_code: "PS0001") - leader.comments.create!(body: "Met at the 2019 training") + def row_for(leader) = described_class.new.rows.find { |row| row.leader == leader } - expect(described_class.new.call).to eq(1) + def apply(outcome) = described_class.new.apply(outcome: outcome.transform_keys { |leader| leader.id.to_s }) - expect(Affiliation.exists?(leader.id)).to be(false) - expect(twin.reload).to have_attributes(primary_contact: true, filemaker_code: "P0001, PS0001") - expect(twin.comments.pluck(:body)).to contain_exactly("Met at the 2019 training", a_string_including("Leader", "##{leader.id}")) - end + describe "#rows" do + it "suggests merging into a Facilitator twin with the same dates" do + twin = affiliation("Facilitator") + leader = affiliation("Leader") + + expect(row_for(leader)).to have_attributes(twin:, same_dates: true, suggestion: "merge") + end - it "merges two Leader rows sharing one twin into it" do - twin = affiliation("Facilitator", primary_contact: true) - affiliation("Leader", primary_contact: true) - # The model refuses an identical row now; prod's pairs predate that. - affiliation("Placeholder", primary_contact: true).update_column(:title, "Leader") + it "suggests renaming when no Facilitator row covers its stretch" do + affiliation("Facilitator", start_date: Date.new(2010, 1, 1), end_date: Date.new(2012, 12, 31)) + leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) - described_class.new.call + row = row_for(leader) + expect(row).to have_attributes(twin: nil, suggestion: "rename") + expect(row.choices).not_to include("merge") + end - expect(person.affiliations.reload).to contain_exactly(twin) - expect(twin.comments.where(topic: described_class::COMMENT_TOPIC).count).to eq(2) + it "suggests leaving one that overlaps a Facilitator row with different dates" do + twin = affiliation("Facilitator", start_date: Date.new(2018, 1, 1), end_date: Date.new(2022, 12, 31)) + leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) + + expect(row_for(leader)).to have_attributes(twin:, same_dates: false, suggestion: "keep") + end end - it "renames a Leader row with no Facilitator row in its stretch" do - affiliation("Facilitator", start_date: Date.new(2010, 1, 1), end_date: Date.new(2012, 12, 31)) - leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1), primary_contact: false) + describe "#planned_changes" do + it "plans only the rows given a choice they offer, skipping keep" do + leader = affiliation("Leader") + other = affiliation("Leader", start_date: Date.new(2019, 1, 1)) - expect(described_class.new.call).to eq(1) + changes = described_class.new.planned_changes(outcome: { leader.id.to_s => "merge", other.id.to_s => "keep" }) - expect(leader.reload).to have_attributes(title: "Facilitator", primary_contact: true) - expect(leader.comments.sole).to have_attributes(topic: described_class::COMMENT_TOPIC, body: a_string_including("Titled Leader")) + expect(changes).to be_empty + expect(described_class.new.planned_changes(outcome: { leader.id.to_s => "rename" }).map(&:action)).to eq([ "rename" ]) + end end - it "leaves a Leader row overlapping a Facilitator row with different dates for an admin" do - affiliation("Facilitator", start_date: Date.new(2018, 1, 1), end_date: Date.new(2022, 12, 31)) - leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) + describe "#apply" do + it "merges a Leader row into its twin, carrying its codes and comments" do + twin = affiliation("Facilitator", primary_contact: false, filemaker_code: "P0001") + leader = affiliation("Leader", primary_contact: true, filemaker_code: "PS0001") + leader.comments.create!(body: "Met at the 2019 training") + + expect(apply(leader => "merge")).to eq(1) + + expect(Affiliation.exists?(leader.id)).to be(false) + expect(twin.reload).to have_attributes(primary_contact: true, filemaker_code: "P0001, PS0001") + expect(twin.comments.pluck(:body)).to contain_exactly("Met at the 2019 training", a_string_including("Leader", "##{leader.id}")) + end + + it "merges two Leader rows sharing one twin, keeping both codes" do + twin = affiliation("Facilitator", primary_contact: true) + first = affiliation("Leader", filemaker_code: "P0001") + # The model refuses an identical row now; prod's pairs predate that. + second = affiliation("Placeholder", filemaker_code: "P0002").tap { |row| row.update_column(:title, "Leader") } + + apply(first => "merge", second => "merge") + + expect(person.affiliations.reload).to contain_exactly(twin) + expect(twin.reload.filemaker_code).to eq("P0001, P0002") + end + + it "renames a Leader row to a primary-contact Facilitator" do + leader = affiliation("Leader", primary_contact: false) + + apply(leader => "rename") + + expect(leader.reload).to have_attributes(title: "Facilitator", primary_contact: true) + expect(leader.comments.sole).to have_attributes(topic: described_class::COMMENT_TOPIC, body: a_string_including("Titled Leader")) + end + + it "renames and ends one, keeping an end date it already has" do + open_leader = affiliation("Leader") + dated = affiliation("Leader", start_date: Date.new(2019, 1, 1), end_date: Date.new(2020, 1, 1)) + + apply(open_leader => "end", dated => "end") + + expect(open_leader.reload).to have_attributes(title: "Facilitator", end_date: Date.current, inactive: true) + expect(dated.reload).to have_attributes(title: "Facilitator", end_date: Date.new(2020, 1, 1), inactive: true) + end + + it "deletes one outright" do + leader = affiliation("Leader") - expect(described_class.new.call).to eq(0) + apply(leader => "delete") - expect(leader.reload.title).to eq("Leader") - expect(leader.comments).to be_empty + expect(Affiliation.exists?(leader.id)).to be(false) + end end end diff --git a/spec/services/data_health/checks_spec.rb b/spec/services/data_health/checks_spec.rb index 9e26f5a523..79585af7d4 100644 --- a/spec/services/data_health/checks_spec.rb +++ b/spec/services/data_health/checks_spec.rb @@ -124,17 +124,14 @@ def affiliation_from(facilitator_training:, title: "Facilitator") end RSpec.describe DataHealth::LegacyLeaderAffiliations do - let(:person) { create(:person) } - let(:organization) { create(:organization) } - - it "finds affiliations still titled Leader, and folds them into Facilitator" do - leader = create(:affiliation, person:, organization:, title: "Leader") - create(:affiliation, person:, organization:, title: "Facilitator") + it "finds affiliations still titled Leader and sends them to the review page" do + leader = create(:affiliation, title: "Leader") + create(:affiliation, title: "Facilitator") check = described_class.new expect(check.scope).to contain_exactly(leader) - expect(check.repair!).to eq(1) - expect(described_class.new).not_to be_any + expect(check).not_to be_repairable + expect(check.review_path).to eq("/admin/leader_affiliations") end end diff --git a/spec/views/page_bg_class_alignment_spec.rb b/spec/views/page_bg_class_alignment_spec.rb index 13caf0ab98..0f4484c0cd 100644 --- a/spec/views/page_bg_class_alignment_spec.rb +++ b/spec/views/page_bg_class_alignment_spec.rb @@ -134,6 +134,8 @@ "app/views/people/email_addresses.html.erb" => "admin-only bg-blue-100", "app/views/comments_and_communications/index.html.erb" => "admin-only bg-blue-100", "app/views/admin/data_health/index.html.erb" => "admin-only bg-blue-100", + "app/views/admin/leader_affiliations/index.html.erb" => "admin-only bg-blue-100", + "app/views/admin/leader_affiliations/confirm.html.erb" => "admin-only bg-blue-100", "app/views/admin/resource_roles/show.html.erb" => "admin-only bg-blue-100", "app/views/comments/index.html.erb" => "admin-only bg-blue-100", "app/views/bookmarks/index.html.erb" => "admin-only bg-blue-100", From 9820107d4930a3975fe1149018d4ad277d5c62e2 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 20:42:12 -0400 Subject: [PATCH 16/25] Only report Leader affiliations; leave the cleanup to person edit Changing a person's affiliations should happen where an admin can see the whole person, not from a bulk page. Data health lists each row still titled Leader beside the Facilitator row that covers it, if any, and opens the person's edit page at that affiliation, with the back link returning to the check. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 7 +- .../admin/leader_affiliations_controller.rb | 39 ------- .../fold_legacy_leaders.rb | 106 ------------------ .../import_from_filemaker.rb | 2 +- .../data_health/legacy_leader_affiliations.rb | 27 ++++- .../data_health/_legacy_leader_row.html.erb | 24 ++++ .../leader_affiliations/confirm.html.erb | 52 --------- .../admin/leader_affiliations/index.html.erb | 79 ------------- config/routes.rb | 3 - ...-24-leader-affiliations-in-data-health.yml | 14 +++ .../2026-10-24-review-leader-affiliations.yml | 15 --- ...maker-organization-and-personnel-import.md | 7 +- spec/requests/admin/data_health_spec.rb | 15 +++ .../admin/leader_affiliations_spec.rb | 51 --------- .../admin_leader_affiliations_routing_spec.rb | 9 -- .../fold_legacy_leaders_spec.rb | 106 ------------------ spec/services/data_health/checks_spec.rb | 36 ++++-- spec/views/page_bg_class_alignment_spec.rb | 2 - 18 files changed, 111 insertions(+), 483 deletions(-) delete mode 100644 app/controllers/admin/leader_affiliations_controller.rb delete mode 100644 app/services/affiliation_services/fold_legacy_leaders.rb create mode 100644 app/views/admin/data_health/_legacy_leader_row.html.erb delete mode 100644 app/views/admin/leader_affiliations/confirm.html.erb delete mode 100644 app/views/admin/leader_affiliations/index.html.erb create mode 100644 db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml delete mode 100644 db/seeds/features/2026-10-24-review-leader-affiliations.yml delete mode 100644 spec/requests/admin/leader_affiliations_spec.rb delete mode 100644 spec/routing/admin_leader_affiliations_routing_spec.rb delete mode 100644 spec/services/affiliation_services/fold_legacy_leaders_spec.rb diff --git a/AGENTS.md b/AGENTS.md index 74501bc836..eb5bb480de 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,7 +57,7 @@ This codebase (Rails 8.1) | Directory | Purpose | Count | |---|---|---| -| `app/controllers/` | Rails controllers (admin/, events/, home/) | ~111 files | +| `app/controllers/` | Rails controllers (admin/, events/, home/) | ~110 files | | `app/views/` | ERB templates | ~824 files | | `app/decorators/` | Draper decorators for view logic | ~60 files | | `app/policies/` | ActionPolicy authorization rules | ~75 files | @@ -223,7 +223,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `Analytics::EventReferenceLoader` — Batch-loads the records referenced in a page of Ahoy event properties (association changes, associated records), one query per type, so the activity table's Details column links each to its show page without an N+1 - `Analytics::ContentCreationBreakdown` — Counts content created in the selected time range for the Ahoy charts page's Total / User-generated / Admin-generated cards. Each record is credited by its author's account (super_user → admin), falling back to the creating account when no author is set, so promoted ideas and admin-entered facilitator content land on the user side and the two cards always sum to the total -- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled Leader (report-only; Review → the Leader affiliations page), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) +- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled Leader (report-only; each row shows its Facilitator twin via `#twin_for` and opens the person's edit page at the affiliation), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) - `DataHealth::Worklists` — Counts for review pages that live elsewhere (possible duplicates per dedupe page via each `DuplicateFinder` / `Dedupable.name_groups`, and author credit divergence sections, each linked to its section via `?section=` which renders the divergence results with the page so the anchor can scroll), rendered in Data health's lazy `data_health_worklists` frame because the finders load records in Ruby - `ResourceServices::ResolveOrganizations` — Links resources to an organization from the legacy free-text `agency` name (case-insensitive exact match, else creates one with status "Unknown"). Idempotent; drives both the `data:resolve_resource_organizations` rake task (dry-run by default) and the Data health repair ### Business Logic @@ -331,8 +331,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationServices::FoldLegacyLeaders` — The review behind the admin **Leader affiliations** page (`Admin::LeaderAffiliationsController`: index → confirm → perform, `/admin/leader_affiliations`, linked from the `LegacyLeaderAffiliations` Data health check). One `Row` per affiliation still titled Leader (the old app's Facilitator who is a primary contact) with its Facilitator twin (same dates, else the nearest overlapping one) and per-row choices: merge into the twin via `ModelDeduper` (twin turns primary contact on and takes the Leader row's FileMaker codes and attachments), rename to Facilitator, rename and end, delete, or keep. Suggests merge for a same-dates twin, rename when none covers its stretch, keep when one overlaps with different dates. `#planned_changes(outcome:)` previews, `#apply(outcome:)` performs; every change but delete leaves a "Leader cleanup" comment -- `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Shared by the FileMaker affiliations import and `FoldLegacyLeaders` +- `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Shared by the FileMaker affiliations import and the `LegacyLeaderAffiliations` Data health check - `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row; an existing Facilitator row matches only for the same stretch (overlapping periods, `#same_stint`), so separate stints stay separate (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing "Leader"-titled row matches as a Facilitator; a row naming a project but no person gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row diff --git a/app/controllers/admin/leader_affiliations_controller.rb b/app/controllers/admin/leader_affiliations_controller.rb deleted file mode 100644 index 88ada7e937..0000000000 --- a/app/controllers/admin/leader_affiliations_controller.rb +++ /dev/null @@ -1,39 +0,0 @@ -module Admin - # Settles the old app's "Leader" affiliations one row at a time: index (choose) → - # confirm (preview, no writes) → create (perform). AffiliationServices::FoldLegacyLeaders - # holds the rules. - class LeaderAffiliationsController < ApplicationController - def index - authorize! :data_health - - @rows = AffiliationServices::FoldLegacyLeaders.new.rows - @pre_outcome = outcome_params - end - - def confirm - authorize! :data_health, to: :repair? - - @outcome = outcome_params - @changes = AffiliationServices::FoldLegacyLeaders.new.planned_changes(outcome: @outcome) - redirect_to admin_leader_affiliations_path(outcome: @outcome), notice: "Nothing selected to change." if @changes.empty? - end - - def create - authorize! :data_health, to: :repair? - - changed = AffiliationServices::FoldLegacyLeaders.new.apply(outcome: outcome_params) - redirect_to admin_leader_affiliations_path, notice: "Changed #{changed} Leader #{'affiliation'.pluralize(changed)}." - end - - private - - # Dynamic keys, so read as a plain string hash (never mass-assigned); the service only - # acts on a row's known choices. - def outcome_params - raw = params[:outcome] - return {} unless raw.respond_to?(:each_pair) - - raw.each_pair.to_h { |key, value| [ key.to_s, value.to_s ] } - end - end -end diff --git a/app/services/affiliation_services/fold_legacy_leaders.rb b/app/services/affiliation_services/fold_legacy_leaders.rb deleted file mode 100644 index 05201db5e7..0000000000 --- a/app/services/affiliation_services/fold_legacy_leaders.rb +++ /dev/null @@ -1,106 +0,0 @@ -module AffiliationServices - # The review behind the Leader affiliations page (index → confirm → perform). A Leader is - # the old app's Facilitator who is a primary contact, but a row titled Leader counts - # toward no program status. Each row offers: - # * merge — into its Facilitator twin: the twin turns primary contact on and takes the - # Leader row's FileMaker codes and anything attached to it, and the Leader - # row goes - # * rename — to Facilitator, primary contact on - # * end — rename it and end it today, kept on the record without counting as active - # * delete — remove the Leader row outright - # * keep — leave it - # The suggestion is merge when a Facilitator row has the same dates, rename when none - # covers its stretch, and keep when one overlaps with different dates — which dates are - # right is the admin's call. Every change but delete leaves a comment saying it was a Leader. - class FoldLegacyLeaders - LEGACY_TITLE = "Leader".freeze - COMMENT_TOPIC = "Leader cleanup".freeze - - Row = Data.define(:leader, :twin, :same_dates) do - def key = leader.id.to_s - - def choices = twin ? %w[merge rename end delete keep] : %w[rename end delete keep] - - def suggestion - return "merge" if same_dates - return "keep" if twin - - "rename" - end - end - - Change = Data.define(:row, :action) - - def self.legacy = Affiliation.where(title: LEGACY_TITLE) - - def rows - @rows ||= self.class.legacy.includes(:person, :organization).order(:organization_id, :person_id, :id).map { |leader| row_for(leader) } - end - - def planned_changes(outcome:) - rows.filter_map do |row| - action = outcome.to_h[row.key].presence_in(row.choices) - Change.new(row:, action:) if action && action != "keep" - end - end - - def apply(outcome:) - planned_changes(outcome:).count { |change| perform(change) } - end - - private - - # The twin is the Facilitator row with the same dates, else the one overlapping the - # Leader row's stretch with the nearest start. - def row_for(leader) - facilitators = Affiliation.facilitators.where(person_id: leader.person_id, organization_id: leader.organization_id).to_a - same = facilitators.find { |affiliation| affiliation.start_date == leader.start_date && affiliation.end_date == leader.end_date } - stint = AffiliationStint.of(leader) - overlapping = facilitators.select { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } - .min_by { |affiliation| leader.start_date && affiliation.start_date ? (affiliation.start_date - leader.start_date).abs : Float::INFINITY } - Row.new(leader:, twin: same || overlapping, same_dates: same.present?) - end - - def perform(change) - leader = change.row.leader - case change.action - when "merge" then merge(leader, change.row.twin) - when "rename" then rename(leader) - when "end" then end_affiliation(leader) - when "delete" then leader.destroy! - end - true - rescue ActiveRecord::RecordInvalid, ActiveRecord::RecordNotSaved, ActiveRecord::RecordNotDestroyed - false - end - - def merge(leader, twin) - Affiliation.transaction do - # Two Leader rows can share a twin, so read it fresh rather than as this row loaded it. - twin.reload.update!(primary_contact: true, filemaker_code: Affiliation.join_filemaker_codes(twin.filemaker_code, leader.filemaker_code), - organization_address_id: twin.organization_address_id || leader.organization_address_id) - deduper.merge(twin, leader) - note(twin, "Also recorded as a Leader affiliation (##{leader.id}), merged into this one.") - end - end - - def rename(leader) - leader.update!(title: Affiliation::FACILITATOR_TITLE, primary_contact: true) - note(leader, "Titled Leader until renamed Facilitator, primary contact.") - end - - def end_affiliation(leader) - leader.update!(title: Affiliation::FACILITATOR_TITLE, primary_contact: true, end_date: leader.end_date || Date.current, - inactive: true, inactive_supplied: true) - note(leader, "Titled Leader until renamed Facilitator, primary contact, and ended.") - end - - def note(affiliation, body) - affiliation.comments.create!(topic: COMMENT_TOPIC, body:) - end - - def deduper - @deduper ||= ModelDeduper.new(model_class: Affiliation, logger: Rails.logger, dry_run: false) - end - end -end diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index fdd5637664..6c4cc0cc55 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -251,7 +251,7 @@ def records_another_row?(affiliation, codes) # Some prod rows still carry the old app's "Leader" title, which is a Facilitator. def same_title?(held, title) held = held.to_s.squish - held.casecmp?(title.to_s.squish) || (title == Affiliation::FACILITATOR_TITLE && held.casecmp?(FoldLegacyLeaders::LEGACY_TITLE)) + held.casecmp?(title.to_s.squish) || (title == Affiliation::FACILITATOR_TITLE && held.casecmp?(DataHealth::LegacyLeaderAffiliations::LEGACY_TITLE)) end def link(affiliation, row, label, title:, attributes:, codes:) diff --git a/app/services/data_health/legacy_leader_affiliations.rb b/app/services/data_health/legacy_leader_affiliations.rb index b7ea5af961..cdc698aee1 100644 --- a/app/services/data_health/legacy_leader_affiliations.rb +++ b/app/services/data_health/legacy_leader_affiliations.rb @@ -1,21 +1,38 @@ module DataHealth # Affiliations still titled Leader, the old app's word for a Facilitator who is a - # primary contact. Settled row by row on the Leader affiliations page. + # primary contact. Report-only: each row shows the Facilitator row beside it, if any, and + # opens the person's edit page to settle by hand. class LegacyLeaderAffiliations < Check + LEGACY_TITLE = "Leader".freeze + def title = "Affiliations still titled Leader" def explanation "A Leader is a Facilitator who is a primary contact, but a row titled Leader counts toward " \ - "no program status. Review them to merge each into its Facilitator row, rename it, end it, " \ - "or delete it — previewing every change before it's made." + "no program status. Each one is listed with the Facilitator row beside it; open the person to " \ + "merge, rename or remove it." end - def scope = AffiliationServices::FoldLegacyLeaders.legacy.includes(:person, :organization) + def scope + Affiliation.where(title: LEGACY_TITLE).includes(:person, :organization).order(:organization_id, :person_id, :id) + end - def review_path = admin_leader_affiliations_path + def row_partial = "admin/data_health/legacy_leader_row" def describe(affiliation) "#{affiliation.person&.name} — #{affiliation.organization&.name}" end + + # The Facilitator row recording the same stretch (AffiliationStint), the same dates first. + def twin_for(leader) + stint = AffiliationStint.of(leader) + Affiliation.facilitators.where(person_id: leader.person_id, organization_id: leader.organization_id) + .select { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } + .min_by { |affiliation| same_dates?(affiliation, leader) ? 0 : 1 } + end + + def same_dates?(affiliation, leader) + affiliation.start_date == leader.start_date && affiliation.end_date == leader.end_date + end end end diff --git a/app/views/admin/data_health/_legacy_leader_row.html.erb b/app/views/admin/data_health/_legacy_leader_row.html.erb new file mode 100644 index 0000000000..be30d82be7 --- /dev/null +++ b/app/views/admin/data_health/_legacy_leader_row.html.erb @@ -0,0 +1,24 @@ +<%# Locals: record (an Affiliation titled Leader). The person opens their edit page at the + affiliation; the line below says which Facilitator row, if any, records the same stretch. %> +<% check = DataHealth::LegacyLeaderAffiliations.new %> +<% twin = check.twin_for(record) %> +<% back = { return_to: "data_health", check: check.key } %> +
+ <% if record.person %> + <%= link_to person_name(record.person), edit_person_path(record.person, **back, anchor: dom_id(record)), target: "_blank", rel: "noopener", + class: "font-medium text-gray-800 hover:text-blue-700 hover:underline" %> + <% end %> + — <%= record.organization&.name %> +
+
+ Leader · <%= record.decorate.period_label.presence || "No dates" %><%= " · primary contact" if record.primary_contact? %> + <% if record.filemaker_code.present? %> · FileMaker <%= record.filemaker_code %><% end %> +
+
"> + <% if twin %> + Facilitator #<%= twin.id %> · <%= twin.decorate.period_label.presence || "No dates" %><%= " · primary contact" if twin.primary_contact? %> + · <%= check.same_dates?(twin, record) ? "same dates" : "overlaps with different dates" %> + <% else %> + No Facilitator row covers this stretch. + <% end %> +
diff --git a/app/views/admin/leader_affiliations/confirm.html.erb b/app/views/admin/leader_affiliations/confirm.html.erb deleted file mode 100644 index 0d5505500c..0000000000 --- a/app/views/admin/leader_affiliations/confirm.html.erb +++ /dev/null @@ -1,52 +0,0 @@ -<% content_for(:page_title, "Confirm Leader affiliation changes") %> -<% content_for(:page_bg_class, "admin-only bg-blue-100") %> -<% sections = { - "merge" => [ "Merge into Facilitator", "bg-blue-50 text-blue-800 border-blue-200", "The Facilitator row turns primary contact on and takes the Leader row's FileMaker codes and anything attached to it. The Leader row is deleted." ], - "rename" => [ "Rename to Facilitator", "bg-blue-50 text-blue-800 border-blue-200", "The row becomes a Facilitator, primary contact on." ], - "end" => [ "Rename and end", "bg-red-50 text-red-800 border-red-200", "The row becomes a Facilitator, primary contact on, ended today (or on its own end date)." ], - "delete" => [ "Delete", "bg-red-100 text-red-900 border-red-300", "The Leader row and its FileMaker codes are permanently deleted." ] - } %> -
-
- <%= link_to "← Go back to edit", admin_leader_affiliations_path(outcome: @outcome), class: "text-sm #{eyebrow_link_class}" %> -
- -

Confirm Leader affiliation changes

-

- Performing will make the <%= @changes.size %> <%= "change".pluralize(@changes.size) %> below. Nothing else is affected. -

- -
- <% sections.each do |action, (label, header_class, description)| %> - <% action_changes = @changes.select { |change| change.action == action } %> - <% next if action_changes.empty? %> -
-
- <%= label %> (<%= action_changes.size %>) -

<%= description %>

-
-
    - <% action_changes.each do |change| %> - <% leader = change.row.leader %> -
  • - <%= leader.person.name %> - — <%= leader.organization.name %> - · <%= leader.decorate.period_label.presence || "No dates" %> - <% if action == "merge" %> - → Facilitator #<%= change.row.twin.id %> - <% end %> -
    • - <% end %> -
-
- <% end %> -
- -
- <%= link_to "← Go back to edit", admin_leader_affiliations_path(outcome: @outcome), class: "text-sm #{eyebrow_link_class}" %> - <%= form_with url: admin_perform_leader_affiliations_path, method: :post do %> - <% @outcome.each do |key, value| %><%= hidden_field_tag "outcome[#{key}]", value %><% end %> - <%= submit_tag "Perform changes", class: button_classes(:primary) %> - <% end %> -
-
diff --git a/app/views/admin/leader_affiliations/index.html.erb b/app/views/admin/leader_affiliations/index.html.erb deleted file mode 100644 index fda46fbab3..0000000000 --- a/app/views/admin/leader_affiliations/index.html.erb +++ /dev/null @@ -1,79 +0,0 @@ -<% content_for(:page_title, "Leader affiliations") %> -<% content_for(:page_bg_class, "admin-only bg-blue-100") %> -<% choice_labels = { "merge" => [ "Merge into Facilitator", :blue ], "rename" => [ "Rename to Facilitator", :blue ], - "end" => [ "Rename and end", :red ], "delete" => [ "Delete", :red ], "keep" => [ "Leave as is", :gray ] } %> -<% color_class = { - red: "has-[:checked]:bg-red-50 has-[:checked]:text-red-700 has-[:checked]:border-red-300", - blue: "has-[:checked]:bg-blue-50 has-[:checked]:text-blue-700 has-[:checked]:border-blue-300", - gray: "has-[:checked]:bg-gray-100 has-[:checked]:text-gray-800 has-[:checked]:border-gray-400" - } %> -<% button_base = "inline-flex items-center gap-2 rounded-md border border-gray-300 bg-white px-3 py-1.5 text-xs font-medium text-gray-500 shadow-sm cursor-pointer hover:bg-gray-50" %> -
-
- <%= link_to "← Data health", data_health_return_path(DataHealth::LegacyLeaderAffiliations.key), class: "text-sm #{eyebrow_link_class}" %> -
- -

Leader affiliations

-
-

- The old app's Leader was a Facilitator who is a primary contact, but a row titled Leader counts - toward no program status. Pick what happens to each one — the suggested choice is preselected, and nothing changes - until you confirm on the next page. -

-
    -
  • Merge into Facilitator — the Facilitator row beside it turns primary contact on and takes its FileMaker codes and anything attached to it; the Leader row is deleted.
    • -
  • Rename to Facilitator — the row becomes a Facilitator, primary contact on.
    • -
  • Rename and end — the same, then ended today, so it stays on the record without counting as active.
    • -
  • Delete — the Leader row and its FileMaker codes are removed.
    • -
-

Every change except Delete leaves a comment on the affiliation saying it was a Leader.

-
- - <% if @rows.empty? %> -
No affiliations are titled Leader.
- <% else %> - <%= form_with url: admin_leader_affiliations_path, method: :post, data: { turbo: false } do %> -
- <% @rows.each do |row| %> - <% leader = row.leader %> - <% chosen = @pre_outcome[row.key] || row.suggestion %> -
-
-
- <%= link_to leader.person.name, edit_person_path(leader.person, anchor: dom_id(leader)), target: "_blank", rel: "noopener", - class: "font-medium text-gray-900 hover:text-blue-700 hover:underline" %> - — <%= leader.organization.name %> -
-
- Leader · <%= leader.decorate.period_label.presence || "No dates" %><%= " · primary contact" if leader.primary_contact? %> - <% if leader.filemaker_code.present? %> · FileMaker <%= leader.filemaker_code %><% end %> -
-
"> - <% if row.twin %> - Facilitator #<%= row.twin.id %> · <%= row.twin.decorate.period_label.presence || "No dates" %><%= " · primary contact" if row.twin.primary_contact? %> - <%= row.same_dates ? "· same dates" : "· overlaps with different dates — check which dates are right" %> - <% else %> - No Facilitator row covers this stretch. - <% end %> -
-
-
- <% row.choices.each do |value| %> - <% label, color = choice_labels[value] %> - - <% end %> -
-
- <% end %> -
- -
- <%= link_to "Cancel", data_health_return_path(DataHealth::LegacyLeaderAffiliations.key), class: "text-sm #{eyebrow_link_class}" %> - <%= submit_tag "Preview changes", class: button_classes(:primary) %> -
- <% end %> - <% end %> -
diff --git a/config/routes.rb b/config/routes.rb index d2e7298281..5fd77b9a13 100644 --- a/config/routes.rb +++ b/config/routes.rb @@ -69,9 +69,6 @@ get "data_health/worklists", to: "data_health#worklists", as: "data_health_worklists" get "data_health/:check/rows", to: "data_health#rows", as: "data_health_rows" post "data_health/:check/repair", to: "data_health#repair", as: "data_health_repair" - get "leader_affiliations", to: "leader_affiliations#index", as: "leader_affiliations" - post "leader_affiliations", to: "leader_affiliations#confirm" - post "leader_affiliations/perform", to: "leader_affiliations#create", as: "perform_leader_affiliations" get "resource_roles", to: "resource_roles#show", as: "resource_roles" patch "resource_roles", to: "resource_roles#update" get "certificate_settings", to: redirect("/admin/resource_roles") diff --git a/db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml b/db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml new file mode 100644 index 0000000000..17e44d7c4a --- /dev/null +++ b/db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml @@ -0,0 +1,14 @@ +name: "See leftover Leader affiliations in Data health" +area: people +display_status: admin_facing +released_on: 2026-10-24 +pr_number: 2788 +summary: >- + Data health lists affiliations still titled Leader, which don't count toward + program status, each with the Facilitator affiliation beside it, so you can + open the person and clean it up. +pro_tips: + - "Find it under Data health → Affiliations → Affiliations still titled Leader." + - "A row marked “same dates” sits beside a Facilitator affiliation that already covers it; one marked “overlaps with different dates” needs a look at which dates are right." + - "Each name opens the person's edit page at that affiliation, and its back link returns you to Data health." +action_path: "/admin/data_health" diff --git a/db/seeds/features/2026-10-24-review-leader-affiliations.yml b/db/seeds/features/2026-10-24-review-leader-affiliations.yml deleted file mode 100644 index ad981afed2..0000000000 --- a/db/seeds/features/2026-10-24-review-leader-affiliations.yml +++ /dev/null @@ -1,15 +0,0 @@ -name: "Review leftover Leader affiliations" -area: people -display_status: admin_facing -released_on: 2026-10-24 -pr_number: 2788 -summary: >- - A review page lists affiliations still titled Leader, which don't count toward - program status, so you can merge, rename, end, or delete each one — previewing - every change before it's made. -pro_tips: - - "Open it from Data health → Affiliations still titled Leader → Review." - - "A Leader row with a Facilitator row for the same dates suggests merging into it, so no duplicate is left behind." - - "A Leader row whose dates overlap a Facilitator row's but don't match is left as is unless you pick a change." - - "Every change except Delete leaves a comment saying the affiliation was a Leader." -action_path: "/admin/leader_affiliations" diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index d8fa588024..cd2e8ed9a1 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -77,10 +77,9 @@ Constraints already decided elsewhere: contacts (402 aren't), all 53 Liaisons are, and 111 rows are still titled "Leader" (which the exact-match Facilitator scope doesn't count, and which the import matches as a Facilitator rather than duplicating). Of those, 109 sit beside an undated, primary - Facilitator twin the January conversion made, and 2 have none. The admin Leader - affiliations page (linked from the Data health check "Affiliations still titled Leader") - settles them row by row: merge into the twin, rename, rename and end, delete, or keep, - previewed before anything changes (`AffiliationServices::FoldLegacyLeaders`). + Facilitator twin the January conversion made, and 2 have none. The Data health check + "Affiliations still titled Leader" lists each with its twin and links to the person's edit + page, where an admin settles it by hand; nothing changes them automatically. - Every role becomes a **Facilitator** affiliation, with a "FileMaker import" comment recording the role FileMaker gave. No prod row is titled Assistant, so the old app's assistants likely became the 402 Facilitators who aren't primary contacts. diff --git a/spec/requests/admin/data_health_spec.rb b/spec/requests/admin/data_health_spec.rb index 6efc97da77..dc7b457628 100644 --- a/spec/requests/admin/data_health_spec.rb +++ b/spec/requests/admin/data_health_spec.rb @@ -182,6 +182,21 @@ def stories_without_workshop(number) expect(frame.text).to include("11–12 of 12") end + it "lists Leader rows with their Facilitator twin, each opening the person's edit page" do + person = create(:person) + twin = create(:affiliation, person:, organization:, title: "Facilitator") + leader = create(:affiliation, person:, organization:, title: "Leader", filemaker_code: "P0001") + + get admin_data_health_rows_path(check: "legacy_leader_affiliations"), + headers: { "Turbo-Frame" => "data_health_rows_legacy_leader_affiliations" } + + expect(response).to have_http_status(:ok) + frame = Nokogiri::HTML(response.body).at_css("turbo-frame#data_health_rows_legacy_leader_affiliations") + expect(frame.text).to include("Facilitator ##{twin.id}", "same dates", "FileMaker P0001") + expect(frame.to_html).to include(CGI.escapeHTML(edit_person_path(person, return_to: "data_health", check: "legacy_leader_affiliations", + anchor: "affiliation_#{leader.id}"))) + end + it "is a 404 for an unknown check" do get admin_data_health_rows_path(check: "no_such_check") diff --git a/spec/requests/admin/leader_affiliations_spec.rb b/spec/requests/admin/leader_affiliations_spec.rb deleted file mode 100644 index b9ff20ca29..0000000000 --- a/spec/requests/admin/leader_affiliations_spec.rb +++ /dev/null @@ -1,51 +0,0 @@ -require "rails_helper" - -RSpec.describe "Admin::LeaderAffiliations", type: :request do - let(:admin) { create(:user, :admin) } - let(:person) { create(:person) } - let(:organization) { create(:organization) } - let!(:twin) { create(:affiliation, person:, organization:, title: "Facilitator") } - let!(:leader) { create(:affiliation, person:, organization:, title: "Leader") } - - it "keeps non-admins out" do - sign_in create(:user) - - get admin_leader_affiliations_path - - expect(response).to redirect_to(root_path) - end - - describe "as an admin" do - before { sign_in admin } - - it "lists each Leader row with its suggested choice selected" do - get admin_leader_affiliations_path - - expect(response).to have_http_status(:ok) - expect(response.body).to include("Facilitator ##{twin.id}") - expect(response.body).to match(/id="outcome_#{leader.id}_merge"[^>]*checked|checked[^>]*id="outcome_#{leader.id}_merge"/) - end - - it "previews the chosen changes without making them" do - post admin_leader_affiliations_path, params: { outcome: { leader.id.to_s => "merge" } } - - expect(response).to have_http_status(:ok) - expect(response.body).to include("Merge into Facilitator (1)") - expect(Affiliation.exists?(leader.id)).to be(true) - end - - it "sends an empty choice back to the page" do - post admin_leader_affiliations_path, params: { outcome: { leader.id.to_s => "keep" } } - - expect(response).to redirect_to(admin_leader_affiliations_path(outcome: { leader.id.to_s => "keep" })) - end - - it "performs the confirmed changes" do - post admin_perform_leader_affiliations_path, params: { outcome: { leader.id.to_s => "merge" } } - - expect(response).to redirect_to(admin_leader_affiliations_path) - expect(Affiliation.exists?(leader.id)).to be(false) - expect(twin.reload.primary_contact).to be(true) - end - end -end diff --git a/spec/routing/admin_leader_affiliations_routing_spec.rb b/spec/routing/admin_leader_affiliations_routing_spec.rb deleted file mode 100644 index 3ce7d61907..0000000000 --- a/spec/routing/admin_leader_affiliations_routing_spec.rb +++ /dev/null @@ -1,9 +0,0 @@ -require "rails_helper" - -RSpec.describe Admin::LeaderAffiliationsController, type: :routing do - it "routes the review, its preview, and performing it" do - expect(get: "/admin/leader_affiliations").to route_to("admin/leader_affiliations#index") - expect(post: "/admin/leader_affiliations").to route_to("admin/leader_affiliations#confirm") - expect(post: "/admin/leader_affiliations/perform").to route_to("admin/leader_affiliations#create") - end -end diff --git a/spec/services/affiliation_services/fold_legacy_leaders_spec.rb b/spec/services/affiliation_services/fold_legacy_leaders_spec.rb deleted file mode 100644 index eabbfe39bc..0000000000 --- a/spec/services/affiliation_services/fold_legacy_leaders_spec.rb +++ /dev/null @@ -1,106 +0,0 @@ -# frozen_string_literal: true - -require "rails_helper" - -RSpec.describe AffiliationServices::FoldLegacyLeaders do - let(:person) { create(:person) } - let(:organization) { create(:organization) } - - def affiliation(title, **attributes) - create(:affiliation, person:, organization:, title:, **attributes) - end - - def row_for(leader) = described_class.new.rows.find { |row| row.leader == leader } - - def apply(outcome) = described_class.new.apply(outcome: outcome.transform_keys { |leader| leader.id.to_s }) - - describe "#rows" do - it "suggests merging into a Facilitator twin with the same dates" do - twin = affiliation("Facilitator") - leader = affiliation("Leader") - - expect(row_for(leader)).to have_attributes(twin:, same_dates: true, suggestion: "merge") - end - - it "suggests renaming when no Facilitator row covers its stretch" do - affiliation("Facilitator", start_date: Date.new(2010, 1, 1), end_date: Date.new(2012, 12, 31)) - leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) - - row = row_for(leader) - expect(row).to have_attributes(twin: nil, suggestion: "rename") - expect(row.choices).not_to include("merge") - end - - it "suggests leaving one that overlaps a Facilitator row with different dates" do - twin = affiliation("Facilitator", start_date: Date.new(2018, 1, 1), end_date: Date.new(2022, 12, 31)) - leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) - - expect(row_for(leader)).to have_attributes(twin:, same_dates: false, suggestion: "keep") - end - end - - describe "#planned_changes" do - it "plans only the rows given a choice they offer, skipping keep" do - leader = affiliation("Leader") - other = affiliation("Leader", start_date: Date.new(2019, 1, 1)) - - changes = described_class.new.planned_changes(outcome: { leader.id.to_s => "merge", other.id.to_s => "keep" }) - - expect(changes).to be_empty - expect(described_class.new.planned_changes(outcome: { leader.id.to_s => "rename" }).map(&:action)).to eq([ "rename" ]) - end - end - - describe "#apply" do - it "merges a Leader row into its twin, carrying its codes and comments" do - twin = affiliation("Facilitator", primary_contact: false, filemaker_code: "P0001") - leader = affiliation("Leader", primary_contact: true, filemaker_code: "PS0001") - leader.comments.create!(body: "Met at the 2019 training") - - expect(apply(leader => "merge")).to eq(1) - - expect(Affiliation.exists?(leader.id)).to be(false) - expect(twin.reload).to have_attributes(primary_contact: true, filemaker_code: "P0001, PS0001") - expect(twin.comments.pluck(:body)).to contain_exactly("Met at the 2019 training", a_string_including("Leader", "##{leader.id}")) - end - - it "merges two Leader rows sharing one twin, keeping both codes" do - twin = affiliation("Facilitator", primary_contact: true) - first = affiliation("Leader", filemaker_code: "P0001") - # The model refuses an identical row now; prod's pairs predate that. - second = affiliation("Placeholder", filemaker_code: "P0002").tap { |row| row.update_column(:title, "Leader") } - - apply(first => "merge", second => "merge") - - expect(person.affiliations.reload).to contain_exactly(twin) - expect(twin.reload.filemaker_code).to eq("P0001, P0002") - end - - it "renames a Leader row to a primary-contact Facilitator" do - leader = affiliation("Leader", primary_contact: false) - - apply(leader => "rename") - - expect(leader.reload).to have_attributes(title: "Facilitator", primary_contact: true) - expect(leader.comments.sole).to have_attributes(topic: described_class::COMMENT_TOPIC, body: a_string_including("Titled Leader")) - end - - it "renames and ends one, keeping an end date it already has" do - open_leader = affiliation("Leader") - dated = affiliation("Leader", start_date: Date.new(2019, 1, 1), end_date: Date.new(2020, 1, 1)) - - apply(open_leader => "end", dated => "end") - - expect(open_leader.reload).to have_attributes(title: "Facilitator", end_date: Date.current, inactive: true) - expect(dated.reload).to have_attributes(title: "Facilitator", end_date: Date.new(2020, 1, 1), inactive: true) - end - - it "deletes one outright" do - leader = affiliation("Leader") - - apply(leader => "delete") - - expect(Affiliation.exists?(leader.id)).to be(false) - end - end -end diff --git a/spec/services/data_health/checks_spec.rb b/spec/services/data_health/checks_spec.rb index 79585af7d4..8dde1ec420 100644 --- a/spec/services/data_health/checks_spec.rb +++ b/spec/services/data_health/checks_spec.rb @@ -124,14 +124,36 @@ def affiliation_from(facilitator_training:, title: "Facilitator") end RSpec.describe DataHealth::LegacyLeaderAffiliations do - it "finds affiliations still titled Leader and sends them to the review page" do - leader = create(:affiliation, title: "Leader") - create(:affiliation, title: "Facilitator") - check = described_class.new + let(:person) { create(:person) } + let(:organization) { create(:organization) } - expect(check.scope).to contain_exactly(leader) - expect(check).not_to be_repairable - expect(check.review_path).to eq("/admin/leader_affiliations") + def affiliation(title, **attributes) = create(:affiliation, person:, organization:, title:, **attributes) + + it "lists affiliations still titled Leader, and reports only" do + leader = affiliation("Leader") + affiliation("Facilitator", start_date: Date.new(2019, 1, 1)) + + expect(described_class.new.scope).to contain_exactly(leader) + expect(described_class.new).not_to be_repairable + end + + describe "#twin_for" do + it "finds the Facilitator row with the same dates first" do + affiliation("Facilitator", start_date: Date.new(2015, 1, 1)) + same = affiliation("Facilitator", start_date: Date.new(2019, 1, 1)) + leader = affiliation("Leader", start_date: Date.new(2019, 1, 1)) + + expect(described_class.new.twin_for(leader)).to eq(same) + end + + it "falls back to one overlapping with different dates, and is nil when none covers the stretch" do + overlapping = affiliation("Facilitator", start_date: Date.new(2018, 1, 1), end_date: Date.new(2022, 12, 31)) + leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) + later = affiliation("Leader", start_date: Date.new(2024, 1, 1), end_date: Date.new(2025, 1, 1)) + + expect(described_class.new.twin_for(leader)).to eq(overlapping) + expect(described_class.new.twin_for(later)).to be_nil + end end end diff --git a/spec/views/page_bg_class_alignment_spec.rb b/spec/views/page_bg_class_alignment_spec.rb index 0f4484c0cd..13caf0ab98 100644 --- a/spec/views/page_bg_class_alignment_spec.rb +++ b/spec/views/page_bg_class_alignment_spec.rb @@ -134,8 +134,6 @@ "app/views/people/email_addresses.html.erb" => "admin-only bg-blue-100", "app/views/comments_and_communications/index.html.erb" => "admin-only bg-blue-100", "app/views/admin/data_health/index.html.erb" => "admin-only bg-blue-100", - "app/views/admin/leader_affiliations/index.html.erb" => "admin-only bg-blue-100", - "app/views/admin/leader_affiliations/confirm.html.erb" => "admin-only bg-blue-100", "app/views/admin/resource_roles/show.html.erb" => "admin-only bg-blue-100", "app/views/comments/index.html.erb" => "admin-only bg-blue-100", "app/views/bookmarks/index.html.erb" => "admin-only bg-blue-100", From 5fc1d55c53381e04955fe6b02798764bd8887db7 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sat, 24 Oct 2026 20:44:44 -0400 Subject: [PATCH 17/25] Show whether a Leader row's person already facilitates there Settling a Leader row by hand starts with one question: does this person already have a Facilitator affiliation at the organization? Each row now answers it with a badge, lists those Facilitator rows and how their stretch meets the Leader row's, and notes when they facilitate elsewhere. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- .../data_health/legacy_leader_affiliations.rb | 25 ++++++++++--- .../data_health/_legacy_leader_row.html.erb | 35 +++++++++++++------ ...-24-leader-affiliations-in-data-health.yml | 1 + spec/requests/admin/data_health_spec.rb | 14 +++++++- spec/services/data_health/checks_spec.rb | 10 ++++++ 6 files changed, 70 insertions(+), 17 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index eb5bb480de..a1c2966c8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -223,7 +223,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `Analytics::EventReferenceLoader` — Batch-loads the records referenced in a page of Ahoy event properties (association changes, associated records), one query per type, so the activity table's Details column links each to its show page without an N+1 - `Analytics::ContentCreationBreakdown` — Counts content created in the selected time range for the Ahoy charts page's Total / User-generated / Admin-generated cards. Each record is credited by its author's account (super_user → admin), falling back to the creating account when no author is set, so promoted ideas and admin-entered facilitator content land on the user side and the two cards always sum to the total -- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled Leader (report-only; each row shows its Facilitator twin via `#twin_for` and opens the person's edit page at the affiliation), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) +- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled Leader (report-only; each row badges whether the person has a Facilitator row at that org, lists those rows with how their stretch meets the Leader row's — `#facilitators_for`, `#twin_for`, `#stretch_label` — notes Facilitator rows at other orgs, and opens the person's edit page at the affiliation), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) - `DataHealth::Worklists` — Counts for review pages that live elsewhere (possible duplicates per dedupe page via each `DuplicateFinder` / `Dedupable.name_groups`, and author credit divergence sections, each linked to its section via `?section=` which renders the divergence results with the page so the anchor can scroll), rendered in Data health's lazy `data_health_worklists` frame because the finders load records in Ruby - `ResourceServices::ResolveOrganizations` — Links resources to an organization from the legacy free-text `agency` name (case-insensitive exact match, else creates one with status "Unknown"). Idempotent; drives both the `data:resolve_resource_organizations` rake task (dry-run by default) and the Data health repair ### Business Logic diff --git a/app/services/data_health/legacy_leader_affiliations.rb b/app/services/data_health/legacy_leader_affiliations.rb index cdc698aee1..7377515ce5 100644 --- a/app/services/data_health/legacy_leader_affiliations.rb +++ b/app/services/data_health/legacy_leader_affiliations.rb @@ -23,12 +23,29 @@ def describe(affiliation) "#{affiliation.person&.name} — #{affiliation.organization&.name}" end + # The person's Facilitator rows at the Leader row's organization, oldest first. + def facilitators_for(leader) + Affiliation.facilitators.where(person_id: leader.person_id, organization_id: leader.organization_id).order(:start_date, :id).to_a + end + + # How many organizations other than this one the person facilitates at. + def facilitator_elsewhere_count(leader) + Affiliation.facilitators.where(person_id: leader.person_id).where.not(organization_id: leader.organization_id) + .distinct.count(:organization_id) + end + # The Facilitator row recording the same stretch (AffiliationStint), the same dates first. - def twin_for(leader) + def twin_for(leader, facilitators = facilitators_for(leader)) stint = AffiliationStint.of(leader) - Affiliation.facilitators.where(person_id: leader.person_id, organization_id: leader.organization_id) - .select { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } - .min_by { |affiliation| same_dates?(affiliation, leader) ? 0 : 1 } + facilitators.select { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } + .min_by { |affiliation| same_dates?(affiliation, leader) ? 0 : 1 } + end + + def stretch_label(facilitator, leader, twin) + return "same dates" if same_dates?(facilitator, leader) + return "overlaps with different dates" if facilitator == twin + + "a different stretch" end def same_dates?(affiliation, leader) diff --git a/app/views/admin/data_health/_legacy_leader_row.html.erb b/app/views/admin/data_health/_legacy_leader_row.html.erb index be30d82be7..dd3d219fff 100644 --- a/app/views/admin/data_health/_legacy_leader_row.html.erb +++ b/app/views/admin/data_health/_legacy_leader_row.html.erb @@ -1,24 +1,37 @@ <%# Locals: record (an Affiliation titled Leader). The person opens their edit page at the - affiliation; the line below says which Facilitator row, if any, records the same stretch. %> + affiliation; below it, whether they already have a Facilitator row here and which one, if + any, records the same stretch. %> <% check = DataHealth::LegacyLeaderAffiliations.new %> -<% twin = check.twin_for(record) %> +<% facilitators = check.facilitators_for(record) %> +<% twin = check.twin_for(record, facilitators) %> +<% elsewhere = check.facilitator_elsewhere_count(record) %> <% back = { return_to: "data_health", check: check.key } %> -
+
<% if record.person %> <%= link_to person_name(record.person), edit_person_path(record.person, **back, anchor: dom_id(record)), target: "_blank", rel: "noopener", class: "font-medium text-gray-800 hover:text-blue-700 hover:underline" %> <% end %> — <%= record.organization&.name %> + <% if facilitators.any? %> + <%= render "shared/badge", label: "Has Facilitator row here", classes: "bg-green-50 text-green-800 border-green-300", icon: "fa-solid fa-circle-check" %> + <% else %> + <%= render "shared/badge", label: "No Facilitator row here", classes: "bg-amber-50 text-amber-800 border-amber-300", icon: "fa-solid fa-circle-exclamation" %> + <% end %>
Leader · <%= record.decorate.period_label.presence || "No dates" %><%= " · primary contact" if record.primary_contact? %> <% if record.filemaker_code.present? %> · FileMaker <%= record.filemaker_code %><% end %>
-
"> - <% if twin %> - Facilitator #<%= twin.id %> · <%= twin.decorate.period_label.presence || "No dates" %><%= " · primary contact" if twin.primary_contact? %> - · <%= check.same_dates?(twin, record) ? "same dates" : "overlaps with different dates" %> - <% else %> - No Facilitator row covers this stretch. - <% end %> -
+<% facilitators.each do |facilitator| %> + <% same = check.same_dates?(facilitator, record) %> +
"> + Facilitator #<%= facilitator.id %> · <%= facilitator.decorate.period_label.presence || "No dates" %><%= " · primary contact" if facilitator.primary_contact? %> + · <%= check.stretch_label(facilitator, record, twin) %> +
+<% end %> +<% if facilitators.any? && twin.nil? %> +
None of these covers this Leader row's stretch.
+<% end %> +<% if elsewhere.positive? %> +
Also a Facilitator at <%= pluralize(elsewhere, "other organization") %>.
+<% end %> diff --git a/db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml b/db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml index 17e44d7c4a..35e68723eb 100644 --- a/db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml +++ b/db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml @@ -9,6 +9,7 @@ summary: >- open the person and clean it up. pro_tips: - "Find it under Data health → Affiliations → Affiliations still titled Leader." + - "Each row says whether the person already has a Facilitator affiliation at that organization, and lists it." - "A row marked “same dates” sits beside a Facilitator affiliation that already covers it; one marked “overlaps with different dates” needs a look at which dates are right." - "Each name opens the person's edit page at that affiliation, and its back link returns you to Data health." action_path: "/admin/data_health" diff --git a/spec/requests/admin/data_health_spec.rb b/spec/requests/admin/data_health_spec.rb index dc7b457628..cd8bb51422 100644 --- a/spec/requests/admin/data_health_spec.rb +++ b/spec/requests/admin/data_health_spec.rb @@ -192,11 +192,23 @@ def stories_without_workshop(number) expect(response).to have_http_status(:ok) frame = Nokogiri::HTML(response.body).at_css("turbo-frame#data_health_rows_legacy_leader_affiliations") - expect(frame.text).to include("Facilitator ##{twin.id}", "same dates", "FileMaker P0001") + expect(frame.text).to include("Has Facilitator row here", "Facilitator ##{twin.id}", "same dates", "FileMaker P0001") expect(frame.to_html).to include(CGI.escapeHTML(edit_person_path(person, return_to: "data_health", check: "legacy_leader_affiliations", anchor: "affiliation_#{leader.id}"))) end + it "says when a Leader row's person has no Facilitator row at its organization" do + person = create(:person) + create(:affiliation, person:, organization:, title: "Leader") + create(:affiliation, person:, title: "Facilitator") + + get admin_data_health_rows_path(check: "legacy_leader_affiliations"), + headers: { "Turbo-Frame" => "data_health_rows_legacy_leader_affiliations" } + + frame = Nokogiri::HTML(response.body).at_css("turbo-frame#data_health_rows_legacy_leader_affiliations") + expect(frame.text).to include("No Facilitator row here", "Also a Facilitator at 1 other organization") + end + it "is a 404 for an unknown check" do get admin_data_health_rows_path(check: "no_such_check") diff --git a/spec/services/data_health/checks_spec.rb b/spec/services/data_health/checks_spec.rb index 8dde1ec420..4d07a94a49 100644 --- a/spec/services/data_health/checks_spec.rb +++ b/spec/services/data_health/checks_spec.rb @@ -137,6 +137,16 @@ def affiliation(title, **attributes) = create(:affiliation, person:, organizatio expect(described_class.new).not_to be_repairable end + it "labels each Facilitator row by how its stretch meets the Leader row's" do + check = described_class.new + leader = affiliation("Leader", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) + same = affiliation("Facilitator", start_date: Date.new(2019, 4, 1), end_date: Date.new(2025, 7, 1)) + earlier = affiliation("Facilitator", start_date: Date.new(2010, 1, 1), end_date: Date.new(2012, 1, 1)) + + expect(check.stretch_label(same, leader, same)).to eq("same dates") + expect(check.stretch_label(earlier, leader, same)).to eq("a different stretch") + end + describe "#twin_for" do it "finds the Facilitator row with the same dates first" do affiliation("Facilitator", start_date: Date.new(2015, 1, 1)) From 8542a4db88846c46c870ccc76285cd6d5949366c Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sun, 25 Oct 2026 04:34:21 -0400 Subject: [PATCH 18/25] Report every old-app role title in Data health, not just Leader MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The January conversion copied all of the old app's roles — Leader, Liaison, Assistant and a blank default — into affiliation titles, and none of them counts toward program status; prod still has 53 Liaison rows beside the 111 Leader ones. The check now lists all of them, and the import matches any of those titles as a Facilitator so it doesn't add a twin beside a Liaison row. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 6 +- .../import_from_filemaker.rb | 6 +- app/services/data_health.rb | 2 +- .../data_health/legacy_leader_affiliations.rb | 55 ---------------- .../data_health/legacy_role_affiliations.rb | 62 +++++++++++++++++++ ...row.html.erb => _legacy_role_row.html.erb} | 12 ++-- ...gacy-role-affiliations-in-data-health.yml} | 11 ++-- ...maker-organization-and-personnel-import.md | 6 +- spec/requests/admin/data_health_spec.rb | 20 +++--- .../import_from_filemaker_spec.rb | 10 +++ spec/services/data_health/checks_spec.rb | 9 +-- 11 files changed, 110 insertions(+), 89 deletions(-) delete mode 100644 app/services/data_health/legacy_leader_affiliations.rb create mode 100644 app/services/data_health/legacy_role_affiliations.rb rename app/views/admin/data_health/{_legacy_leader_row.html.erb => _legacy_role_row.html.erb} (77%) rename db/seeds/features/{2026-10-24-leader-affiliations-in-data-health.yml => 2026-10-24-legacy-role-affiliations-in-data-health.yml} (62%) diff --git a/AGENTS.md b/AGENTS.md index a1c2966c8e..7009383b77 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -223,7 +223,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `Analytics::EventReferenceLoader` — Batch-loads the records referenced in a page of Ahoy event properties (association changes, associated records), one query per type, so the activity table's Details column links each to its show page without an N+1 - `Analytics::ContentCreationBreakdown` — Counts content created in the selected time range for the Ahoy charts page's Total / User-generated / Admin-generated cards. Each record is credited by its author's account (super_user → admin), falling back to the creating account when no author is set, so promoted ideas and admin-entered facilitator content land on the user side and the two cards always sum to the total -- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled Leader (report-only; each row badges whether the person has a Facilitator row at that org, lists those rows with how their stretch meets the Leader row's — `#facilitators_for`, `#twin_for`, `#stretch_label` — notes Facilitator rows at other orgs, and opens the person's edit page at the affiliation), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) +- `DataHealth` + `DataHealth::Check` subclasses — Consistency checks spanning the whole database, rendered on the admin Data health page (`/admin/data_health`). Each subclass supplies `scope` (a relation, so counting doesn't load the table), `title`/`explanation`, and either a `repair!` or nothing — report-only is the default, because a wrong row isn't always one we know how to put right. Register a new check under its page heading in `DataHealth::GROUPS` (Affiliations, People, Form answers, Payments, Stories, Resources, Communications); `CHECKS` is derived from it. Rows page 10 at a time in a per-check `data_health_rows_` frame (`Admin::DataHealthController#rows`). A check can also link each row (`path_for`, or a `row_partial` for several links) and point the card at an existing review page (`review_path`). Row links pass `return_to=data_health&check=`; destinations build their eyebrow with `DataHealthHelper#data_health_return_path`, which reopens that check. Current checks: facilitator affiliations minted by non-training registrations (deletes), affiliations whose minting registration belongs to another org (report-only; person → edit, org → the registration's Link organizations page), affiliations still titled with an old-app role — Leader, Liaison, Assistant or "default" (`LegacyRoleAffiliations::LEGACY_TITLES`; report-only; each row badges whether the person has a Facilitator row at that org, lists those rows with how their stretch meets the Leader row's — `#facilitators_for`, `#twin_for`, `#stretch_label` — notes Facilitator rows at other orgs, and opens the person's edit page at the affiliation), legacy organization-status drift, people with several logins, people with several primary sectors/age ranges, over-allocated registrations (incl. CE), stories without a specific workshop (none linked, or only a generic Adult/Children's Windows Workshop), portal emails stuck pending past the grace period and not yet resent, incoming messages not marked responded, pending "Other" form answers (deep-linked to their Other responses group) (all report-only; the notification checks count only since `Notification::LAUNCHED_ON` and show old-system rows as a separate linked `aside` count), and resources with a legacy agency name but no organization (links via `ResourceServices::ResolveOrganizations`) - `DataHealth::Worklists` — Counts for review pages that live elsewhere (possible duplicates per dedupe page via each `DuplicateFinder` / `Dedupable.name_groups`, and author credit divergence sections, each linked to its section via `?section=` which renders the divergence results with the page so the anchor can scroll), rendered in Data health's lazy `data_health_worklists` frame because the finders load records in Ruby - `ResourceServices::ResolveOrganizations` — Links resources to an organization from the legacy free-text `agency` name (case-insensitive exact match, else creates one with status "Unknown"). Idempotent; drives both the `data:resolve_resource_organizations` rake task (dry-run by default) and the Data health repair ### Business Logic @@ -331,8 +331,8 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Shared by the FileMaker affiliations import and the `LegacyLeaderAffiliations` Data health check -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row; an existing Facilitator row matches only for the same stretch (overlapping periods, `#same_stint`), so separate stints stay separate (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing "Leader"-titled row matches as a Facilitator; a row naming a project but no person gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Shared by the FileMaker affiliations import and the `LegacyRoleAffiliations` Data health check +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row; an existing Facilitator row matches only for the same stretch (overlapping periods, `#same_stint`), so separate stints stay separate (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing row titled with an old-app role (Leader, Liaison, …) matches as a Facilitator; a row naming a project but no person gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index 6c4cc0cc55..ef46be6368 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -248,10 +248,10 @@ def records_another_row?(affiliation, codes) codes && affiliation.filemaker_codes.any? { |code| codes.include?(code) } end - # Some prod rows still carry the old app's "Leader" title, which is a Facilitator. + # Some prod rows still carry an old-app role title (Leader, Liaison, …), each a Facilitator. def same_title?(held, title) - held = held.to_s.squish - held.casecmp?(title.to_s.squish) || (title == Affiliation::FACILITATOR_TITLE && held.casecmp?(DataHealth::LegacyLeaderAffiliations::LEGACY_TITLE)) + held.to_s.squish.casecmp?(title.to_s.squish) || + (title == Affiliation::FACILITATOR_TITLE && DataHealth::LegacyRoleAffiliations.legacy_title?(held)) end def link(affiliation, row, label, title:, attributes:, codes:) diff --git a/app/services/data_health.rb b/app/services/data_health.rb index 9d7ca51e7e..1a2762ead7 100644 --- a/app/services/data_health.rb +++ b/app/services/data_health.rb @@ -8,7 +8,7 @@ module DataHealth "Affiliations" => [ FacilitatorAffiliationsFromNonTrainings, MisalignedAffiliationProvenance, - LegacyLeaderAffiliations, + LegacyRoleAffiliations, LegacyOrganizationStatusDrift ], "People" => [ diff --git a/app/services/data_health/legacy_leader_affiliations.rb b/app/services/data_health/legacy_leader_affiliations.rb deleted file mode 100644 index 7377515ce5..0000000000 --- a/app/services/data_health/legacy_leader_affiliations.rb +++ /dev/null @@ -1,55 +0,0 @@ -module DataHealth - # Affiliations still titled Leader, the old app's word for a Facilitator who is a - # primary contact. Report-only: each row shows the Facilitator row beside it, if any, and - # opens the person's edit page to settle by hand. - class LegacyLeaderAffiliations < Check - LEGACY_TITLE = "Leader".freeze - - def title = "Affiliations still titled Leader" - - def explanation - "A Leader is a Facilitator who is a primary contact, but a row titled Leader counts toward " \ - "no program status. Each one is listed with the Facilitator row beside it; open the person to " \ - "merge, rename or remove it." - end - - def scope - Affiliation.where(title: LEGACY_TITLE).includes(:person, :organization).order(:organization_id, :person_id, :id) - end - - def row_partial = "admin/data_health/legacy_leader_row" - - def describe(affiliation) - "#{affiliation.person&.name} — #{affiliation.organization&.name}" - end - - # The person's Facilitator rows at the Leader row's organization, oldest first. - def facilitators_for(leader) - Affiliation.facilitators.where(person_id: leader.person_id, organization_id: leader.organization_id).order(:start_date, :id).to_a - end - - # How many organizations other than this one the person facilitates at. - def facilitator_elsewhere_count(leader) - Affiliation.facilitators.where(person_id: leader.person_id).where.not(organization_id: leader.organization_id) - .distinct.count(:organization_id) - end - - # The Facilitator row recording the same stretch (AffiliationStint), the same dates first. - def twin_for(leader, facilitators = facilitators_for(leader)) - stint = AffiliationStint.of(leader) - facilitators.select { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } - .min_by { |affiliation| same_dates?(affiliation, leader) ? 0 : 1 } - end - - def stretch_label(facilitator, leader, twin) - return "same dates" if same_dates?(facilitator, leader) - return "overlaps with different dates" if facilitator == twin - - "a different stretch" - end - - def same_dates?(affiliation, leader) - affiliation.start_date == leader.start_date && affiliation.end_date == leader.end_date - end - end -end diff --git a/app/services/data_health/legacy_role_affiliations.rb b/app/services/data_health/legacy_role_affiliations.rb new file mode 100644 index 0000000000..0af2e7f297 --- /dev/null +++ b/app/services/data_health/legacy_role_affiliations.rb @@ -0,0 +1,62 @@ +module DataHealth + # Affiliations still titled with one of the old app's roles, which the January 2026 + # conversion copied into titles: a Leader or Liaison was a Facilitator who is a primary + # contact, an Assistant a Facilitator who isn't. None of those titles counts toward program + # status. Report-only: each row shows the person's Facilitator rows at that organization and + # opens their edit page to settle by hand. + class LegacyRoleAffiliations < Check + LEGACY_TITLES = %w[Leader Liaison Assistant default].freeze + + def title = "Affiliations still titled with an old-app role" + + def explanation + "The old app's Leader, Liaison and Assistant roles (and its blank \"default\") became titles in " \ + "January, but only a Facilitator affiliation counts toward program status. Each row is listed " \ + "with the person's Facilitator rows at that organization; open the person to merge, rename or remove it." + end + + def scope + Affiliation.where(title: LEGACY_TITLES).includes(:person, :organization).order(:title, :organization_id, :person_id, :id) + end + + def self.legacy_title?(title) + LEGACY_TITLES.any? { |legacy| legacy.casecmp?(title.to_s.squish) } + end + + def row_partial = "admin/data_health/legacy_role_row" + + def describe(affiliation) + "#{affiliation.person&.name} — #{affiliation.organization&.name} (#{affiliation.title})" + end + + # The person's Facilitator rows at the affiliation's organization, oldest first. + def facilitators_for(affiliation) + Affiliation.facilitators.where(person_id: affiliation.person_id, organization_id: affiliation.organization_id) + .order(:start_date, :id).to_a + end + + # How many organizations other than this one the person facilitates at. + def facilitator_elsewhere_count(affiliation) + Affiliation.facilitators.where(person_id: affiliation.person_id).where.not(organization_id: affiliation.organization_id) + .distinct.count(:organization_id) + end + + # The Facilitator row recording the same stretch (AffiliationStint), the same dates first. + def twin_for(affiliation, facilitators = facilitators_for(affiliation)) + stint = AffiliationStint.of(affiliation) + facilitators.select { |facilitator| AffiliationStint.of(facilitator).overlap?(stint) } + .min_by { |facilitator| same_dates?(facilitator, affiliation) ? 0 : 1 } + end + + def stretch_label(facilitator, affiliation, twin) + return "same dates" if same_dates?(facilitator, affiliation) + return "overlaps with different dates" if facilitator == twin + + "a different stretch" + end + + def same_dates?(facilitator, affiliation) + facilitator.start_date == affiliation.start_date && facilitator.end_date == affiliation.end_date + end + end +end diff --git a/app/views/admin/data_health/_legacy_leader_row.html.erb b/app/views/admin/data_health/_legacy_role_row.html.erb similarity index 77% rename from app/views/admin/data_health/_legacy_leader_row.html.erb rename to app/views/admin/data_health/_legacy_role_row.html.erb index dd3d219fff..74c13158f4 100644 --- a/app/views/admin/data_health/_legacy_leader_row.html.erb +++ b/app/views/admin/data_health/_legacy_role_row.html.erb @@ -1,7 +1,7 @@ -<%# Locals: record (an Affiliation titled Leader). The person opens their edit page at the - affiliation; below it, whether they already have a Facilitator row here and which one, if - any, records the same stretch. %> -<% check = DataHealth::LegacyLeaderAffiliations.new %> +<%# Locals: record (an Affiliation titled with an old-app role). The person opens their edit + page at the affiliation; below it, whether they already have a Facilitator row here and + which one, if any, records the same stretch. %> +<% check = DataHealth::LegacyRoleAffiliations.new %> <% facilitators = check.facilitators_for(record) %> <% twin = check.twin_for(record, facilitators) %> <% elsewhere = check.facilitator_elsewhere_count(record) %> @@ -19,7 +19,7 @@ <% end %>
- Leader · <%= record.decorate.period_label.presence || "No dates" %><%= " · primary contact" if record.primary_contact? %> + <%= record.title %> · <%= record.decorate.period_label.presence || "No dates" %><%= " · primary contact" if record.primary_contact? %> <% if record.filemaker_code.present? %> · FileMaker <%= record.filemaker_code %><% end %>
<% facilitators.each do |facilitator| %> @@ -30,7 +30,7 @@ <% end %> <% if facilitators.any? && twin.nil? %> -
None of these covers this Leader row's stretch.
+
None of these covers this row's stretch.
<% end %> <% if elsewhere.positive? %>
Also a Facilitator at <%= pluralize(elsewhere, "other organization") %>.
diff --git a/db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml b/db/seeds/features/2026-10-24-legacy-role-affiliations-in-data-health.yml similarity index 62% rename from db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml rename to db/seeds/features/2026-10-24-legacy-role-affiliations-in-data-health.yml index 35e68723eb..ec23864060 100644 --- a/db/seeds/features/2026-10-24-leader-affiliations-in-data-health.yml +++ b/db/seeds/features/2026-10-24-legacy-role-affiliations-in-data-health.yml @@ -1,14 +1,15 @@ -name: "See leftover Leader affiliations in Data health" +name: "See leftover old-app role affiliations in Data health" area: people display_status: admin_facing released_on: 2026-10-24 pr_number: 2788 summary: >- - Data health lists affiliations still titled Leader, which don't count toward - program status, each with the Facilitator affiliation beside it, so you can - open the person and clean it up. + Data health lists affiliations still titled Leader, Liaison, Assistant or + "default" — the old app's roles, which don't count toward program status — each + with the person's Facilitator affiliations there, so you can open the person + and clean it up. pro_tips: - - "Find it under Data health → Affiliations → Affiliations still titled Leader." + - "Find it under Data health → Affiliations → Affiliations still titled with an old-app role." - "Each row says whether the person already has a Facilitator affiliation at that organization, and lists it." - "A row marked “same dates” sits beside a Facilitator affiliation that already covers it; one marked “overlaps with different dates” needs a look at which dates are right." - "Each name opens the person's edit page at that affiliation, and its back link returns you to Data health." diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index cd2e8ed9a1..81f262b3fb 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -78,8 +78,10 @@ Constraints already decided elsewhere: the exact-match Facilitator scope doesn't count, and which the import matches as a Facilitator rather than duplicating). Of those, 109 sit beside an undated, primary Facilitator twin the January conversion made, and 2 have none. The Data health check - "Affiliations still titled Leader" lists each with its twin and links to the person's edit - page, where an admin settles it by hand; nothing changes them automatically. + "Affiliations still titled with an old-app role" lists every Leader, Liaison, Assistant or + "default" row with the person's Facilitator rows at that org and links to their edit page, + where an admin settles it by hand; nothing changes them automatically. The import matches + any of those titles as a Facilitator. - Every role becomes a **Facilitator** affiliation, with a "FileMaker import" comment recording the role FileMaker gave. No prod row is titled Assistant, so the old app's assistants likely became the 402 Facilitators who aren't primary contacts. diff --git a/spec/requests/admin/data_health_spec.rb b/spec/requests/admin/data_health_spec.rb index cd8bb51422..f0ad514be2 100644 --- a/spec/requests/admin/data_health_spec.rb +++ b/spec/requests/admin/data_health_spec.rb @@ -182,30 +182,30 @@ def stories_without_workshop(number) expect(frame.text).to include("11–12 of 12") end - it "lists Leader rows with their Facilitator twin, each opening the person's edit page" do + it "lists old-app role rows with their Facilitator twin, each opening the person's edit page" do person = create(:person) twin = create(:affiliation, person:, organization:, title: "Facilitator") leader = create(:affiliation, person:, organization:, title: "Leader", filemaker_code: "P0001") - get admin_data_health_rows_path(check: "legacy_leader_affiliations"), - headers: { "Turbo-Frame" => "data_health_rows_legacy_leader_affiliations" } + get admin_data_health_rows_path(check: "legacy_role_affiliations"), + headers: { "Turbo-Frame" => "data_health_rows_legacy_role_affiliations" } expect(response).to have_http_status(:ok) - frame = Nokogiri::HTML(response.body).at_css("turbo-frame#data_health_rows_legacy_leader_affiliations") + frame = Nokogiri::HTML(response.body).at_css("turbo-frame#data_health_rows_legacy_role_affiliations") expect(frame.text).to include("Has Facilitator row here", "Facilitator ##{twin.id}", "same dates", "FileMaker P0001") - expect(frame.to_html).to include(CGI.escapeHTML(edit_person_path(person, return_to: "data_health", check: "legacy_leader_affiliations", + expect(frame.to_html).to include(CGI.escapeHTML(edit_person_path(person, return_to: "data_health", check: "legacy_role_affiliations", anchor: "affiliation_#{leader.id}"))) end - it "says when a Leader row's person has no Facilitator row at its organization" do + it "says when an old-app role row's person has no Facilitator row at its organization" do person = create(:person) - create(:affiliation, person:, organization:, title: "Leader") + create(:affiliation, person:, organization:, title: "Liaison") create(:affiliation, person:, title: "Facilitator") - get admin_data_health_rows_path(check: "legacy_leader_affiliations"), - headers: { "Turbo-Frame" => "data_health_rows_legacy_leader_affiliations" } + get admin_data_health_rows_path(check: "legacy_role_affiliations"), + headers: { "Turbo-Frame" => "data_health_rows_legacy_role_affiliations" } - frame = Nokogiri::HTML(response.body).at_css("turbo-frame#data_health_rows_legacy_leader_affiliations") + frame = Nokogiri::HTML(response.body).at_css("turbo-frame#data_health_rows_legacy_role_affiliations") expect(frame.text).to include("No Facilitator row here", "Also a Facilitator at 1 other organization") end diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index 39b74956e5..9643bea16f 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -167,6 +167,16 @@ def stint(starts, ends = nil) expect(person.affiliations.count).to eq(1) end + it "links a row the old app titled Liaison instead of adding a Facilitator beside it" do + existing = create(:affiliation, person:, organization: program, title: "Liaison", primary_contact: true) + personnel("PS0001", Role: "Liaison") + + import + + expect(person.affiliations.sole).to eq(existing) + expect(existing.reload.filemaker_code).to eq("PS0001") + end + it "links a row the old app titled Leader instead of adding a Facilitator beside it" do existing = create(:affiliation, person:, organization: program, title: "Leader", primary_contact: true) personnel("PS0001") diff --git a/spec/services/data_health/checks_spec.rb b/spec/services/data_health/checks_spec.rb index 4d07a94a49..ea8936823e 100644 --- a/spec/services/data_health/checks_spec.rb +++ b/spec/services/data_health/checks_spec.rb @@ -123,17 +123,18 @@ def affiliation_from(facilitator_training:, title: "Facilitator") end end -RSpec.describe DataHealth::LegacyLeaderAffiliations do +RSpec.describe DataHealth::LegacyRoleAffiliations do let(:person) { create(:person) } let(:organization) { create(:organization) } def affiliation(title, **attributes) = create(:affiliation, person:, organization:, title:, **attributes) - it "lists affiliations still titled Leader, and reports only" do - leader = affiliation("Leader") + it "lists affiliations titled with any old-app role, and reports only" do + titled = %w[Leader liaison Assistant default].map { |title| affiliation(title, start_date: Date.new(2019, 1, title.length)) } affiliation("Facilitator", start_date: Date.new(2019, 1, 1)) + affiliation("Program Director") - expect(described_class.new.scope).to contain_exactly(leader) + expect(described_class.new.scope).to match_array(titled) expect(described_class.new).not_to be_repairable end From aae3ef1d7ec7c3b75d9ddc3277cc469f539e31e1 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sun, 25 Oct 2026 11:32:39 -0400 Subject: [PATCH 19/25] Recount services and rake tasks after main dropped the address translation task Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7009383b77..df16b4e281 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -276,7 +276,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `AddressTranslation` — Turns imported and legacy address values into what the address dropdowns store (ADR-0009): `#country(value, state:)` → a `GeographyHelper::COUNTRIES` name (aliases for our data's spellings, ISO codes, plain names, a country read from the state — a US state incl. "CA - California", a `REGIONS` province, or a country named as the state; placeholders like "-"/"n/a"/"unknown" → nil; "CA" next to state CA → US), `#state(value, country:)` → USPS code for a US state (other regions kept), `#address_type` (the archive's/forms' words via `CONTACT_TYPES`), `#locality` (`AddressLocality` on the translated parts), and `#address(attrs)` for a whole hash. Lookup tables only, never guessed (like `TagTranslation`): a country nothing answers to is cleared, so every stored country is a dropdown name or blank, and counted in `#unmatched`; an unreadable state is kept and counted. Used by `FmAddressAttributes` (every FileMaker import) and `EventRegistrationImporter` (country/state answers + address/phone types; unmatched values listed on the import preview and counted in its notice). No save hook — the dropdowns own new data - `FmAddressAttributes` — Turns an `FmAddress` row into `Address` attributes for any FileMaker import (street from `StreetNo` + lines, ZIP or its `PostalCodeLookup`, county from the archive's ZIP table read once, everything clipped to 255), with state/country/type/locality through `AddressTranslation`; nil when the row lacks a street, city or state. An `address_type:` overrides the row's (an organization's addresses are work); `#place(city:, state:, county:)` builds a street-less address for a place the archive records by city alone (a project). `#unmatched` gathers what it couldn't translate. Used by `PersonServices::ImportFromRolodex` (listed in `fm:import_people`'s report) and the FileMaker organizations/affiliations imports (ADR-0008) - `AddressDefaults` — The country, county and locality an address leaves blank, from its other parts: a US state → United States (`Address::DEFAULT_COUNTRY`), a US ZIP → its county (`FmPostalCode.county_for`, FileMaker's ZIP table, skipped when the ZIP is in another state), then `AddressLocality`. Used on create by `OrganizationServices::UpsertAddress` (both org-linking flows and public registration capture), `PersonServices::CaptureFromSubmission` and `ProfileChangeRequests::Apply`; `UpsertAddress` also fills a matched address's blank country/county and blank or Unknown locality, never replacing a value someone set -- `AddressLocality` — Works out which `Address::LOCALITIES` bucket an address falls in from its own city/state/country/county, for a writer that has an address but never asked the question (`PersonServices::ImportFromRolodex`, and through `AddressDefaults` for the rest — a locality a person picked is never overwritten). Reads a named non-US country as **Outside USA**, a non-CA US state as **Outside CA**, a mailing city inside the City of Los Angeles — Los Angeles or a neighborhood (`LA_CITY_MAILING_CITIES`) — as **LA City**, the rest of Los Angeles County as **LA County**, Orange County as **Orange County**, and any other California county by `CALIFORNIA_REGIONS` (the region most FileMaker projects in that county name, plus a hand-checked few) as **Northern/Central/Southern CA**. A state no US state answers to, with no country, is **Outside USA** (the forms only offer US states). Everything else stays **Unknown**: a California address with no or an unplaced county, and a US address whose state no US state answers to. Every comparison goes through `AddressNormalizer`. FileMaker records a `Locality` only on a **project**, so the organizations pass (ADR-0008) carries that across rather than deriving it; a person's own address has none +- `AddressLocality` — Works out which `Address::LOCALITIES` bucket an address falls in from its own city/state/country/county, for a writer that has an address but never asked the question (`PersonServices::ImportFromRolodex`, and through `AddressDefaults` for the rest — a locality a person picked is never overwritten). Reads a named non-US country as **Outside USA**, a non-CA US state as **Outside CA**, a mailing city inside the City of Los Angeles — Los Angeles or a neighborhood (`LA_CITY_MAILING_CITIES`) — as **LA City**, the rest of Los Angeles County as **LA County**, Orange County as **Orange County**, and any other California county by `CALIFORNIA_REGIONS` (the region most FileMaker projects in that county name, plus a hand-checked few) as **Northern/Central/Southern CA**. A state no US state answers to, with no country, is **Outside USA** (the forms only offer US states). Everything else stays **Unknown**: a California address with no or an unplaced county, and a US address whose state no US state answers to. Every comparison goes through `AddressNormalizer`. FileMaker records a `Locality` only on a **project**, so `OrganizationServices::ImportFromFilemaker` carries that across rather than deriving it (falling back here when a project names none); a person's own address has none - `OutsideUsaState` — Resolves a US-state dropdown + "Not on list?" box pair: `.pick(dropdown, typed)` for raw params (CE callout, admin CE registration) and `.form_field_answers(params, scope)`, which lays `form_fields_outside_usa` answers over `form_fields` for every `_state` question (public forms, registration, callout forms). The box itself is `shared/_outside_usa` — a no-JS `
` - `AddressDeduper` — Merges a person's or organization's addresses that are one place into a single address. `.matching_sets` groups by `AddressNormalizer.signature`, then merges a less complete copy (blank ZIP, street, country, …) into the single fuller address whose every part it shares — skipped when it fits more than one; works fullest-first so a chain of partial copies lands on the fullest. Each merge goes through `ModelDeduper#merge`, which repoints the merged address's inbound FKs — `contact_methods.address_id`, `invoices.bill_to_address_id`, `affiliations.organization_address_id` — onto the survivor. Within a set the survivor is the copy still in use — active before inactive, then the primary, then the oldest (`.keeper_order`) — since `Address.active` drives bill-to addresses, an org's program location and the geographic breakdowns; blank columns on the survivor (`CARRIED_COLUMNS` — the street/ZIP/country/county/district/type/locality, `phone` and the three `la_*` districts; "unknown" type/locality counts as blank) are filled from the merged copies. Then settles a single primary on the set holding the signature captured in the deduper's `merge_keeper` (the survivor's own pre-merge primary address, else the deleted record's), demoting the rest. Runs in the people and organizations dedupers' `after_merge`; `AddressDeduper.merge_duplicates(addressable)` runs it standalone (keeping the owner's current primary) for the **Dedupe addresses** admin page (`AddressesController#dedupe_index`/`#dedupe_perform`) - `AddressServices::DuplicateFinder` — Lists the people and organizations holding addresses that `AddressDeduper.matching_sets` would merge, for the **Dedupe addresses** page. Compares only within one owner — two orgs at one street are separate records, not duplicates @@ -606,7 +606,7 @@ RuboCop linting on PRs and pushes to main. ## Rake Tasks -Located in `lib/tasks/` (18 files): +Located in `lib/tasks/` (19 files): - `dev.rake` — Development database seeding from XML/CSV - `rhino_migrator.rake` — Rich text editor migration - `attachment_report.rake` — Attachment reporting From e029bd1d120c20cef87caa0d22f0ebcd425657cf Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sun, 25 Oct 2026 15:08:41 -0400 Subject: [PATCH 20/25] Land a Personnel row with no imported project at the person's own organization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 27 Personnel rows name no project, and none of their agreements names one either, so the import skipped them and the program history went with them. The person's Rolodex card still says where they worked, so the row lands there — creating the organization from its typed name when nothing matches. A row whose only person is an agency's own card is treated as naming no one, so AWBW Placeholder keeps that program's history. Features entries move to today, the date this ships. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- .../import_from_filemaker.rb | 46 +++++++++++++++++-- ...emaker-organizations-and-affiliations.yml} | 17 ++++--- ...gacy-role-affiliations-in-data-health.yml} | 2 +- ...maker-organization-and-personnel-import.md | 12 ++++- .../import_from_filemaker_spec.rb | 38 +++++++++++++++ 6 files changed, 103 insertions(+), 14 deletions(-) rename db/seeds/features/{2026-10-22-filemaker-organizations-and-affiliations.yml => 2026-10-25-filemaker-organizations-and-affiliations.yml} (53%) rename db/seeds/features/{2026-10-24-legacy-role-affiliations-in-data-health.yml => 2026-10-25-legacy-role-affiliations-in-data-health.yml} (97%) diff --git a/AGENTS.md b/AGENTS.md index df16b4e281..50b3d447fd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -332,7 +332,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations - `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Shared by the FileMaker affiliations import and the `LegacyRoleAffiliations` Data health check -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row; an existing Facilitator row matches only for the same stretch (overlapping periods, `#same_stint`), so separate stints stay separate (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing row titled with an old-app role (Leader, Liaison, …) matches as a Facilitator; a row naming a project but no person gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row; an existing Facilitator row matches only for the same stretch (overlapping periods, `#same_stint`), so separate stints stay separate (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing row titled with an old-app role (Leader, Liaison, …) matches as a Facilitator; a row whose project wasn't imported lands at the org on the person's Rolodex card (its `OrgID`, else its typed name via `NameMatcher`, created when nothing matches); a row naming no person — or only an agency's own card — gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index ef46be6368..5540f32ce2 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -8,7 +8,10 @@ module AffiliationServices # closed, with a comment naming the role FileMaker gave. Only a Leader or Liaison is a # primary contact — turned on for a Facilitator row the portal already had. It links # an existing Facilitator row only for the same stretch (#same_stint), so separate - # stints stay separate. Each carries the row's PrsnlRecID as its filemaker_code. + # stints stay separate. Each carries the row's PrsnlRecID as its filemaker_code. A row + # whose project wasn't imported lands at the organization on the person's Rolodex card + # (created from its name when nothing matches); one naming no person — or only an + # agency's own card — gets AWBW Placeholder where nothing else affiliates anyone. # 2. People-to-organizations (FmPeopleToOrganization) — a job at an organization, with # its title, carrying the row's RecordID. # 3. The Rolodex card itself (FmRolodex) — the organization its OrgID (or its typed @@ -67,6 +70,8 @@ def call @projects = FmProject.all.index_by(&:fm_id) @personnel_ids = FmPersonnel.pluck(:fm_id).to_set @job_ids = FmPeopleToOrganization.pluck(:fm_id).to_set + @cards = FmRolodex.where(fm_id: FmPersonnel.distinct.pluck(Arel.sql("JSON_UNQUOTE(JSON_EXTRACT(data, '$.PersonID'))")).compact_blank) + .index_by(&:fm_id) @unstaffed = [] @read = 0 @@ -105,10 +110,12 @@ def import_personnel(row) label = "#{row.fm_id} #{data["Name"].to_s.squish}".squish return record(:already_imported, label) if @affiliations.include?(row.fm_id) + card = @cards[data["PersonID"].to_s] + organization = organization_for(data["ProjectID"]) || card_organization(card, label) + return record(:skipped, "#{label}: #{missing_organization(data, card)}") unless organization + people = @people.records(data["PersonID"]) - organization = organization_for(data["ProjectID"]) - return record(:skipped, "#{label}: project #{data["ProjectID"].presence || "(none)"} wasn't imported") unless organization - return @unstaffed << [ row, organization, label ] if data["PersonID"].blank? + return @unstaffed << [ row, organization, label ] if data["PersonID"].blank? || (people.empty? && card&.agency?) return record(:skipped, "#{label}: person #{data["PersonID"]} wasn't imported") if people.empty? dates = personnel_dates(data, @projects[data["ProjectID"].to_s]) @@ -120,6 +127,37 @@ def import_personnel(row) end end + # A row whose project wasn't imported (or names none) lands at the organization on the + # person's Rolodex card — the one its OrgID names, else the one its typed name means, + # created when nothing answers to it, since the person staffed a program there. + def card_organization(card, label) + return unless card + + organization_for(card.data["OrgID"]) || named_organization(card.data["Organization"]) || + create_card_organization(card.data["Organization"].to_s.squish, label) + end + + def create_card_organization(name, label) + return if name.blank? + + organization = Organization.create(name: clip(name), created_by_id: @import_user_id, updated_by_id: @import_user_id) + unless organization.persisted? + failed(organization, label) + return + end + + record(:created, "#{label} → organization ##{organization.id} (#{organization.name}), named on the person's Rolodex card") + organization + end + + def missing_organization(data, card) + project = data["ProjectID"].presence + prefix = project ? "project #{project} wasn't imported" : "no project" + return "#{prefix}, and no Rolodex card for person #{data["PersonID"].presence || "(none)"}" unless card + + "#{prefix}, and the person's Rolodex card names no organization" + end + # A Personnel row naming no one still records that the program ran. Read last, once every # other row has had its say, so the placeholder only stands in at an organization nothing # else affiliates anyone with — one placeholder there is enough. diff --git a/db/seeds/features/2026-10-22-filemaker-organizations-and-affiliations.yml b/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml similarity index 53% rename from db/seeds/features/2026-10-22-filemaker-organizations-and-affiliations.yml rename to db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml index 33644a0db7..48adc098d7 100644 --- a/db/seeds/features/2026-10-22-filemaker-organizations-and-affiliations.yml +++ b/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml @@ -1,7 +1,7 @@ name: "FileMaker organizations, programs, and affiliations" area: people display_status: admin_facing -released_on: 2026-10-22 +released_on: 2026-10-25 pr_number: 2788 summary: >- Organizations and art programs from the old FileMaker system can now be @@ -12,11 +12,16 @@ description: >- One running several gets each program nested under it.
  • Each program carries FileMaker's locality and districts onto its address.
  • FileMaker's facilitator assignments become Facilitator affiliations, dated from when they - were entered to when the program closed.
  • Job titles from FileMaker - become job affiliations.
  • Work addresses land on the organization, and - the person's affiliation there points at them.
  • Agency contact cards - fill in their organization's website and address, and the contact person they - name now comes in as a person.
    • + were entered to when the program closed. Leaders and liaisons are marked as + primary contacts, and a comment on each records the role FileMaker gave.
  • A + person with a separate stretch of time at the same organization keeps a separate + affiliation for it.
  • An assignment whose program wasn't in FileMaker lands at + the organization on the person's contact card.
  • A program FileMaker lists + with nobody assigned keeps its history through the placeholder person AWBW + Placeholder.
  • Job titles from FileMaker become job affiliations.
  • Work + addresses land on the organization, and the person's affiliation there points at + them.
  • Agency contact cards become organizations, and the contact person + they name now comes in as a person.
    • pro_tips: - "An organization or affiliation already in the portal is linked rather than duplicated, and only its blank fields are filled in." - "Where FileMaker says something different, the portal's value is kept and a comment on the record lists what FileMaker had." diff --git a/db/seeds/features/2026-10-24-legacy-role-affiliations-in-data-health.yml b/db/seeds/features/2026-10-25-legacy-role-affiliations-in-data-health.yml similarity index 97% rename from db/seeds/features/2026-10-24-legacy-role-affiliations-in-data-health.yml rename to db/seeds/features/2026-10-25-legacy-role-affiliations-in-data-health.yml index ec23864060..ec436cedca 100644 --- a/db/seeds/features/2026-10-24-legacy-role-affiliations-in-data-health.yml +++ b/db/seeds/features/2026-10-25-legacy-role-affiliations-in-data-health.yml @@ -1,7 +1,7 @@ name: "See leftover old-app role affiliations in Data health" area: people display_status: admin_facing -released_on: 2026-10-24 +released_on: 2026-10-25 pr_number: 2788 summary: >- Data health lists affiliations still titled Leader, Liaison, Assistant or diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 81f262b3fb..d4b76c1612 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -155,5 +155,13 @@ the real run. - **Merged projects** (`MergerWithID`): folded — the merged-away project's code joins the survivor's org, so its personnel land there. One the portal already keeps as its own org is reported for a hand merge rather than merged automatically. -- **Unmatched people:** skipped and reported. A row naming no person at all gets AWBW - Placeholder instead, when its organization has no other affiliations (decided 2026-10-24). +- **Unmatched people:** skipped and reported. A row naming no person at all — or only an + agency's own Rolodex card — gets AWBW Placeholder instead, when its organization has no + other affiliations (decided 2026-10-24). +- **Personnel with no project** (27 rows in prod, none with a project on their agreement): + the row lands at the organization on the person's Rolodex card — its `OrgID`, else its + typed name, created as a new organization when nothing matches. A card naming no + organization is skipped and reported (decided 2026-10-25). +- **A full accounting of what didn't come over** — every FileMaker person and organization no + portal record carries a code for — is a later Data health check, read through + `filemaker_code`. diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index 9643bea16f..a108ec742f 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -230,6 +230,44 @@ def stint(starts, ends = nil) end end + describe "a row whose project wasn't imported" do + it "lands at the organization the person's Rolodex card names" do + rolodex(OrgID: "ORG00001") + personnel("PS0001", ProjectID: "") + + import + + expect(person.affiliations.sole).to have_attributes(organization: umbrella, title: "Facilitator", filemaker_code: "PS0001") + end + + it "creates the organization the card only names, and reports it" do + rolodex(Organization: "Foothill Family Services") + personnel("PS0001", ProjectID: "") + + import = described_class.new.call + + created = Organization.find_by!(name: "Foothill Family Services") + expect(person.affiliations.sole.organization).to eq(created) + expect(import.results[:created]).to include(a_string_including("organization ##{created.id}", "Rolodex card")) + end + + it "gives an agency's own card AWBW Placeholder at the agency" do + FmRolodex.create!(fm_id: "00777", data: { "isAgency__c" => "1", "OrgID" => "ORG00001" }) + personnel("PS0001", PersonID: "00777", ProjectID: "") + + import + + expect(umbrella.affiliations.sole.person).to have_attributes(first_name: "AWBW", last_name: "Placeholder") + end + + it "is skipped, saying why, when the card names no organization either" do + rolodex + personnel("PS0001", ProjectID: "") + + expect(import.results[:skipped].sole).to include("PS0001", "no project", "Rolodex card names no organization") + end + end + it "skips a row whose person or project wasn't imported" do personnel("PS0001", PersonID: "99999") personnel("PS0002", ProjectID: "P9999") From 271f414d55741ffe2004a4a96044e27ad2732246 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sun, 25 Oct 2026 15:15:01 -0400 Subject: [PATCH 21/25] Bring differing FileMaker affiliations and phones in as their own records An affiliation whose dates differ from the portal's is a separate stretch of facilitation, not a disagreement about one, so folding it into the portal's row and noting the difference lost the history. It now comes in as its own affiliation; only one whose dates agree (or fill each other's blanks) is linked. An organization's phone numbers each go onto an address that has none, so a second number isn't dropped; one with no free address is still noted on the organization. The agency-card path is pinned end to end: the card becomes an organization with its work address, and the person it names is affiliated there at that address. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 6 +- .../import_from_filemaker.rb | 72 ++++++++----------- .../import_from_filemaker.rb | 31 +++++--- ...lemaker-organizations-and-affiliations.yml | 8 ++- ...maker-organization-and-personnel-import.md | 14 ++-- lib/tasks/fm_import_organizations.rake | 2 +- .../import_from_filemaker_spec.rb | 34 +++++---- .../import_from_filemaker_spec.rb | 32 +++++++++ spec/tasks/fm_import_organizations_spec.rb | 16 +++++ 9 files changed, 134 insertions(+), 81 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 50b3d447fd..c96453d727 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -331,8 +331,8 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Affiliations -- `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Shared by the FileMaker affiliations import and the `LegacyRoleAffiliations` Data health check -- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row; an existing Facilitator row matches only for the same stretch (overlapping periods, `#same_stint`), so separate stints stay separate (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing row titled with an old-app role (Leader, Liaison, …) matches as a Facilitator; a row whose project wasn't imported lands at the org on the person's Rolodex card (its `OrgID`, else its typed name via `NameMatcher`, created when nothing matches); a row naming no person — or only an agency's own card — gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled, differences commented) unless it already records another row of that table. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report +- `AffiliationStint` — One affiliation's stretch of time for telling a person's separate stints at one org apart (`.of(affiliation)`, `.for(start_date:, end_date:, ended:)`, `#overlap?`): a blank start is "since whenever", a blank end "still going", except an ended row with no end date, which is just its start. Used by the `LegacyRoleAffiliations` Data health check to find each row's Facilitator twin +- `AffiliationServices::ImportFromFilemaker` — The FileMaker affiliations pass (ADR-0008), after the people and organizations passes. Personnel (`FmPersonnel`) → every role becomes a Facilitator affiliation on the project's org with a "FileMaker import" comment naming the role — only a Leader or Liaison (`PRIMARY_ROLES`) as a primary contact, turning it on for a matched existing row (start = row entered, end = project's `CloseDate`/`EndDate`, a `Status` other than Active ends it undated), an existing row titled with an old-app role (Leader, Liaison, …) matches as a Facilitator; a row whose project wasn't imported lands at the org on the person's Rolodex card (its `OrgID`, else its typed name via `NameMatcher`, created when nothing matches); a row naming no person — or only an agency's own card — gives an organization with no other affiliations one Facilitator affiliation for the placeholder person "AWBW Placeholder" (`PLACEHOLDER_NAME`), read last; people-to-organizations → a job affiliation with its `Title`; each Rolodex card's `OrgID` (or typed organization name via `NameMatcher`) + `Title` → a job affiliation; and its work addresses onto the organization they belong to (the address's own OrgID, else the card's org, else the person's only org), pointing the person's address-less affiliations there at them — a work address with nowhere to go stays on the person. Rows carry their `PrsnlRecID` / `RecordID` as `filemaker_code`; an existing same-title affiliation is linked (blank dates filled) only when its dates agree — same or blank on one side, still start-before-end — and it records no other row of that table; differing dates come in as a new affiliation. `event_registration_id` stays NULL. Tallies `#statuses` and `#roles` for the report - `AffiliationServices::EndOtherOrganizations` — Someone left their job for a new organization: ends their active-or-pending affiliations at every *other* org — or only the `only:` orgs — (job and facilitator alike), dated the day before the new one starts (ADR-0002 D4). Returns what it ended. Used by `ProfileChangeRequests::Apply` (an add request, only the orgs the requester said they left) - `AffiliationServices::CloseProgram` — Processes a program closure: end-dates the person's active-or-pending **Facilitator** affiliation(s) at the given org to the effective date, plus their **job** affiliation(s) there when `leaving_job:` is set, leaving a `Program closure`-topic comment (reason + date) on each. Returns the ended affiliations; the org's status re-derives Inactive via the `Affiliation` callbacks. Run by `ProfileChangeRequests::Apply` when approving removal of a Facilitator row - `AffiliationServices::CreateFromRegistration` — On registration / org linking, creates a "job affiliation" with the typed title (when present) plus a standing "Facilitator" affiliation, in one transaction. Skips the facilitator one only when the person already has an active-or-pending affiliation titled exactly "Facilitator" with that org (a current one or one dated to a future training); an ended facilitator affiliation gets a fresh second one. Dedupe is by title + org + dates, so a job title like "Lead Facilitator" still gets its own Facilitator affiliation. Accepts an optional `organization_address:` and sets it on every affiliation it creates (the registrant's typed organization address, upserted onto the org); when an affiliation already exists and is skipped, it backfills that address onto the existing one only if it has none (an admin-set address is never overwritten) @@ -357,7 +357,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Organizations -- `OrganizationServices::ImportFromFilemaker` — The FileMaker organizations pass (ADR-0008): each `FmOrganization` and its `FmProject`s become portal orgs — an org with exactly one project stays one org carrying both codes, otherwise the projects nest under it (`parent`). Matches by code (`FilemakerCodeIndex`), then exact name only to an org carrying no code from that table (a namesake gets `"Name (code)"`). Fills blanks only (website, description, mission, `organization_type` from `OrgType`/`AgencyType` via `ORGANIZATION_TYPE_PATTERNS` — any other word lands as Other with the word in `organization_type_other`, tallied in `#unmatched_types`), noting differences as comments; adds the org's `fm_addresses`, its phone onto the primary address, its `fm_email_addresses`; copies each project's `Locality`/`District`/`SPA`/`LASupervisorialDistrict` onto the program's address in its city (a city-only one when none); folds a merged-away project's code (`MergerWithID`) into the survivor; and reads agencies' own Rolodex cards (website, mission, addresses, and email when no contact person is named) onto their org, creating it by name when nothing answers. Driven by `fm:import_organizations` +- `OrganizationServices::ImportFromFilemaker` — The FileMaker organizations pass (ADR-0008): each `FmOrganization` and its `FmProject`s become portal orgs — an org with exactly one project stays one org carrying both codes, otherwise the projects nest under it (`parent`). Matches by code (`FilemakerCodeIndex`), then exact name only to an org carrying no code from that table (a namesake gets `"Name (code)"`). Fills blanks only (website, description, mission, `organization_type` from `OrgType`/`AgencyType` via `ORGANIZATION_TYPE_PATTERNS` — any other word lands as Other with the word in `organization_type_other`, tallied in `#unmatched_types`), noting differences as comments; adds the org's `fm_addresses`, each phone number no address holds onto an address without one (noted when none is free), its `fm_email_addresses`; copies each project's `Locality`/`District`/`SPA`/`LASupervisorialDistrict` onto the program's address in its city (a city-only one when none); folds a merged-away project's code (`MergerWithID`) into the survivor; and reads agencies' own Rolodex cards (website, mission, addresses, and email when no contact person is named) onto their org, creating it by name when nothing answers. Driven by `fm:import_organizations` - `OrganizationServices::AutofillChange` — One thing a registrant's answers wrote onto an org: `field`, human `label`, the `value` that landed in it, `change_type` (`"new"` filled a blank / `"update"` replaced something), `previous_value` on an update, and an optional `scope` naming which work address. `change_type` is derived from `previous_value` so the two can't disagree, but is written to the JSON anyway so the stored row reads on its own. Round-trips through the `form_autofill_changes` column; `#description` renders "Website" or "ZIP on the Austin work address" for a flash, while the linking page shows the value and what it displaced. `#key` is (field, scope), which is what makes a later submission's value replace an earlier one instead of stacking. Drops a stored entry with no `field` rather than raising — it's a display aid, not a ledger. **Not a substitute for the Ahoy audit trail**, which records every before/after on Organization and Address independently; this is the denormalized, registration-scoped copy the linking page can render without correlating events - `OrganizationServices::AddressMatcher` — Given a submitted city/state/street/ZIP, finds the org address it corresponds to so `UpsertAddress`/`ProfileDiff` update the right one instead of duplicating. Tries same city + state (preferring one whose street also matches, so an org with several offices in one city doesn't get the wrong one rewritten), then same city + street (unifies an address with a submission that merely adds the state/country we were missing), then — across cities — same street + same ZIP, which reunites an address whose city was renamed ("Ventura" / "San Buenaventura") without merging two offices that share a street name in different towns. Every comparison goes through `AddressNormalizer`, so known spellings ("St. Louis" / "Saint Louis", "TX" / "Texas", "N Main St" / "North Main Street") match at every step. Module function, used by both address services - `OrganizationServices::UpsertAddress` — Find-or-create an organization's "work" address from a registrant's submitted organization fields (street/city/state/zip/country), matching via `AddressMatcher`; updates the match, else adds a new address; never demotes the org's existing primary (a registrant's address becomes primary only when the org has none yet). `overwrite: true` (default, public flow) replaces the matching address's details; `overwrite: false` (admin org-linking) fills only its blank fields so a conflicting saved address is kept and flagged instead. City and state are always fill-only (never flipped, in either mode), so a cross-city ZIP match keeps the saved city and lets `ProfileDiff` flag the difference; that fill still earns its keep because a legacy row can hold `""` in either column, and the street + ZIP match is the one path that reaches such a row to repair it. Returns a `Result` (`address`, `created`, and `changes` — an `AutofillChange` per field actually written, carrying the value) so callers can report what was really saved. Every field is trimmed before it's stored, so a padded answer isn't later read back as a difference from itself. Tolerates optional answers: a skipped street/ZIP is stored blank (both are `NOT NULL` columns), and `address` is nil — nothing saved — when there's no city, or no state and no existing address to update, rather than failing the whole registration. Shared by `PublicRegistration` and the admin org-linking actions diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index 5540f32ce2..cb55e2f28a 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -7,8 +7,8 @@ module AffiliationServices # Facilitator affiliation, dated from when the row was entered to when the project # closed, with a comment naming the role FileMaker gave. Only a Leader or Liaison is a # primary contact — turned on for a Facilitator row the portal already had. It links - # an existing Facilitator row only for the same stretch (#same_stint), so separate - # stints stay separate. Each carries the row's PrsnlRecID as its filemaker_code. A row + # an existing Facilitator row only when their dates agree (#same_dates); other dates + # are a separate stretch and become their own affiliation. Each carries the row's PrsnlRecID as its filemaker_code. A row # whose project wasn't imported lands at the organization on the person's Rolodex card # (created from its name when nothing matches); one naming no person — or only an # agency's own card — gets AWBW Placeholder where nothing else affiliates anyone. @@ -23,8 +23,7 @@ module AffiliationServices # code the earlier pass gave them. A row whose person or organization wasn't imported is # skipped and reported. An existing affiliation of the same person, organization and # title is linked rather than duplicated: it takes the row's code and fills its blank - # dates, keeping its own where FileMaker disagrees (noted in #differences and as a - # comment on the affiliation). + # dates. # # None of these is minted from a training, so event_registration_id stays NULL: the # reconcile page treats them as hand-entered rows. @@ -43,9 +42,8 @@ class ImportFromFilemaker ROLE_COMMENT_TOPIC = "FileMaker import" # Stands in where a Personnel row names a project but no person. PLACEHOLDER_NAME = { first_name: "AWBW", last_name: "Placeholder" }.freeze - SOURCES = { FmPersonnel => "Personnel", FmPeopleToOrganization => "People-to-organizations", FmRolodex => "Rolodex" }.freeze - attr_reader :results, :differences, :statuses, :roles + attr_reader :results, :statuses, :roles def unmatched_addresses = @address_attributes.unmatched @@ -53,7 +51,6 @@ def initialize(progress: nil, run: nil) @progress = progress @run = run @results = OUTCOMES.index_with { [] } - @differences = [] @statuses = Hash.new(0) @roles = Hash.new(0) @staff = StaffTranslation.new @@ -260,26 +257,35 @@ def named_organization(name) def affiliate(person, organization, row, label, title:, attributes:, codes:, dated: false) existing = person.affiliations.where(organization:).to_a candidates = existing.select { |affiliation| same_title?(affiliation.title, title) && !records_another_row?(affiliation, codes) } - match = dated ? same_stint(candidates, attributes) : candidates.first + match = dated ? same_dates(candidates, attributes) : candidates.first match ||= existing.find { |affiliation| affiliation.title.blank? } if title.present? && codes.nil? return link(match, row, label, title:, attributes:, codes:) if match create(person, organization, row, label, title:, attributes:, code: codes && row.fm_id) end - # A person can facilitate at one organization over several separate stretches, so a - # Personnel row links only the affiliation recording the same one (AffiliationStint). - # Identical dates win, then the nearest start, then one with dates at all. - def same_stint(candidates, dates) - stint = AffiliationStint.for(start_date: dates[:start_date], end_date: dates[:end_date], ended: dates[:inactive]) - candidates.select { |affiliation| AffiliationStint.of(affiliation).overlap?(stint) } - .min_by { |affiliation| stint_rank(affiliation, dates) } + # A Personnel row links only an affiliation whose dates agree with it — each the same, or + # blank on one side — and that still reads start-before-end once the blanks fill. Any other + # dates are a separate stretch, so the row becomes its own affiliation. Identical dates win, + # then the one sharing the most. + def same_dates(candidates, dates) + candidates.select { |affiliation| dates_agree?(affiliation, dates) } + .min_by { |affiliation| [ identical_dates?(affiliation, dates) ? 0 : 1, -shared_dates(affiliation, dates) ] } end - def stint_rank(affiliation, dates) - identical = affiliation.start_date == dates[:start_date] && affiliation.end_date == dates[:end_date] - gap = (affiliation.start_date - dates[:start_date]).abs if affiliation.start_date && dates[:start_date] - [ identical ? 0 : 1, gap || Float::INFINITY ] + def dates_agree?(affiliation, dates) + starts = affiliation.start_date || dates[:start_date] + ends = affiliation.end_date || dates[:end_date] + %i[start_date end_date].all? { |attribute| affiliation[attribute].nil? || dates[attribute].nil? || affiliation[attribute] == dates[attribute] } && + (starts.nil? || ends.nil? || starts <= ends) + end + + def identical_dates?(affiliation, dates) + affiliation.start_date == dates[:start_date] && affiliation.end_date == dates[:end_date] + end + + def shared_dates(affiliation, dates) + %i[start_date end_date].count { |attribute| affiliation[attribute] && affiliation[attribute] == dates[attribute] } end def records_another_row?(affiliation, codes) @@ -298,12 +304,11 @@ def link(affiliation, row, label, title:, attributes:, codes:) affiliation.filemaker_code = Affiliation.join_filemaker_codes(affiliation.filemaker_code, row.fm_id) if codes affiliation.title = title if affiliation.title.blank? affiliation.primary_contact = true if attributes[:primary_contact] - kept = fill_dates(affiliation, attributes) + fill_dates(affiliation, attributes) affiliation.updated_by_id ||= updated_by(row.data) return failed(affiliation, label) unless affiliation.save @affiliations.add(row.fm_id, affiliation.id) if codes - note_differences(affiliation, row, label, kept) if kept.any? record(:linked, "#{label} → affiliation ##{affiliation.id}") affiliation end @@ -311,21 +316,11 @@ def link(affiliation, row, label, title:, attributes:, codes:) # A blank date takes the archive's. The archive's status only ends an affiliation the # portal still holds open with no dates of its own to say otherwise. def fill_dates(affiliation, attributes) - kept = attributes.slice(:start_date, :end_date).filter_map do |attribute, value| - next if value.nil? || affiliation[attribute] == value - - if affiliation[attribute].nil? - affiliation[attribute] = value - next - end + attributes.slice(:start_date, :end_date).each { |attribute, value| affiliation[attribute] ||= value } + return unless attributes[:inactive] && !affiliation.inactive? && affiliation.end_date.nil? - [ Affiliation.human_attribute_name(attribute), affiliation[attribute], value ] - end - if attributes[:inactive] && !affiliation.inactive? && affiliation.end_date.nil? - affiliation.inactive = true - affiliation.inactive_supplied = true - end - kept + affiliation.inactive = true + affiliation.inactive_supplied = true end def create(person, organization, row, label, title:, attributes:, code:) @@ -405,13 +400,6 @@ def organization_for(code) Organization.find_by(id: ids.first) if ids.any? end - def note_differences(affiliation, row, label, kept) - lines = kept.map { |field, portal, archive| "#{field}: kept “#{portal}”, FileMaker has “#{archive}”" } - Comment.create!(commentable: affiliation, - body: "FileMaker #{SOURCES[row.class]} #{row.fm_id} differs from this affiliation:\n#{lines.join("\n")}") - @differences << "#{label} → affiliation ##{affiliation.id}: #{lines.join('; ')}" - end - def failed(record, label) record(:failed, "#{label}: #{record.class.name} #{record.errors.full_messages.to_sentence.presence || "matches one already on file"}") false diff --git a/app/services/organization_services/import_from_filemaker.rb b/app/services/organization_services/import_from_filemaker.rb index ee0135e197..f7864fae15 100644 --- a/app/services/organization_services/import_from_filemaker.rb +++ b/app/services/organization_services/import_from_filemaker.rb @@ -164,7 +164,7 @@ def save(organization, row, label, attributes, also: nil) stamp(organization, row.data) add_addresses(organization, @addresses.fetch(row.fm_id, []).map(&:data)) rows.select { |source| source.is_a?(FmProject) }.each { |project| place_project(organization, project.data) } - add_phone(organization, @phones.fetch(row.fm_id, []).map(&:data)) + kept += add_phones(organization, @phones.fetch(row.fm_id, []).map(&:data)) fill_email(organization, @emails.fetch(row.fm_id, []).map { |email| email.data["EmailAddress"] }) return failed(organization, label) unless organization.save @@ -225,7 +225,7 @@ def import_agency(rolodex) kept = fill(organization, website_url: clip(data["WebSite"]), mission_vision_values: clip(data["Mission"])) stamp(organization, data) add_addresses(organization, rolodex.addresses.map(&:data)) - add_phone(organization, rolodex.phone_numbers.map(&:data)) + kept += add_phones(organization, rolodex.phone_numbers.map(&:data)) fill_email(organization, [ data["EMail"] ]) unless names_a_person?(data) outcome = agency_outcome(organization) return failed(organization, label) unless organization.save @@ -403,12 +403,27 @@ def district_number(value) value.to_s[/\d+/]&.to_i end - # The portal keeps an organization's phone on its address, so the archive's main - # number goes onto the primary one when that has none. - def add_phone(organization, rows) - number = rows.sort_by { |fields| flag?(fields["Primacy"]) ? 0 : 1 }.filter_map { |fields| clip(fields["PhoneNumber"].to_s.strip) }.first - address = organization.addresses.find(&:primary?) || organization.addresses.first - address.phone = number if number && address && address.phone.blank? + # The portal keeps an organization's phone on its addresses, so each number the archive + # has that no address holds yet goes onto one without a phone, the primary first. One + # with nowhere left to go is returned as a difference to note. + def add_phones(organization, rows) + held = organization.addresses.filter_map { |address| phone_key(address.phone) } + numbers = rows.sort_by { |fields| flag?(fields["Primacy"]) ? 0 : 1 } + .filter_map { |fields| clip(fields["PhoneNumber"].to_s.strip) } + .uniq { |number| phone_key(number) }.reject { |number| held.include?(phone_key(number)) } + numbers.filter_map do |number| + address = organization.addresses.sort_by { |held_address| held_address.primary? ? 0 : 1 }.find { |held_address| held_address.phone.blank? } + if address + address.phone = number + next + end + + [ "Phone", organization.addresses.filter_map(&:phone).to_sentence.presence || "no address to hold it", number ] + end + end + + def phone_key(value) + value.to_s.gsub(/\D/, "").presence end def fill_email(organization, values) diff --git a/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml b/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml index 48adc098d7..349eb82646 100644 --- a/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml +++ b/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml @@ -20,9 +20,11 @@ description: >- with nobody assigned keeps its history through the placeholder person AWBW Placeholder.
  • Job titles from FileMaker become job affiliations.
  • Work addresses land on the organization, and the person's affiliation there points at - them.
  • Agency contact cards become organizations, and the contact person - they name now comes in as a person.
    • + them.
  • Agency contact cards become organizations with their addresses as + work addresses, and a contact person they name comes in as a person + affiliated there, at that work address.
    • pro_tips: - "An organization or affiliation already in the portal is linked rather than duplicated, and only its blank fields are filled in." - - "Where FileMaker says something different, the portal's value is kept and a comment on the record lists what FileMaker had." + - "Where FileMaker has an organization's website, mission or type differently, the portal's is kept and a comment on the organization lists what FileMaker had." + - "An affiliation with different dates, an address the organization doesn't have, or a new phone number comes in as its own record, leaving the portal's as it was." action_path: "/organizations" diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index d4b76c1612..7b94b954fe 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -92,10 +92,9 @@ Constraints already decided elsewhere: leaves a matched row's flag alone. - **One row, one stretch.** A person can facilitate at one org over several separate stretches (2020–2022, 2024–Oct 2025, Feb 2026–now), so a Personnel row links an existing - Facilitator row only when their periods overlap. A blank start reads as "since whenever" - and a blank end as "still going", except an ended row with no end date, which reads as - the moment it started so it can't reach a later stretch. Identical dates win, then the - nearest start, then a row with no dates at all. No overlap creates a new row. + Facilitator row only when their dates agree — each the same or blank on one side, and + still start-before-end once the blanks fill. Any other dates are a separate stretch and + come in as a new affiliation (decided 2026-10-25), leaving the portal's untouched. - **A row naming a project but no person** (29 of the blank-role rows) still records that the program ran. Once every other row is read, an organization nothing else affiliates anyone with gets one Facilitator affiliation for the placeholder person **AWBW @@ -131,8 +130,11 @@ Constraints already decided elsewhere: - **Addresses are built by `FmAddressAttributes`** (ADR-0009), as the Rolodex pass does, so state, country, type and locality are stored as the dropdowns offer; a project's city/state goes through `AddressTranslation` the same way. -- **Conflicts:** same rule as people — fill blanks, keep the portal value where the archive - disagrees, and report each difference as a comment plus in the task output. +- **Conflicts:** a field on a record — an organization's website, mission, type — fills blanks + and keeps the portal value where the archive disagrees, reported as a comment plus in the + task output. A record itself comes in new instead: an affiliation whose dates differ, an + address the organization doesn't hold, and a phone number no address holds — placed on an + address without one, and noted only when none is free (decided 2026-10-25). - **Out of scope:** event registrations (`fm_participants`, 68,497 rows), payments, funding, notes and activities — each its own pass. diff --git a/lib/tasks/fm_import_organizations.rake b/lib/tasks/fm_import_organizations.rake index 5c9e7411c9..924c173869 100644 --- a/lib/tasks/fm_import_organizations.rake +++ b/lib/tasks/fm_import_organizations.rake @@ -67,7 +67,7 @@ namespace :fm do messages.each { |message| say.(" #{message}") } end - if import.differences.any? + if import.try(:differences).present? say.("\nKept the portal's value where FileMaker differs (#{import.differences.size}, also left as a comment):") import.differences.each { |difference| say.(" #{difference}") } end diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index a108ec742f..c9035caf63 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -107,48 +107,46 @@ def import "Something else → Facilitator" => 1) end - describe "matching an existing Facilitator row by its stretch" do + describe "matching an existing Facilitator row by its dates" do before { FmProject.find_by!(fm_id: "P0001").update!(data: { "OrgID" => "ORG00001" }) } def stint(starts, ends = nil) create(:affiliation, person:, organization: program, title: "Facilitator", start_date: starts, end_date: ends) end - it "links the stretch the row falls in and leaves the others alone" do + it "links the row whose dates agree, filling the one it leaves blank" do stint(Date.new(2020, 1, 1), Date.new(2022, 12, 31)) - middle = stint(Date.new(2024, 1, 1), Date.new(2025, 10, 31)) - stint(Date.new(2026, 2, 1)) + same = stint(Date.new(2024, 3, 1)) personnel("PS0001", DateEntered: "3/1/2024") import - expect(middle.reload.filemaker_code).to eq("PS0001") - expect(person.affiliations.where.not(id: middle.id).pluck(:filemaker_code)).to all(be_nil) - expect(person.affiliations.count).to eq(3) + expect(same.reload.filemaker_code).to eq("PS0001") + expect(person.affiliations.count).to eq(2) end - it "adds a new stretch when none overlaps" do - stint(Date.new(2020, 1, 1), Date.new(2022, 12, 31)) + it "imports a row whose dates differ as its own affiliation, leaving the portal's alone" do + existing = stint(Date.new(2024, 1, 1), Date.new(2025, 10, 31)) personnel("PS0001", DateEntered: "3/1/2024") import - expect(person.affiliations.order(:start_date).pluck(:start_date, :filemaker_code)) - .to eq([ [ Date.new(2020, 1, 1), nil ], [ Date.new(2024, 3, 1), "PS0001" ] ]) + expect(existing.reload).to have_attributes(filemaker_code: nil, start_date: Date.new(2024, 1, 1)) + expect(person.affiliations.find_by(filemaker_code: "PS0001")).to have_attributes(start_date: Date.new(2024, 3, 1)) + expect(Comment.where(commentable: existing)).to be_empty end - it "reads an ended row with no end date as the moment it started" do - stint(Date.new(2014, 1, 1), Date.new(2016, 12, 31)) - later = stint(Date.new(2024, 1, 1)) - personnel("PS0001", DateEntered: "3/1/2015", Status: "Inactive") + it "doesn't link a row whose blank start would put its end before the archive's start" do + ended = stint(nil, Date.new(2012, 12, 31)) + personnel("PS0001", DateEntered: "3/1/2015") import - expect(later.reload.filemaker_code).to be_nil - expect(person.affiliations.find_by(start_date: Date.new(2014, 1, 1)).filemaker_code).to eq("PS0001") + expect(ended.reload.filemaker_code).to be_nil + expect(person.affiliations.count).to eq(2) end - it "links a row with no dates at all when nothing better fits" do + it "links a row with no dates at all, taking the archive's" do undated = stint(nil) personnel("PS0001", DateEntered: "3/1/2024") diff --git a/spec/services/organization_services/import_from_filemaker_spec.rb b/spec/services/organization_services/import_from_filemaker_spec.rb index 0ff1fce4fa..c320e1ddd7 100644 --- a/spec/services/organization_services/import_from_filemaker_spec.rb +++ b/spec/services/organization_services/import_from_filemaker_spec.rb @@ -76,6 +76,38 @@ def import end end + describe "phones" do + def address(fm_id, street) + FmAddress.create!(fm_id:, data: { "OrgID" => "ORG00001", "AddrsLine1" => street, "AddrsCity" => "Pasadena", "AddrsState" => "CA" }) + end + + it "puts each number on an address with no phone yet, the primary first" do + fm_organization("ORG00001") + address("1", "1 Main St") + address("2", "2 Main St") + FmPhoneNumber.create!(fm_id: "10", data: { "OrgID" => "ORG00001", "PhoneNumber" => "555-0101", "Primacy" => "1" }) + FmPhoneNumber.create!(fm_id: "11", data: { "OrgID" => "ORG00001", "PhoneNumber" => "555-0102" }) + + import + + expect(Organization.sole.addresses.order(:id).pluck(:street_address, :phone)) + .to eq([ [ "1 Main St", "555-0101" ], [ "2 Main St", "555-0102" ] ]) + end + + it "notes a number no address is free to hold, keeping the ones on file" do + existing = create(:organization, filemaker_code: "ORG00001") + create(:address, addressable: existing, street_address: "1 Main St", city: "Pasadena", state: "CA", phone: "555-0100") + fm_organization("ORG00001") + FmPhoneNumber.create!(fm_id: "10", data: { "OrgID" => "ORG00001", "PhoneNumber" => "555-0199" }) + + import = described_class.new.call + + expect(existing.addresses.sole.phone).to eq("555-0100") + expect(existing.comments.sole.body).to include("Phone", "555-0100", "555-0199") + expect(import.differences.sole).to include("555-0199") + end + end + it "nests each of an organization's several projects under it" do fm_organization("ORG00002", OrgName: "Valley Services") project("P0002", "ORG00002", ProjectName: "Valley Adult Program", City: "Fresno", State: "CA", Locality: "Central CA") diff --git a/spec/tasks/fm_import_organizations_spec.rb b/spec/tasks/fm_import_organizations_spec.rb index e887059ab5..b9d02e58eb 100644 --- a/spec/tasks/fm_import_organizations_spec.rb +++ b/spec/tasks/fm_import_organizations_spec.rb @@ -57,6 +57,22 @@ def run_task(dry_run: true, task: "fm:import_organizations", run: nil) expect(person.affiliations.sole).to have_attributes(organization:, title: "Facilitator", filemaker_code: "PS0001") end + it "turns an agency card into an organization with its work address, and points the person named on it there" do + contact = create(:person, filemaker_code: "00300") + FmAddress.create!(fm_id: "9", data: { "RolodexID" => "00300", "AddrsLine1" => "9 Elm St", "AddrsCity" => "Ojai", + "AddrsState" => "CA", "AddrsType" => "Home" }) + FmRolodex.create!(fm_id: "00300", data: { "isAgency__c" => "1", "Organization" => "Elm Street Clinic", "Title" => "Director", + "FirstName" => "Ada", "LastName" => "Lovelace" }) + + run_task(dry_run: false) + + clinic = Organization.find_by!(name: "Elm Street Clinic") + address = clinic.addresses.sole + expect(address).to have_attributes(street_address: "9 Elm St", address_type: "work") + expect(contact.affiliations.sole).to have_attributes(organization: clinic, title: "Director", organization_address: address) + expect(contact.addresses).to be_empty + end + it "names the run so fm:undo_import can reverse it" do run = run_task(dry_run: false)[/ RUN=(fm-import-organizations-\d+)/, 1] From 1865938f2f5fa8d9af2241d420976fcc92a8e7d0 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sun, 25 Oct 2026 15:44:21 -0400 Subject: [PATCH 22/25] Pin that an undated affiliation on either side updates the other The rule was already in the matching; a spec and the docs now say so, so a later change to date matching can't quietly start duplicating them. Co-Authored-By: Claude Opus 5.5 (1M context) --- ...-10-25-filemaker-organizations-and-affiliations.yml | 1 + ...0008-filemaker-organization-and-personnel-import.md | 5 +++-- .../affiliation_services/import_from_filemaker_spec.rb | 10 ++++++++++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml b/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml index 349eb82646..abf00d286e 100644 --- a/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml +++ b/db/seeds/features/2026-10-25-filemaker-organizations-and-affiliations.yml @@ -27,4 +27,5 @@ pro_tips: - "An organization or affiliation already in the portal is linked rather than duplicated, and only its blank fields are filled in." - "Where FileMaker has an organization's website, mission or type differently, the portal's is kept and a comment on the organization lists what FileMaker had." - "An affiliation with different dates, an address the organization doesn't have, or a new phone number comes in as its own record, leaving the portal's as it was." + - "When either the portal's affiliation or FileMaker's has no dates, the portal's is updated instead of adding a second one." action_path: "/organizations" diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index 7b94b954fe..b0f1f23808 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -93,8 +93,9 @@ Constraints already decided elsewhere: - **One row, one stretch.** A person can facilitate at one org over several separate stretches (2020–2022, 2024–Oct 2025, Feb 2026–now), so a Personnel row links an existing Facilitator row only when their dates agree — each the same or blank on one side, and - still start-before-end once the blanks fill. Any other dates are a separate stretch and - come in as a new affiliation (decided 2026-10-25), leaving the portal's untouched. + still start-before-end once the blanks fill. A row with no dates on either side updates + the other (taking its dates, or the code alone). Any other dates are a separate stretch + and come in as a new affiliation (decided 2026-10-25), leaving the portal's untouched. - **A row naming a project but no person** (29 of the blank-role rows) still records that the program ran. Once every other row is read, an organization nothing else affiliates anyone with gets one Facilitator affiliation for the placeholder person **AWBW diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index c9035caf63..5700a45b4d 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -146,6 +146,16 @@ def stint(starts, ends = nil) expect(person.affiliations.count).to eq(2) end + it "links the portal's dated row when the archive's row has no dates, keeping its dates" do + dated = stint(Date.new(2019, 1, 1), Date.new(2021, 6, 30)) + personnel("PS0001", DateEntered: "") + + import + + expect(dated.reload).to have_attributes(filemaker_code: "PS0001", start_date: Date.new(2019, 1, 1), end_date: Date.new(2021, 6, 30)) + expect(person.affiliations.count).to eq(1) + end + it "links a row with no dates at all, taking the archive's" do undated = stint(nil) personnel("PS0001", DateEntered: "3/1/2024") From 05754d99cae29b5de00cf9b6c1950272aa847a0f Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sun, 25 Oct 2026 15:48:10 -0400 Subject: [PATCH 23/25] Bring agency contact people in from the organizations import MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The people import already ran and its people have been cleaned up, so it won't be re-run — but it skipped every agency card, so the contact person a card names never came in and couldn't be affiliated with the agency. The organizations import now runs the Rolodex import over agency cards alone as its first step, leaving everyone else as the cleanup left them. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 4 ++-- app/models/fm_rolodex.rb | 3 +++ .../import_from_filemaker.rb | 2 +- ...maker-organization-and-personnel-import.md | 6 ++++-- lib/tasks/fm_import_organizations.rake | 21 ++++++++++++------- spec/tasks/fm_import_organizations_spec.rb | 12 +++++++++++ 6 files changed, 35 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c96453d727..8f6ef298fd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -316,7 +316,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `OrganizationServices::NameMatcher` — Resolves a typed organization name tolerating spreadsheet spelling drift: exact (case-insensitive), then `DuplicateFinder.normalized_name` (punctuation), then `DuplicateFinder.suffixless_name` (Inc/LLC/…); a loose match counts only when it fits one org. Indexes names once per instance. Used by the registration CSV import - `OrganizationServices::CaptureFromSubmission` — The shared org-capture for **both** the event registration (`PublicRegistration`) and standalone (`PublicFormSubmission`) flows: resolves the submitted organization by name (`FormField::ORGANIZATION_NAME_FIELD_IDENTIFIER`, case-insensitive per the column collation; never creates one — an unmatched name is left for an admin), syncs its website/type (`SyncProfile`, latest-wins) and work address (`UpsertAddress`, which adds another work address rather than rewriting a different one). `AffiliationServices::CreateFromRegistration` then mints the person's job affiliation from the typed position, plus the standing Facilitator one when `facilitator_training:` — which only an event supplies (`@event.facilitator_training`), so a standalone form gets the job affiliation alone, the same as registering for a non-training event. **Agreement-role submissions never come through here** — they run `LinkSubmittedOrganization`, which confers the scenario's Facilitator affiliation. Returns the org + the autofill changes, which the event flow records on its registration↔org link. `answers:` (identifier → value) stands in for form + params and `organization:` for the name lookup when the caller resolved them (the CSV import); `overwrite: false` only fills the org's blanks - `PersonServices::CaptureFromSubmission` — The shared person-capture for **both** the event registration (`PublicRegistration`) and standalone (`PublicFormSubmission`) flows, run for a newly-created **and** an existing person, so the person smart fields on the Smart form settings page behave identically across both forms. Fills identity extras (pronouns, name `pronunciation`, legal first name, primary email type, secondary email + type), the profile (`racial_ethnic_identity` + `SyncSharingPreferences`), the mailing address, the phone contact method, and sector/age tags (on the person only — organizations roll theirs up from their active facilitators). A non-blank answer overwrites what's on file (newest submission wins); a blank never clobbers. Sector and age tags follow one rule: submitting both a primary and additional set **overwrites** the person's tags with the latest (`replace: true`), submitting only a primary **reassigns** the primary and leaves the rest (additive). Self-gates on each field identifier, so a form that doesn't ask a question no-ops for it. `answers:` (identifier → value) stands in for form + params (the CSV import); `overwrite: false` keeps the person's name, primary address/phone, and primary tags, only filling blanks and adding new entries beside them -- `PersonServices::ImportFromRolodex` — Creates people from FileMaker Rolodex archive rows (`FmRolodex`), carrying the row's ID as `filemaker_code` plus every field `FmRolodex::PORTAL_FIELDS` maps onto a person (profile fields, email type — a precedence word like "Preferred" records none, since the email's own slot already says that — facilitator start → `member_since`, service areas, `Settings`, `LifeExperiences` and `PrimaryAgeGroup` → sectors and age groups translated through `TagTranslation` (ADR-0007; primary fields → primary, the rest additional; one primary sector and one primary age group per person — each kind takes it from the field that owns it, and an answer naming several lets the other field answer instead) via `tag_sectors` / `tag_age_groups`, and the archive's own entry/modification dates as `created_at`/`updated_at`, plus `EnteredBy`/`ModifiedBy` as `created_by`/`updated_by` through `StaffTranslation`, falling back to `Setting.import_user` (picked on `/admin/settings` → Imports) when the name is blank, a shared login, or unresolved, so every person carries both stamps; a linked person keeps any stamps they already have; a name no `StaffAlias` answers to is reported in `#unmatched_staff`). Plus every address and phone the row answers to (`FmRolodex#addresses` / `#phone_numbers`, primary first; both tables are read once via `FmRolodex.contact_index` + `#preload_contacts`), except organization addresses (`FmRolodex#organization_address?`: the worksite one, any typed Work, and every address on an agency's card), which `AffiliationServices::ImportFromFilemaker` and `OrganizationServices::ImportFromFilemaker` place. Links an existing person through `PersonMatcher`, trying each address the row holds in `EMail` (`EmailPrimary` is a flag, never an address: `1` → the first address becomes `email`, a second one the person doesn't already hold `email_2`; `2` → the address and `EmailType` land in `email_2`/`email_2_type`; for a row with no address at all, a same-name person with no email) by adding the row's ID to their `filemaker_code` list — every match when several answer, and alongside any code they already carry — and fills them from the row: blank fields take the archive's value, missing addresses/phones/tags are added (an existing primary stays primary), and a field the archive disagrees on keeps the portal value and is noted as a `Comment` on the person and in `#differences`; creates a new person when nobody matches, even when the email belongs to a differently named person; an agency card naming a contact person becomes that person; skips soft-deleted and nameless rows (a nameless agency card is the agency itself, which the organizations pass reads). Reads raw values through `FilemakerValues` and addresses through `FmAddressAttributes`. A match that doesn't validate is reported under `failed` and left out, so the row's other matches still link; `relink: true` reads named rows past the idempotence guard to pick one up once the person is valid. Exposes `TagTranslation#unmatched` as `#unmatched_tags`. Idempotent on `filemaker_code`. Driven by `fm:import_people` +- `PersonServices::ImportFromRolodex` — Creates people from FileMaker Rolodex archive rows (`FmRolodex`), carrying the row's ID as `filemaker_code` plus every field `FmRolodex::PORTAL_FIELDS` maps onto a person (profile fields, email type — a precedence word like "Preferred" records none, since the email's own slot already says that — facilitator start → `member_since`, service areas, `Settings`, `LifeExperiences` and `PrimaryAgeGroup` → sectors and age groups translated through `TagTranslation` (ADR-0007; primary fields → primary, the rest additional; one primary sector and one primary age group per person — each kind takes it from the field that owns it, and an answer naming several lets the other field answer instead) via `tag_sectors` / `tag_age_groups`, and the archive's own entry/modification dates as `created_at`/`updated_at`, plus `EnteredBy`/`ModifiedBy` as `created_by`/`updated_by` through `StaffTranslation`, falling back to `Setting.import_user` (picked on `/admin/settings` → Imports) when the name is blank, a shared login, or unresolved, so every person carries both stamps; a linked person keeps any stamps they already have; a name no `StaffAlias` answers to is reported in `#unmatched_staff`). Plus every address and phone the row answers to (`FmRolodex#addresses` / `#phone_numbers`, primary first; both tables are read once via `FmRolodex.contact_index` + `#preload_contacts`), except organization addresses (`FmRolodex#organization_address?`: the worksite one, any typed Work, and every address on an agency's card), which `AffiliationServices::ImportFromFilemaker` and `OrganizationServices::ImportFromFilemaker` place. Links an existing person through `PersonMatcher`, trying each address the row holds in `EMail` (`EmailPrimary` is a flag, never an address: `1` → the first address becomes `email`, a second one the person doesn't already hold `email_2`; `2` → the address and `EmailType` land in `email_2`/`email_2_type`; for a row with no address at all, a same-name person with no email) by adding the row's ID to their `filemaker_code` list — every match when several answer, and alongside any code they already carry — and fills them from the row: blank fields take the archive's value, missing addresses/phones/tags are added (an existing primary stays primary), and a field the archive disagrees on keeps the portal value and is noted as a `Comment` on the person and in `#differences`; creates a new person when nobody matches, even when the email belongs to a differently named person; an agency card naming a contact person becomes that person; skips soft-deleted and nameless rows (a nameless agency card is the agency itself, which the organizations pass reads). Reads raw values through `FilemakerValues` and addresses through `FmAddressAttributes`. A match that doesn't validate is reported under `failed` and left out, so the row's other matches still link; `relink: true` reads named rows past the idempotence guard to pick one up once the person is valid. Exposes `TagTranslation#unmatched` as `#unmatched_tags`. Idempotent on `filemaker_code`. Driven by `fm:import_people`, and by `fm:import_organizations` for agency cards only - `PersonServices::UndoRolodexImport` — Reverses one FileMaker import run (`fm:import_people` or `fm:import_organizations`) from the Ahoy lifecycle events it tagged with its `import_run`: newest first, deletes each record the run created and reverts each field it filled, unless the record was edited since (its `updated_at` is newer than the event) or the field has changed — those are kept and reported. Runs under `no_touching` so deleting a person's address doesn't make the person look edited. Driven by `fm:undo_import_people` (alias `fm:undo_import`) - `PersonServices::SyncSharingPreferences` — Writes the two content-sharing answers (`anonymous_contributions`, `display_name_preference`) onto the `Person`, mapping each submitted option label back to its stored value; an unanswered or unrecognized answer leaves the profile alone, so a form that doesn't ask the questions is a no-op and every flow can call it unconditionally. The person-side counterpart to `OrganizationServices::SyncProfile`. Shared by `PersonServices::CaptureFromSubmission` (so a submission's answers land in the "what this submission changed" audit via the Ahoy trail) and `CalloutFormSubmission` (whose `#profile_changes` re-reports the same edits for its own Ahoy tracking) - `EventRegistrationServices::TransferContinuingEducation` — Splits/relocates a registrant's CE when they transfer events (issue #1944): a simple forward transfer leaves a paid, zero-hours **stub** on the source (its payments count at the original event) and creates a **live** record on the destination carrying the hours and the outstanding balance; when the reg being transferred out is itself a transfer-in (a collapsing double transfer, or a transfer back to the origin) its live record is relocated forward — merging back into the origin's stub — instead of split again, so no third record appears. Runs inside the transfer transaction, after the destination is saved and before a collapsing middle reg is destroyed @@ -621,7 +621,7 @@ Located in `lib/tasks/` (19 files): - `backfill_notification_stamps.rake` — Backfill notifications' `created_by_id`/`updated_by_id` from `sender_id`, nil-only (`data:backfill_notification_stamps`) - `fm_import.rake` — Load the FileMaker CSV exports into the `fm_*` archive tables (`fm:import`, reads each CSV from the `FM Archive` resources, batch `insert_all`, skips rows whose key is already stored) - `fm_import_people.rake` — Create people from the FileMaker Rolodex archive (`fm:import_people`, via `PersonServices::ImportFromRolodex`, links existing by name + email, idempotent, DRY_RUN; VERBOSE=true lists every row, FM_IDS=a,b re-runs just those rows past the idempotence guard; commits every 1,000 rows, prints progress, saves the full report as an ActiveStorage blob even on a crash, and aborts when no `TagAlias` rows exist or no `Setting.import_user` is chosen; logs every record it creates or fills as activity under a run name (`Current.source = "fm_import_people"`), which `fm:undo_import_people RUN=…` reverses via `PersonServices::UndoRolodexImport`) -- `fm_import_organizations.rake` — Create organizations and affiliations from the FileMaker archive after `fm:import_people` (`fm:import_organizations`: `OrganizationServices::ImportFromFilemaker` then `AffiliationServices::ImportFromFilemaker`; idempotent, DRY_RUN rolls back as a whole while a real run commits per batch, VERBOSE=true lists every row; prints progress, the skipped/failed rows, differences, unmatched org types and the Personnel status/role tallies, and saves the report as an ActiveStorage blob; aborts with no `Setting.import_user`; logs under a run name that `fm:undo_import RUN=…` reverses) +- `fm_import_organizations.rake` — Create organizations and affiliations from the FileMaker archive after `fm:import_people` (`fm:import_organizations`: `PersonServices::ImportFromRolodex` over `FmRolodex.agency_cards` — the contact people `fm:import_people` skipped — then `OrganizationServices::ImportFromFilemaker`, then `AffiliationServices::ImportFromFilemaker`; idempotent, DRY_RUN rolls back as a whole while a real run commits per batch, VERBOSE=true lists every row; prints progress, the skipped/failed rows, differences, unmatched org types and the Personnel status/role tallies, and saves the report as an ActiveStorage blob; aborts with no `Setting.import_user`; logs under a run name that `fm:undo_import RUN=…` reverses) - `resolve_resource_organizations.rake` — Link resources to organizations from the legacy `agency` free-text name (`data:resolve_resource_organizations`, match-or-create, idempotent, DRY_RUN) - `seed_tag_aliases.rake` — Record the legacy sector and age-group terms from ADR-0007 as tag aliases (`data:seed_tag_aliases`, via `TagAliasSeeder`, idempotent; also part of `db:seed`) - `load_staff_aliases.rake` — Record FileMaker staff names → portal users as staff aliases from `FILE=` (`data:load_staff_aliases`, via `StaffAliasLoader`, idempotent) diff --git a/app/models/fm_rolodex.rb b/app/models/fm_rolodex.rb index 2d376a334d..81f4cb8267 100644 --- a/app/models/fm_rolodex.rb +++ b/app/models/fm_rolodex.rb @@ -225,6 +225,9 @@ def addresses @addresses ||= contact_records(FmAddress, primary_ids: address_ids) end + # Cards flagged as an agency's, narrowed in SQL; #agency? reads the flag exactly. + scope :agency_cards, -> { where("JSON_UNQUOTE(JSON_EXTRACT(data, '$.isAgency__c')) NOT IN ('', '0', 'n', 'no', 'false', 'null')") } + # The row is an agency's own contact card, sometimes naming the person to reach there. def agency? FilemakerValues.flag?(data["isAgency__c"]) diff --git a/app/services/organization_services/import_from_filemaker.rb b/app/services/organization_services/import_from_filemaker.rb index f7864fae15..e18d7e876e 100644 --- a/app/services/organization_services/import_from_filemaker.rb +++ b/app/services/organization_services/import_from_filemaker.rb @@ -241,7 +241,7 @@ def agency_outcome(organization) end def agency_cards - FmRolodex.where("JSON_UNQUOTE(JSON_EXTRACT(data, '$.isAgency__c')) <> ''").order(:fm_id).select(&:agency?) + FmRolodex.agency_cards.order(:fm_id).select(&:agency?) .reject { |rolodex| flag?(rolodex.data["Delete"]) } end diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index b0f1f23808..c58af6a93a 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -64,8 +64,10 @@ Constraints already decided elsewhere: - **Agencies:** a Rolodex card flagged `isAgency__c` is the agency's own card. Its website, mission and addresses fill the org its `OrgID` names (or the one its `Organization` name matches, created when nothing does); its email too, unless the card names a contact - person, whose email it then is. A card naming a contact person also becomes that person - in the people pass, and every address on it stays with the org. + person, whose email it then is. A card naming a contact person also becomes that person, + and every address on it stays with the org. `fm:import_people` skipped every agency card, + so `fm:import_organizations` runs the Rolodex import over agency cards alone as its first + step — the people already imported and cleaned up are left as they are. - **Personnel → facilitator affiliation:** an `fm_personnels` row becomes an `Affiliation` between the person (by Rolodex code) and the project's org, carrying `PrsnlRecID` as its `filemaker_code` so each row imports once. `event_registration_id` stays NULL — these are diff --git a/lib/tasks/fm_import_organizations.rake b/lib/tasks/fm_import_organizations.rake index 924c173869..59cc0725e4 100644 --- a/lib/tasks/fm_import_organizations.rake +++ b/lib/tasks/fm_import_organizations.rake @@ -2,12 +2,14 @@ # Bring FileMaker's organizations, projects and the ties between them and people into the # portal (ADR-0008). Run after fm:import_people, which makes the people; this pass finds them -# by their Rolodex code. Two steps, in order: -# 1. OrganizationServices::ImportFromFilemaker — organizations, projects (as programs nested +# by their Rolodex code. Three steps, in order: +# 1. PersonServices::ImportFromRolodex over agency cards only — the contact person an agency +# card names, which fm:import_people skipped +# 2. OrganizationServices::ImportFromFilemaker — organizations, projects (as programs nested # under them), their addresses, phones, emails, and agencies' own Rolodex cards -# 2. AffiliationServices::ImportFromFilemaker — personnel, people-to-organizations, and each +# 3. AffiliationServices::ImportFromFilemaker — personnel, people-to-organizations, and each # Rolodex card's organization, title and work addresses -# Both are idempotent: a row whose code a record already carries is left alone. +# Each is idempotent: a row whose code a record already carries is left alone. # # Dry-run by default. Set DRY_RUN=false to write changes; VERBOSE=true lists every row. # A real run commits each batch as it goes, so stopping partway keeps what it finished and @@ -34,17 +36,20 @@ namespace :fm do Current.source = "fm_import_organizations" say.("Run #{run}#{" (dry run)" if dry_run}") started = Time.current + # The people pass skipped every agency card, so the contact person one names comes in + # here — only those cards, leaving everyone fm:import_people already brought over alone. steps = { - "Organizations and projects" => OrganizationServices::ImportFromFilemaker, - "Affiliations and work addresses" => AffiliationServices::ImportFromFilemaker + "People named on agency cards" => ->(progress) { PersonServices::ImportFromRolodex.new(scope: FmRolodex.agency_cards, progress:, run:) }, + "Organizations and projects" => ->(progress) { OrganizationServices::ImportFromFilemaker.new(progress:, run:) }, + "Affiliations and work addresses" => ->(progress) { AffiliationServices::ImportFromFilemaker.new(progress:, run:) } } imports = {} run_steps = lambda do - steps.each do |title, service| + steps.each do |title, build| say.("\n#{title}…") progress = ->(read) { puts " #{read} rows read — #{(Time.current - started).round}s" } - imports[title] = service.new(progress:, run:).call + imports[title] = build.call(progress).call end end diff --git a/spec/tasks/fm_import_organizations_spec.rb b/spec/tasks/fm_import_organizations_spec.rb index b9d02e58eb..2136ff7d9b 100644 --- a/spec/tasks/fm_import_organizations_spec.rb +++ b/spec/tasks/fm_import_organizations_spec.rb @@ -73,6 +73,18 @@ def run_task(dry_run: true, task: "fm:import_organizations", run: nil) expect(contact.addresses).to be_empty end + it "brings in the person an agency card names, which the people import skipped, and affiliates them there" do + FmRolodex.create!(fm_id: "00301", data: { "isAgency__c" => "1", "Organization" => "Harbor Clinic", "Title" => "Director", + "FirstName" => "Grace", "LastName" => "Hopper", "EMail" => "grace@harbor.example.org" }) + FmRolodex.create!(fm_id: "00302", data: { "FirstName" => "Ada", "LastName" => "Lovelace" }) + + run_task(dry_run: false) + + contact = Person.find_by!(filemaker_code: "00301") + expect(contact.affiliations.sole).to have_attributes(organization: Organization.find_by!(name: "Harbor Clinic"), title: "Director") + expect(Person.where(filemaker_code: "00302")).to be_empty + end + it "names the run so fm:undo_import can reverse it" do run = run_task(dry_run: false)[/ RUN=(fm-import-organizations-\d+)/, 1] From 0104f9b25a91edef7460f8cd5adb8b70c5a484bb Mon Sep 17 00:00:00 2001 From: maebeale Date: Sun, 25 Oct 2026 16:39:21 -0400 Subject: [PATCH 24/25] FileMaker codes on imported addresses and phones; phone records on organizations (#2860) * Start: FileMaker codes on imported addresses and phones Co-Authored-By: Claude Opus 5.5 (1M context) * Trace imported addresses and phones back to their FileMaker rows People carried their Rolodex code, but the addresses and phones the import brought along didn't, so the archive couldn't say which of its rows had landed or where. Each now carries its AddrsID / PhoneID, an address or number the person already held takes the code instead of a copy, and merges keep every code. Co-Authored-By: Claude Opus 5.5 (1M context) * Give organizations phone records, and trace their FileMaker addresses and phones The organizations import squeezed an org's numbers onto its addresses, dropping any with no free address and leaving no record to carry the FileMaker code. Orgs now keep phones as contact methods like people, edited on the org form and folded on merge, so each imported number and address carries its archive ID. ReconcileContactMethods moves to ContactMethodDeduper since orgs share it now. Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- AGENTS.md | 10 +- app/controllers/fm_archives_controller.rb | 2 +- app/controllers/organizations_controller.rb | 5 +- app/controllers/people_controller.rb | 4 +- app/helpers/filemaker_helper.rb | 2 + app/models/address.rb | 2 + app/models/contact_method.rb | 3 + app/models/organization.rb | 8 ++ app/services/address_deduper.rb | 7 +- .../import_from_filemaker.rb | 26 ++++-- app/services/contact_method_deduper.rb | 91 ++++++++++++++++++ .../import_from_filemaker.rb | 59 ++++++------ .../person_services/import_from_rolodex.rb | 42 ++++++--- .../reconcile_contact_methods.rb | 92 ------------------- app/views/organizations/_form.html.erb | 14 +++ app/views/organizations/show.html.erb | 6 +- ...r_code_to_addresses_and_contact_methods.rb | 11 +++ db/schema.rb | 4 +- ...ilemaker-codes-on-addresses-and-phones.yml | 13 +++ .../2026-10-25-organization-phones.yml | 14 +++ .../0005-filemaker-code-archive-mapping.md | 8 +- ...maker-organization-and-personnel-import.md | 5 +- spec/models/organization_spec.rb | 20 ++++ spec/requests/dedupable_spec.rb | 14 +++ spec/requests/fm_archives_spec.rb | 22 +++++ spec/requests/organizations_spec.rb | 8 ++ spec/services/address_deduper_spec.rb | 9 ++ .../import_from_filemaker_spec.rb | 18 +++- ...spec.rb => contact_method_deduper_spec.rb} | 11 ++- .../import_from_filemaker_spec.rb | 47 ++++++---- .../import_from_rolodex_spec.rb | 36 +++++++- 31 files changed, 424 insertions(+), 189 deletions(-) create mode 100644 app/services/contact_method_deduper.rb delete mode 100644 app/services/person_services/reconcile_contact_methods.rb create mode 100644 db/migrate/20261025200004_add_filemaker_code_to_addresses_and_contact_methods.rb create mode 100644 db/seeds/features/2026-10-25-filemaker-codes-on-addresses-and-phones.yml create mode 100644 db/seeds/features/2026-10-25-organization-phones.yml rename spec/services/{person_services/reconcile_contact_methods_spec.rb => contact_method_deduper_spec.rb} (90%) diff --git a/AGENTS.md b/AGENTS.md index 8f6ef298fd..4b340c5eec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -111,7 +111,7 @@ This codebase (Rails 8.1) | `StaffTagging` | Polymorphic join linking a `StaffTag` to the record it tags (`staff_taggable`); `created_by`/`updated_by` (stamped from `Current.user`) record which admin applied and last touched it. Has its own admin edit page (`StaffTaggingsController`, `/staff_taggings/:id/edit`) rendering the shared comments & communications section (`Communicable` + `commentable`) and a tag reassignment select, with communications keyed on the taggable person's email; plus a searchable/filterable index (`/staff_taggings`, `search_by_params` over tag + tagged person's name/email/org) | | `StaffAlias` | A name FileMaker stamped in `EnteredBy`/`ModifiedBy` and the portal `User` it means, so an import can fill `created_by`/`updated_by` (unrelated to `StaffTag`). Names a user **or** is `ignored` (a shared login like "Staff"), never both or neither; unique by name, case-insensitively. Loaded from a staff list kept outside the repo (it pairs real names with accounts) via `data:load_staff_aliases`; resolved by `StaffTranslation` | | `OtherResponse` | A free-text "Other" typed on a form question, captured at submission time (registration, scholarship, bulk payment). Polymorphic `owner`: a **sector** "Other" is owned by the `Person` (promotable into a `Sector`, shown on their profile/edit chip); an **organization_type** "Other" is owned by the `Organization` (stored now, not promotable until `OrganizationType` is a model). `generic` questions aren't captured — that stays searchable in the form answers. `field_identifier` records the question; `kind` is derived. Curated at `/other_responses` (grouped by kind/question): `promote` (sectors only), `keep`, `dismiss`. `dismissed` hides the chip from the profile but stays in the review queue (still promotable later); only `promoted` leaves the queue. Admins deep-link there from a person's chip. | -| `Organization` | Groups with affiliations, addresses, logos via ActiveStorage; self-nests under a parent org (`parent_id` self-FK, `#ancestors`/`#descendants`/`#root`, structural only) per ADR-0004. Has no sectors/age groups of its own: `active_facilitator_sectors` / `active_facilitator_age_groups` roll up its active Facilitator affiliations' primary tags for profile/index/edit chips; search (index filters, tag search, tag matrix, public tag list via its own `sector_names_all` / `category_names_all` scopes) and the populations-served page count primary + additional | +| `Organization` | Groups with affiliations, addresses, phones (`contact_methods`, as on `Person`; `#phone_number` falls back to an active address's `phone`), logos via ActiveStorage; self-nests under a parent org (`parent_id` self-FK, `#ancestors`/`#descendants`/`#root`, structural only) per ADR-0004. Has no sectors/age groups of its own: `active_facilitator_sectors` / `active_facilitator_age_groups` roll up its active Facilitator affiliations' primary tags for profile/index/edit chips; search (index filters, tag search, tag matrix, public tag list via its own `sector_names_all` / `category_names_all` scopes) and the populations-served page count primary + additional | | `Grant` | Funds (polymorphic `funder`: Organization or Person) with eligibility criteria, tasks, deadlines; parent of `Scholarship`. Scholarship totals cannot exceed the grant amount | | `Scholarship` | Award to a `Person`; optionally drawn from a `Grant`, syncs to event registration `Allocation`. Tri-state `agreement_response_status` (pending/accepted/declined) drives the agreement; declined awards zero their allocation and drop out of all totals | | `ScholarshipAgreementResponse` | Append-only history of a scholarship's accept ↔ decline back-and-forth (status, reason, responder, amount at the time); the scholarship's `agreement_response_status` is the denormalized latest row, and `responded_at`/reason are read from the latest response, not stored on the scholarship | @@ -150,7 +150,7 @@ This codebase (Rails 8.1) | `Communicable` | Gives a record the `notifications` association, nested attributes, a default `communications_scope`, and a `before_validation` addressing a hand-logged communication to the record's `communications_email`. Included by every model whose form renders `shared/_comments_and_communications` | | `AuthorCreditable` | Author attribution. Credits are formatted by the credited **person's profile** (`Person#display_name_preference`), not by the record. The record's `author_credit_preference` is the consent snapshot taken at create time and is human-editable only on the author credit divergences page. Anonymity is a one-way latch: the profile or the record can set it, neither can strip it from the other. `credits_to_org` flips the unattributed label to "AWBW Staff"; `credits_creator` (idea models, workshop logs) credits the submitting account's person when no author is named | | `Featureable` | `featured`, `publicly_featured` scopes | -| `FilemakerCoded` | Reads `filemaker_code` as the trimmed, comma-separated **list** it is (Person, Organization, Affiliation, Payment, Workshop, Resource): `#filemaker_codes` for the set a record answers to, `.join_filemaker_codes` to combine any mix of strings/lists into one normalized column value (both dedupers union the codes on merge). `filemaker_archived_in` names the `Fm*` archive table(s) a model's codes point into (`.filemaker_archives`), and `.with_filemaker_code(code)` finds records carrying a code. Nothing should compare the whole column against a single code — `FilemakerHelper#filemaker_code_links` renders one archive link per code, and `fm_archives#show` links back to the portal records carrying its code | +| `FilemakerCoded` | Reads `filemaker_code` as the trimmed, comma-separated **list** it is (Person, Organization, Affiliation, Payment, Workshop, Resource, Address, ContactMethod): `#filemaker_codes` for the set a record answers to, `.join_filemaker_codes` to combine any mix of strings/lists into one normalized column value (both dedupers union the codes on merge, as do `AddressDeduper` and `ContactMethodDeduper`). `filemaker_archived_in` names the `Fm*` archive table(s) a model's codes point into (`.filemaker_archives`), and `.with_filemaker_code(code)` finds records carrying a code. Nothing should compare the whole column against a single code — `FilemakerHelper#filemaker_code_links` renders one archive link per code, and `fm_archives#show` links back to the portal records carrying its code | | `Mentionable` | ActionText mention grouping, plus `connected_records(klass)` — records linked by a mention in either direction (the only Resource↔Workshop link: `Resource#connected_workshops`, `Workshop#connected_resources`) | | `NameFilterable` | Name-based filtering | | `OutsideUsaTypeable` | `typed_outside_usa :state` adds the `_outside_usa` box `shared/_outside_usa` reveals beside a US-state dropdown; a typed value wins whichever is assigned first (ProfessionalLicense, Address, ProfileChangeRequest) | @@ -279,12 +279,12 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `AddressLocality` — Works out which `Address::LOCALITIES` bucket an address falls in from its own city/state/country/county, for a writer that has an address but never asked the question (`PersonServices::ImportFromRolodex`, and through `AddressDefaults` for the rest — a locality a person picked is never overwritten). Reads a named non-US country as **Outside USA**, a non-CA US state as **Outside CA**, a mailing city inside the City of Los Angeles — Los Angeles or a neighborhood (`LA_CITY_MAILING_CITIES`) — as **LA City**, the rest of Los Angeles County as **LA County**, Orange County as **Orange County**, and any other California county by `CALIFORNIA_REGIONS` (the region most FileMaker projects in that county name, plus a hand-checked few) as **Northern/Central/Southern CA**. A state no US state answers to, with no country, is **Outside USA** (the forms only offer US states). Everything else stays **Unknown**: a California address with no or an unplaced county, and a US address whose state no US state answers to. Every comparison goes through `AddressNormalizer`. FileMaker records a `Locality` only on a **project**, so `OrganizationServices::ImportFromFilemaker` carries that across rather than deriving it (falling back here when a project names none); a person's own address has none - `OutsideUsaState` — Resolves a US-state dropdown + "Not on list?" box pair: `.pick(dropdown, typed)` for raw params (CE callout, admin CE registration) and `.form_field_answers(params, scope)`, which lays `form_fields_outside_usa` answers over `form_fields` for every `_state` question (public forms, registration, callout forms). The box itself is `shared/_outside_usa` — a no-JS `
    ` - `AddressDeduper` — Merges a person's or organization's addresses that are one place into a single address. `.matching_sets` groups by `AddressNormalizer.signature`, then merges a less complete copy (blank ZIP, street, country, …) into the single fuller address whose every part it shares — skipped when it fits more than one; works fullest-first so a chain of partial copies lands on the fullest. Each merge goes through `ModelDeduper#merge`, which repoints the merged address's inbound FKs — `contact_methods.address_id`, `invoices.bill_to_address_id`, `affiliations.organization_address_id` — onto the survivor. Within a set the survivor is the copy still in use — active before inactive, then the primary, then the oldest (`.keeper_order`) — since `Address.active` drives bill-to addresses, an org's program location and the geographic breakdowns; blank columns on the survivor (`CARRIED_COLUMNS` — the street/ZIP/country/county/district/type/locality, `phone` and the three `la_*` districts; "unknown" type/locality counts as blank) are filled from the merged copies. Then settles a single primary on the set holding the signature captured in the deduper's `merge_keeper` (the survivor's own pre-merge primary address, else the deleted record's), demoting the rest. Runs in the people and organizations dedupers' `after_merge`; `AddressDeduper.merge_duplicates(addressable)` runs it standalone (keeping the owner's current primary) for the **Dedupe addresses** admin page (`AddressesController#dedupe_index`/`#dedupe_perform`) +- `ContactMethodDeduper` — After a person or organization merge, merges duplicate contact methods (phone/sms/whatsapp) the generic merge moves onto the survivor wholesale (contact_methods have no unique index). Groups by normalized `kind`+`value`+`contact_type` and merges duplicates via `ModelDeduper#merge` — survivor chosen active before inactive, then primary, then oldest (`#phone_number` reads only active phones), carrying the duplicate's `address_id` up when the survivor has none and unioning `filemaker_code` — then keeps one primary per kind matching the signatures captured in `merge_keeper` (the survivor's own pre-merge primary of that kind, else the deleted record's), demoting the rest. Runs in `after_merge` - `AddressServices::DuplicateFinder` — Lists the people and organizations holding addresses that `AddressDeduper.matching_sets` would merge, for the **Dedupe addresses** page. Compares only within one owner — two orgs at one street are separate records, not duplicates - `PersonServices::ReconcilePrimaryDesignations` — After a person merge, keeps the survivor's own primary sector + primary age range and demotes the duplicate's, which would otherwise leave the kept person with two primaries and trip Person's single-primary validations. The survivor's pre-merge primary ids are captured in the people deduper's `merge_keeper` (they're indistinguishable once merged); reconciliation runs in `after_merge` - `ProfileChangeRequests::Apply` — Applies an auto-appliable `ProfileChangeRequest` when an admin clicks **Approve**: primary email via `UserServices::ProcessEmailChange` (sets `unconfirmed_email`, sends the confirmation), secondary email by writing `email_2` (+ `email_2_type`), the requested email type onto the person (and onto the login with an address change, so the confirm-time sync keeps it), organization name by renaming the person's `primary_organization`, and the structured affiliation categories (title, dates, org rename, remove — a Facilitator row closes the program via `AffiliationServices::CloseProgram` — add an existing org (ending affiliations at the requester's `left_organization_ids`, plus a Facilitator row when the reviewer passes `confer_facilitator:`), a work address picked from the org's list or entered as street/city/state/ZIP). Free-text affiliation requests return `applied: false` (the admin edits by hand, then marks the request resolved). Returns a `Result(applied, message)` - `PersonServices::ReconcileProfessionalLicenses` — After a person merge, merges redundant licenses the generic `(kind, number)` collapse leaves behind. Within each kind, every blank-number license (nil or `""`) merges into a sibling — preferring one with a real number — via `ModelDeduper`, so its CE registrations move onto the survivor. Left as separate rows: two different real numbers of the same kind, and licenses of different kinds (a shared number across kinds is allowed). Runs first in the people deduper's `after_merge`, so any CE it moves is then collapsed by `ContinuingEducationDeduper` - `PersonServices::ReconcileDefaultPayCustomer` — After a person merge, settles the survivor to a single active default Pay customer. Merging two billable people can leave the survivor owning two live `pay_customers` both flagged `default` (the soft-delete unique index doesn't bound them in MySQL, so `ModelDeduper` moves both instead of collapsing), which makes Pay's `payment_processor` (`has_one … default: true`) ambiguous — and a later checkout (`set_payment_processor` clears every default, then re-picks `.first` active customer) could silently flip it. Picks the winner — the customer carrying an active, uncanceled subscription (`Pay::Subscription.active_uncanceled`, so membership autopay reading `payment_processor.subscribed?` off the default keeps working), else the survivor's own pre-merge default (id captured in the people deduper's `merge_keeper`), else the newest — marks it `default: true`, and **soft-deletes** each redundant same-processor sibling (sets `deleted_at`; charges/subscriptions are preserved, not destroyed) so it drops out of the active/checkout scope. A customer with an active subscription is never soft-deleted. No-op unless more than one live customer remains. Runs in `after_merge`. Two people who each carry an active subscription can't be merged — `PeopleController`'s `merge_blocks` blocks it (via `Dedupable`), telling the admin to cancel one first -- `PersonServices::ReconcileContactMethods` — After a person merge, merges duplicate contact methods (phone/sms/whatsapp) the generic merge moves onto the survivor wholesale (contact_methods have no unique index). Groups by normalized `kind`+`value`+`contact_type` and merges duplicates via `ModelDeduper#merge` — survivor chosen active before inactive, then primary, then oldest (`Person#phone_number` reads only active phones), carrying the duplicate's `address_id` up when the survivor has none — then keeps one primary per kind matching the signatures captured in `merge_keeper` (the survivor's own pre-merge primary of that kind, else the deleted person's), demoting the rest. Runs in `after_merge` - `RichTextMigrator` — Rich text migration utility - `EventRegistrationImporter` — Bulk-imports registrants into one existing event (live or on-demand, training or not) from an uploaded CSV (stdlib `csv`). Only FirstName/LastName are required; `Columns::TEMPLATE_HEADERS` backs a downloadable CSV template (one spelling per readable column, `#template`, spec-held to `HEADER_ALIASES`) and `EventRegistrationImporter::Columns` maps loosely-matched headers (case/spacing/punctuation ignored) for status, Day 1–5, registered date, person extras, org + org address + position, mailing address/phone, sectors/age groups (by name), CE license, scholarship, payment, and follow-up flags, and the preview lists unrecognized headers. **Person:** a `FileMakerCode`/`ParticipantID` the Rolodex import already recorded on somebody matches them outright (`Person.with_filemaker_code`), and makes the name columns optional for that row; otherwise, with an email, `PersonMatcher` (PublicRegistration's rule); without, `PersonNameMatcher` (name at the row's org, else a name only one person has); a miss creates the person (attributed to the importing admin), an email-less one with `Person#next_free_placeholder_email` (numbered per name, so the same placeholder serves two different names). **Status:** the Status column (blank → `attended`, or derived from Day N columns when only those are filled), applied only where the portal holds no deliberate outcome — an attended / cancelled / no-show / transferred-out / incomplete-attendance registration keeps its status and the sheet's lands as a comment on it (`PORTAL_WINS_STATUSES`, topic "Spreadsheet import"), with the row's Day columns left out since they describe the outcome that lost, so one person still holds one registration per event; attended marks every day complete; a status the import does apply is saved without emails (`skip_cancellation_emails`). **Org:** `OrganizationServices::NameMatcher` (exact, then punctuation/legal-suffix tolerant), then `OrganizationServices::CaptureFromSubmission` (website/type/work address/job + facilitator affiliations) with `answers:` and `overwrite: false`; an unmatched name lands in the link-organization queue via the import's OWN `FormSubmission` (role registration, marked `imported_from`, created alongside never in place of a real one, carrying the answers this event's form asks). **Profile:** `PersonServices::CaptureFromSubmission` with `answers:` — a matched person's values are kept (`overwrite: false` only fills blanks and adds new addresses/phones as non-primary); what fill-only declined comes back on `#declined` (and, for the org, from `OrganizationServices::ProfileDiff`) and is reported as a row warning **and** a comment on that person/organization, so a curated sheet's value is never dropped silently. **CE** via `ContinuingEducationRegistration.enroll!` when the event offers hours and the row asks (`Columns.requested?` — a Yes/No cell or FileMaker's `CEHours` count). `Notes`/`Comments` become comments on the registration; a row whose `NoOfPersons` is above 1 stands for a group: the person it names is registered, the payment becomes a bulk payment for that many heads with the unnamed ones padded as `UNNAMED_ATTENDEE` ("TBD") for an admin to add and link, and the count is noted on the registration. **Money:** planned per row (`set_money_preview` records what's owed), then applied once every row is read, so rows sharing a `BulkPayment` reference — or one check number — become **one payment split across their registrations** rather than a full payment each (`apply_payment_batch`; an amount covering the group's whole bill is read as the payment, anything less as a share of it, and the payer is the organization they share, else the first registrant); a group payment also gets a role `bulk_payment` `FormSubmission` (attendees + linked registrations) so it lists on the event's **Bulk payments** page, reusing `Event#bulk_payment_form` or attaching a `Form.standalone` bulk payment form, creating one only when the install has none. Order per member: an accepted Scholarship, then the payment (registration, then a CE *the row asked for* — a CE it doesn't mention is left untouched), then a Discount comping what a paying row still owes. One calculation drives both the preview and the run; a row the scholarship already covers reports `:nothing_owed` and records no payment. Idempotent, sends **no emails**, never creates organizations. Requires a registration form with an org field (`.importable?`). Runs in a transaction with a `dry_run` mode that rolls back for the preview; returns a `Result` + per-row `RowPreview`s. Drives `EventRegistrationImportsController` (`event_registrations/import`, admin-only via `EventRegistrationPolicy#import?`; linked from the registrants page's Bulk actions menu and admin home's Actions) - `StoryImporter` — Imports stories from the curated stories spreadsheet (the columns this importer's own preview produces, reviewed/edited by staff and re-uploaded; headers in `StoryImporter::TEMPLATE_HEADERS`). Every importable row becomes a Story; the `status` column drives publish state ("Published story" vs "Draft story"), whether a StoryIdea is also promoted ("… + story idea"; organization optional), and `Skipped — …` rows. Column headers may be the curated sheet's human labels (`COLUMN_ALIASES`, any order) or the snake_case keys, and a duplicate human-label header row is skipped. Trusts the sheet's resolved taxonomy — looks up `sectors`/`window_type` (exact match; unknown window type skips the row) by name and `categories` as "Type: Name" — an age group (Children/Teens/Adults/Elders) always resolves to its `AgeRange` category whichever type the sheet filed it under, so the retired StoryPopulation age twins (`Teens_`) are never tagged; either kind of audience can be the starred `primary_story_population`, and flags `is_primary` on the join rows matching `primary_sector` / `primary_story_population` (tagging them even if absent from the sectors/categories lists); taxonomy, grants, or facilitator names that can't be matched are preserved as Comments on the story rather than invented. Strips HTML from titles and renames a duplicate title to `[COPY N]` instead of skipping. Resolves up to two author People (first + optional co-author) from parallel `facilitator_*`/`co_facilitator_*` columns, each with its own `name_display`/`anonymous` — matched through the shared `PersonMatcher` (email in any of the person's three addresses + last name + stored-or-legal first name), falling back to an exact name for a row with no email — `AWBW` rows pull the real name out of the last-name column and flag the profile anonymous; a co-author sticks only behind a distinct first author. Credit is synced to each author's profile and left blank on the story (both `author_credit_preference` and `co_author_credit_preference`) unless that person's rows disagree. Both authors and the workshop link carry onto the promoted StoryIdea as well; the workshop links through the `story_workshops`/`story_idea_workshops` join (matched Workshop or free-text external title). Links `grants` through the author's Scholarship, attaches comma-separated `professional_licenses` (by kind) to the author, records the source `wp_id` as a Comment, converts `content` via wpautop, enqueues a `StoryAssetImportJob` per story to download its `image_urls` (with per-image `image_alt_titles`) in the background, and returns a row-by-row preview for the dry-run interstitial @@ -316,7 +316,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve - `OrganizationServices::NameMatcher` — Resolves a typed organization name tolerating spreadsheet spelling drift: exact (case-insensitive), then `DuplicateFinder.normalized_name` (punctuation), then `DuplicateFinder.suffixless_name` (Inc/LLC/…); a loose match counts only when it fits one org. Indexes names once per instance. Used by the registration CSV import - `OrganizationServices::CaptureFromSubmission` — The shared org-capture for **both** the event registration (`PublicRegistration`) and standalone (`PublicFormSubmission`) flows: resolves the submitted organization by name (`FormField::ORGANIZATION_NAME_FIELD_IDENTIFIER`, case-insensitive per the column collation; never creates one — an unmatched name is left for an admin), syncs its website/type (`SyncProfile`, latest-wins) and work address (`UpsertAddress`, which adds another work address rather than rewriting a different one). `AffiliationServices::CreateFromRegistration` then mints the person's job affiliation from the typed position, plus the standing Facilitator one when `facilitator_training:` — which only an event supplies (`@event.facilitator_training`), so a standalone form gets the job affiliation alone, the same as registering for a non-training event. **Agreement-role submissions never come through here** — they run `LinkSubmittedOrganization`, which confers the scenario's Facilitator affiliation. Returns the org + the autofill changes, which the event flow records on its registration↔org link. `answers:` (identifier → value) stands in for form + params and `organization:` for the name lookup when the caller resolved them (the CSV import); `overwrite: false` only fills the org's blanks - `PersonServices::CaptureFromSubmission` — The shared person-capture for **both** the event registration (`PublicRegistration`) and standalone (`PublicFormSubmission`) flows, run for a newly-created **and** an existing person, so the person smart fields on the Smart form settings page behave identically across both forms. Fills identity extras (pronouns, name `pronunciation`, legal first name, primary email type, secondary email + type), the profile (`racial_ethnic_identity` + `SyncSharingPreferences`), the mailing address, the phone contact method, and sector/age tags (on the person only — organizations roll theirs up from their active facilitators). A non-blank answer overwrites what's on file (newest submission wins); a blank never clobbers. Sector and age tags follow one rule: submitting both a primary and additional set **overwrites** the person's tags with the latest (`replace: true`), submitting only a primary **reassigns** the primary and leaves the rest (additive). Self-gates on each field identifier, so a form that doesn't ask a question no-ops for it. `answers:` (identifier → value) stands in for form + params (the CSV import); `overwrite: false` keeps the person's name, primary address/phone, and primary tags, only filling blanks and adding new entries beside them -- `PersonServices::ImportFromRolodex` — Creates people from FileMaker Rolodex archive rows (`FmRolodex`), carrying the row's ID as `filemaker_code` plus every field `FmRolodex::PORTAL_FIELDS` maps onto a person (profile fields, email type — a precedence word like "Preferred" records none, since the email's own slot already says that — facilitator start → `member_since`, service areas, `Settings`, `LifeExperiences` and `PrimaryAgeGroup` → sectors and age groups translated through `TagTranslation` (ADR-0007; primary fields → primary, the rest additional; one primary sector and one primary age group per person — each kind takes it from the field that owns it, and an answer naming several lets the other field answer instead) via `tag_sectors` / `tag_age_groups`, and the archive's own entry/modification dates as `created_at`/`updated_at`, plus `EnteredBy`/`ModifiedBy` as `created_by`/`updated_by` through `StaffTranslation`, falling back to `Setting.import_user` (picked on `/admin/settings` → Imports) when the name is blank, a shared login, or unresolved, so every person carries both stamps; a linked person keeps any stamps they already have; a name no `StaffAlias` answers to is reported in `#unmatched_staff`). Plus every address and phone the row answers to (`FmRolodex#addresses` / `#phone_numbers`, primary first; both tables are read once via `FmRolodex.contact_index` + `#preload_contacts`), except organization addresses (`FmRolodex#organization_address?`: the worksite one, any typed Work, and every address on an agency's card), which `AffiliationServices::ImportFromFilemaker` and `OrganizationServices::ImportFromFilemaker` place. Links an existing person through `PersonMatcher`, trying each address the row holds in `EMail` (`EmailPrimary` is a flag, never an address: `1` → the first address becomes `email`, a second one the person doesn't already hold `email_2`; `2` → the address and `EmailType` land in `email_2`/`email_2_type`; for a row with no address at all, a same-name person with no email) by adding the row's ID to their `filemaker_code` list — every match when several answer, and alongside any code they already carry — and fills them from the row: blank fields take the archive's value, missing addresses/phones/tags are added (an existing primary stays primary), and a field the archive disagrees on keeps the portal value and is noted as a `Comment` on the person and in `#differences`; creates a new person when nobody matches, even when the email belongs to a differently named person; an agency card naming a contact person becomes that person; skips soft-deleted and nameless rows (a nameless agency card is the agency itself, which the organizations pass reads). Reads raw values through `FilemakerValues` and addresses through `FmAddressAttributes`. A match that doesn't validate is reported under `failed` and left out, so the row's other matches still link; `relink: true` reads named rows past the idempotence guard to pick one up once the person is valid. Exposes `TagTranslation#unmatched` as `#unmatched_tags`. Idempotent on `filemaker_code`. Driven by `fm:import_people`, and by `fm:import_organizations` for agency cards only +- `PersonServices::ImportFromRolodex` — Creates people from FileMaker Rolodex archive rows (`FmRolodex`), carrying the row's ID as `filemaker_code` plus every field `FmRolodex::PORTAL_FIELDS` maps onto a person (profile fields, email type — a precedence word like "Preferred" records none, since the email's own slot already says that — facilitator start → `member_since`, service areas, `Settings`, `LifeExperiences` and `PrimaryAgeGroup` → sectors and age groups translated through `TagTranslation` (ADR-0007; primary fields → primary, the rest additional; one primary sector and one primary age group per person — each kind takes it from the field that owns it, and an answer naming several lets the other field answer instead) via `tag_sectors` / `tag_age_groups`, and the archive's own entry/modification dates as `created_at`/`updated_at`, plus `EnteredBy`/`ModifiedBy` as `created_by`/`updated_by` through `StaffTranslation`, falling back to `Setting.import_user` (picked on `/admin/settings` → Imports) when the name is blank, a shared login, or unresolved, so every person carries both stamps; a linked person keeps any stamps they already have; a name no `StaffAlias` answers to is reported in `#unmatched_staff`). Plus every address and phone the row answers to (`FmRolodex#addresses` / `#phone_numbers`, primary first; both tables are read once via `FmRolodex.contact_index` + `#preload_contacts`), except organization addresses (`FmRolodex#organization_address?`: the worksite one, any typed Work, and every address on an agency's card), which `AffiliationServices::ImportFromFilemaker` and `OrganizationServices::ImportFromFilemaker` place. Each address and phone carries its archive row's ID (`AddrsID` / `PhoneID`) as its own `filemaker_code`; a row the person already holds (same address or number) adds its code to that record instead of a copy. Links an existing person through `PersonMatcher`, trying each address the row holds in `EMail` (`EmailPrimary` is a flag, never an address: `1` → the first address becomes `email`, a second one the person doesn't already hold `email_2`; `2` → the address and `EmailType` land in `email_2`/`email_2_type`; for a row with no address at all, a same-name person with no email) by adding the row's ID to their `filemaker_code` list — every match when several answer, and alongside any code they already carry — and fills them from the row: blank fields take the archive's value, missing addresses/phones/tags are added (an existing primary stays primary), and a field the archive disagrees on keeps the portal value and is noted as a `Comment` on the person and in `#differences`; creates a new person when nobody matches, even when the email belongs to a differently named person; an agency card naming a contact person becomes that person; skips soft-deleted and nameless rows (a nameless agency card is the agency itself, which the organizations pass reads). Reads raw values through `FilemakerValues` and addresses through `FmAddressAttributes`. A match that doesn't validate is reported under `failed` and left out, so the row's other matches still link; `relink: true` reads named rows past the idempotence guard to pick one up once the person is valid. Exposes `TagTranslation#unmatched` as `#unmatched_tags`. Idempotent on `filemaker_code`. Driven by `fm:import_people`, and by `fm:import_organizations` for agency cards only - `PersonServices::UndoRolodexImport` — Reverses one FileMaker import run (`fm:import_people` or `fm:import_organizations`) from the Ahoy lifecycle events it tagged with its `import_run`: newest first, deletes each record the run created and reverts each field it filled, unless the record was edited since (its `updated_at` is newer than the event) or the field has changed — those are kept and reported. Runs under `no_touching` so deleting a person's address doesn't make the person look edited. Driven by `fm:undo_import_people` (alias `fm:undo_import`) - `PersonServices::SyncSharingPreferences` — Writes the two content-sharing answers (`anonymous_contributions`, `display_name_preference`) onto the `Person`, mapping each submitted option label back to its stored value; an unanswered or unrecognized answer leaves the profile alone, so a form that doesn't ask the questions is a no-op and every flow can call it unconditionally. The person-side counterpart to `OrganizationServices::SyncProfile`. Shared by `PersonServices::CaptureFromSubmission` (so a submission's answers land in the "what this submission changed" audit via the Ahoy trail) and `CalloutFormSubmission` (whose `#profile_changes` re-reports the same edits for its own Ahoy tracking) - `EventRegistrationServices::TransferContinuingEducation` — Splits/relocates a registrant's CE when they transfer events (issue #1944): a simple forward transfer leaves a paid, zero-hours **stub** on the source (its payments count at the original event) and creates a **live** record on the destination carrying the hours and the outstanding balance; when the reg being transferred out is itself a transfer-in (a collapsing double transfer, or a transfer back to the origin) its live record is relocated forward — merging back into the origin's stub — instead of split again, so no third record appears. Runs inside the transfer transaction, after the destination is saved and before a collapsing middle reg is destroyed @@ -357,7 +357,7 @@ comparison — add a rule if none fits (e.g. `PersonPolicy#own_profile?`). Conve ### Organizations -- `OrganizationServices::ImportFromFilemaker` — The FileMaker organizations pass (ADR-0008): each `FmOrganization` and its `FmProject`s become portal orgs — an org with exactly one project stays one org carrying both codes, otherwise the projects nest under it (`parent`). Matches by code (`FilemakerCodeIndex`), then exact name only to an org carrying no code from that table (a namesake gets `"Name (code)"`). Fills blanks only (website, description, mission, `organization_type` from `OrgType`/`AgencyType` via `ORGANIZATION_TYPE_PATTERNS` — any other word lands as Other with the word in `organization_type_other`, tallied in `#unmatched_types`), noting differences as comments; adds the org's `fm_addresses`, each phone number no address holds onto an address without one (noted when none is free), its `fm_email_addresses`; copies each project's `Locality`/`District`/`SPA`/`LASupervisorialDistrict` onto the program's address in its city (a city-only one when none); folds a merged-away project's code (`MergerWithID`) into the survivor; and reads agencies' own Rolodex cards (website, mission, addresses, and email when no contact person is named) onto their org, creating it by name when nothing answers. Driven by `fm:import_organizations` +- `OrganizationServices::ImportFromFilemaker` — The FileMaker organizations pass (ADR-0008): each `FmOrganization` and its `FmProject`s become portal orgs — an org with exactly one project stays one org carrying both codes, otherwise the projects nest under it (`parent`). Matches by code (`FilemakerCodeIndex`), then exact name only to an org carrying no code from that table (a namesake gets `"Name (code)"`). Fills blanks only (website, description, mission, `organization_type` from `OrgType`/`AgencyType` via `ORGANIZATION_TYPE_PATTERNS` — any other word lands as Other with the word in `organization_type_other`, tallied in `#unmatched_types`), noting differences as comments; adds the org's `fm_addresses` and `fm_phone_numbers` (phones as `contact_methods`, the archive's primary first), each carrying its row's ID as `filemaker_code` (an address or number the org already holds takes the code instead of a copy), and its `fm_email_addresses`; copies each project's `Locality`/`District`/`SPA`/`LASupervisorialDistrict` onto the program's address in its city (a city-only one when none); folds a merged-away project's code (`MergerWithID`) into the survivor; and reads agencies' own Rolodex cards (website, mission, addresses, and email when no contact person is named) onto their org, creating it by name when nothing answers. Driven by `fm:import_organizations` - `OrganizationServices::AutofillChange` — One thing a registrant's answers wrote onto an org: `field`, human `label`, the `value` that landed in it, `change_type` (`"new"` filled a blank / `"update"` replaced something), `previous_value` on an update, and an optional `scope` naming which work address. `change_type` is derived from `previous_value` so the two can't disagree, but is written to the JSON anyway so the stored row reads on its own. Round-trips through the `form_autofill_changes` column; `#description` renders "Website" or "ZIP on the Austin work address" for a flash, while the linking page shows the value and what it displaced. `#key` is (field, scope), which is what makes a later submission's value replace an earlier one instead of stacking. Drops a stored entry with no `field` rather than raising — it's a display aid, not a ledger. **Not a substitute for the Ahoy audit trail**, which records every before/after on Organization and Address independently; this is the denormalized, registration-scoped copy the linking page can render without correlating events - `OrganizationServices::AddressMatcher` — Given a submitted city/state/street/ZIP, finds the org address it corresponds to so `UpsertAddress`/`ProfileDiff` update the right one instead of duplicating. Tries same city + state (preferring one whose street also matches, so an org with several offices in one city doesn't get the wrong one rewritten), then same city + street (unifies an address with a submission that merely adds the state/country we were missing), then — across cities — same street + same ZIP, which reunites an address whose city was renamed ("Ventura" / "San Buenaventura") without merging two offices that share a street name in different towns. Every comparison goes through `AddressNormalizer`, so known spellings ("St. Louis" / "Saint Louis", "TX" / "Texas", "N Main St" / "North Main Street") match at every step. Module function, used by both address services - `OrganizationServices::UpsertAddress` — Find-or-create an organization's "work" address from a registrant's submitted organization fields (street/city/state/zip/country), matching via `AddressMatcher`; updates the match, else adds a new address; never demotes the org's existing primary (a registrant's address becomes primary only when the org has none yet). `overwrite: true` (default, public flow) replaces the matching address's details; `overwrite: false` (admin org-linking) fills only its blank fields so a conflicting saved address is kept and flagged instead. City and state are always fill-only (never flipped, in either mode), so a cross-city ZIP match keeps the saved city and lets `ProfileDiff` flag the difference; that fill still earns its keep because a legacy row can hold `""` in either column, and the street + ZIP match is the one path that reaches such a row to repair it. Returns a `Result` (`address`, `created`, and `changes` — an `AutofillChange` per field actually written, carrying the value) so callers can report what was really saved. Every field is trimmed before it's stored, so a padded answer isn't later read back as a difference from itself. Tolerates optional answers: a skipped street/ZIP is stored blank (both are `NOT NULL` columns), and `address` is nil — nothing saved — when there's no city, or no state and no existing address to update, rather than failing the whole registration. Shared by `PublicRegistration` and the admin org-linking actions diff --git a/app/controllers/fm_archives_controller.rb b/app/controllers/fm_archives_controller.rb index 901e525cc5..52eb42073f 100644 --- a/app/controllers/fm_archives_controller.rb +++ b/app/controllers/fm_archives_controller.rb @@ -63,7 +63,7 @@ class FmArchivesController < ApplicationController { label: "Other", icon: "📎", tables: %w[notes] } ] - PORTAL_MODELS = [ Person, Organization, Affiliation, Payment, Workshop ].freeze + PORTAL_MODELS = [ Person, Organization, Affiliation, Payment, Workshop, Address, ContactMethod ].freeze def index authorize! :fm_archive diff --git a/app/controllers/organizations_controller.rb b/app/controllers/organizations_controller.rb index 760f51e7b4..25a42ee9f0 100644 --- a/app/controllers/organizations_controller.rb +++ b/app/controllers/organizations_controller.rb @@ -207,7 +207,7 @@ def populations_served def set_organization @organization = Organization.includes( - :organization_status, :windows_type, :addresses, + :organization_status, :windows_type, :addresses, :contact_methods, :categorizable_items, { comments: [ :created_by, :updated_by ] }, affiliations: { person: [ :user, :affiliations, *Organization::PEOPLE_TAGGINGS ] } @@ -238,6 +238,7 @@ def organization_params ], comments_attributes: [ :id, :topic, :body, :flagged, :_destroy ], notifications_attributes: Notification::PERMITTED_LOG_ATTRIBUTES, + contact_methods_attributes: [ :id, :kind, :contact_type, :value, :primary, :inactive, :_destroy ], addresses_attributes: [ :id, :address_type, @@ -275,9 +276,11 @@ def dedupe_config merge_keeper: ->(keep, delete) { keep.filemaker_code = Organization.join_filemaker_codes(keep.filemaker_code, delete.filemaker_code) @keeper_primary_address_signature = AddressDeduper.primary_signature(keep, delete) + @keeper_primary_contact_signatures = ContactMethodDeduper.primary_signatures(keep, delete) }, after_merge: ->(keep) { AddressDeduper.new(keep, primary_signature: @keeper_primary_address_signature).call + ContactMethodDeduper.new(keep, primary_signatures: @keeper_primary_contact_signatures).call }, belongs_to_options: -> { { diff --git a/app/controllers/people_controller.rb b/app/controllers/people_controller.rb index 5f9ef75eee..83d6e0c417 100644 --- a/app/controllers/people_controller.rb +++ b/app/controllers/people_controller.rb @@ -429,7 +429,7 @@ def dedupe_config delete.age_range_categorizable_items.find_by(is_primary: true)&.category_id @keeper_default_pay_customer_id = keep.pay_customers.active.find_by(default: true)&.id @keeper_primary_address_signature = AddressDeduper.primary_signature(keep, delete) - @keeper_primary_contact_signatures = PersonServices::ReconcileContactMethods.primary_signatures(keep, delete) + @keeper_primary_contact_signatures = ContactMethodDeduper.primary_signatures(keep, delete) @primary_login_id = primary_login(keep, delete)&.id }, # Reconcile what the generic merge leaves inconsistent on the kept person: @@ -456,7 +456,7 @@ def dedupe_config preferred_customer_id: @keeper_default_pay_customer_id).call AddressDeduper.new(keep, primary_signature: @keeper_primary_address_signature).call - PersonServices::ReconcileContactMethods.new(keep, + ContactMethodDeduper.new(keep, primary_signatures: @keeper_primary_contact_signatures).call sync_primary_login_email(keep) }, diff --git a/app/helpers/filemaker_helper.rb b/app/helpers/filemaker_helper.rb index d153b30c5a..2b2baaf7ee 100644 --- a/app/helpers/filemaker_helper.rb +++ b/app/helpers/filemaker_helper.rb @@ -44,6 +44,8 @@ def filemaker_portal_link(record) when Affiliation then [ "#{record.person&.full_name} at #{record.organization&.name}", edit_affiliation_path(record) ] when Payment then [ [ dollars_from_cents(record.amount_cents), person_name(record.payer) ].compact.join(" from "), payment_path(record) ] when Workshop then [ record.title, workshop_path(record) ] + when Address then [ "#{record.addressable.name}: #{record.street_address}", polymorphic_path(record.addressable) ] + when ContactMethod then [ "#{record.contactable.name}: #{record.value}", polymorphic_path(record.contactable) ] end link_to text, path, class: button_classes(:primary) diff --git a/app/models/address.rb b/app/models/address.rb index c8304c562b..19cdf984b5 100644 --- a/app/models/address.rb +++ b/app/models/address.rb @@ -1,5 +1,7 @@ class Address < ApplicationRecord + include FilemakerCoded include OutsideUsaTypeable + filemaker_archived_in FmAddress belongs_to :created_by, class_name: "User", optional: true belongs_to :updated_by, class_name: "User", optional: true diff --git a/app/models/contact_method.rb b/app/models/contact_method.rb index f3ed4e0dc4..420e421a1e 100644 --- a/app/models/contact_method.rb +++ b/app/models/contact_method.rb @@ -1,4 +1,7 @@ class ContactMethod < ApplicationRecord + include FilemakerCoded + filemaker_archived_in FmPhoneNumber + UNKNOWN_CONTACT_TYPE = "unknown" CONTACT_TYPES = [ nil, "work", "personal", UNKNOWN_CONTACT_TYPE ].freeze diff --git a/app/models/organization.rb b/app/models/organization.rb index e00f5281a8..08a159b04e 100644 --- a/app/models/organization.rb +++ b/app/models/organization.rb @@ -16,6 +16,7 @@ class Organization < ApplicationRecord has_many :children, class_name: "Organization", foreign_key: :parent_id, inverse_of: :parent, dependent: :nullify has_many :addresses, as: :addressable, dependent: :destroy has_many :active_addresses, -> { active }, as: :addressable, class_name: "Address", inverse_of: :addressable + has_many :contact_methods, as: :contactable, dependent: :destroy has_many :bookmarks, as: :bookmarkable, dependent: :destroy has_many :other_responses, as: :owner, dependent: :destroy has_many :invoices, as: :invoicee, dependent: :restrict_with_error @@ -26,6 +27,12 @@ class Organization < ApplicationRecord has_many :users, through: :people has_many :comments, -> { newest_first }, as: :commentable, dependent: :destroy + # Its primary active phone, else any active one, else a phone an active address holds. + def phone_number + phones = contact_methods.select { |method| method.phone? && !method.inactive? } + (phones.find(&:primary?) || phones.first)&.value || addresses.reject(&:inactive?).filter_map(&:phone).first + end + # An organization is its own correspondent — there is no person behind it. def communications_email email @@ -77,6 +84,7 @@ def communications_email # Nested attributes accepts_nested_attributes_for :addresses, allow_destroy: true, reject_if: proc { |attrs| Address.unfilled_attributes?(attrs) } + accepts_nested_attributes_for :contact_methods, allow_destroy: true, reject_if: :all_blank accepts_nested_attributes_for :affiliations, allow_destroy: true, reject_if: proc { |attrs| attrs["person_id"].blank? } accepts_nested_attributes_for :comments, allow_destroy: true, reject_if: proc { |attrs| attrs["body"].blank? } diff --git a/app/services/address_deduper.rb b/app/services/address_deduper.rb index 309041019d..ba7409081c 100644 --- a/app/services/address_deduper.rb +++ b/app/services/address_deduper.rb @@ -8,6 +8,7 @@ class AddressDeduper # Filled in on the survivor from a merged copy when the survivor has none, so a # merge never drops a phone number, a ZIP, or a geocoded district only one copy had. + # FileMaker codes are unioned. CARRIED_COLUMNS = %w[ street_address zip_code country county district address_type locality phone la_city_council_district la_service_planning_area la_supervisorial_district @@ -86,9 +87,9 @@ def merge(loser, into:) def carry_over(loser, survivor) filled = CARRIED_COLUMNS.select { |column| missing?(survivor, column) && !missing?(loser, column) } - return if filled.empty? - - survivor.update!(filled.index_with { |column| loser.public_send(column) }) + survivor.assign_attributes(filled.index_with { |column| loser.public_send(column) }) + survivor.filemaker_code = Address.join_filemaker_codes(survivor.filemaker_code, loser.filemaker_code) + survivor.save! if survivor.changed? end def missing?(address, column) diff --git a/app/services/affiliation_services/import_from_filemaker.rb b/app/services/affiliation_services/import_from_filemaker.rb index cb55e2f28a..fe2a4a4b98 100644 --- a/app/services/affiliation_services/import_from_filemaker.rb +++ b/app/services/affiliation_services/import_from_filemaker.rb @@ -350,14 +350,14 @@ def updated_by(data) # person as their own work address. def place_work_addresses(rolodex, people, organization, work, label) work.sort_by { |address| address.fm_id == rolodex.data["WorksiteAddrsID"] ? 0 : 1 }.each do |fm_address| - attributes = @address_attributes.call(fm_address.data, address_type: "work") + attributes = @address_attributes.call(fm_address.data, address_type: "work")&.merge(filemaker_code: fm_address.fm_id) next unless attributes people.each do |person| target = organization_for(fm_address.data["OrgID"]) || organization || sole_organization(person) next keep_with_person(person, attributes, label) unless target - address = organization_address(target, attributes) + address = organization_address(target, attributes, label) next failed(address, label) unless address.persisted? person.affiliations.where(organization: target, organization_address_id: nil).find_each do |affiliation| @@ -372,15 +372,17 @@ def sole_organization(person) Organization.find_by(id: ids.first) if ids.one? end - def organization_address(organization, attributes) - held = organization.addresses.find { |address| address_key(address.street_address, address.city) == address_key(attributes[:street_address], attributes[:city]) } - return held if held + def organization_address(organization, attributes, label) + held = held_address(organization, attributes) + return organization.addresses.create(attributes.merge(primary: organization.addresses.where(primary: true).none?)) unless held - organization.addresses.create(attributes.merge(primary: organization.addresses.where(primary: true).none?)) + add_code(held, attributes[:filemaker_code], label) + held end def keep_with_person(person, attributes, label) - return if person.addresses.any? { |address| address_key(address.street_address, address.city) == address_key(attributes[:street_address], attributes[:city]) } + held = held_address(person, attributes) + return add_code(held, attributes[:filemaker_code], label) if held address = person.addresses.create(attributes.merge(primary: person.addresses.where(primary: true).none?)) return failed(address, label) unless address.persisted? @@ -388,6 +390,16 @@ def keep_with_person(person, attributes, label) record(:created, "#{label} → work address ##{address.id} kept on person ##{person.id}, no organization to place it on") end + def held_address(addressable, attributes) + key = address_key(attributes[:street_address], attributes[:city]) + addressable.addresses.find { |address| address_key(address.street_address, address.city) == key } + end + + # An address already on file takes the archive row's code rather than a copy. + def add_code(address, code, label) + address.update(filemaker_code: Address.join_filemaker_codes(address.filemaker_code, code)) || failed(address, label) + end + def address_key(street, city) [ AddressNormalizer.street(street), AddressNormalizer.city(city) ] end diff --git a/app/services/contact_method_deduper.rb b/app/services/contact_method_deduper.rb new file mode 100644 index 0000000000..f7f928709e --- /dev/null +++ b/app/services/contact_method_deduper.rb @@ -0,0 +1,91 @@ +# After a person or organization merge, the generic merge moves both records' contact +# methods onto the keeper wholesale (contact_methods have no unique index), leaving +# identical duplicates and possibly two primary phones of the same kind. Merge contact +# methods that share the same kind + value into one, then settle a single primary per +# kind: the keeper's own pre-merge primary, or the deleted record's when the keeper had +# none. Runs in the people and organizations dedupers' after_merge. +class ContactMethodDeduper + IDENTITY_COLUMNS = %w[kind value contact_type].freeze + + # Outside the identity, so a merge never drops the address link only one copy had. + # FileMaker codes are unioned. + CARRIED_COLUMNS = %w[address_id].freeze + + def self.signature(contact_method) + return unless contact_method + + IDENTITY_COLUMNS.map do |column| + value = contact_method.public_send(column).to_s.strip.downcase + # A row written before an untyped number said so holds a blank, so unknown and + # blank have to read alike or an otherwise identical pair won't fold. + value == ContactMethod::UNKNOWN_CONTACT_TYPE && column == "contact_type" ? "" : value + end + end + + # One preserved-primary signature per kind, read before the merge while the two + # records' contact methods can still be told apart: the keeper's own primary of + # that kind, else the deleted record's. + def self.primary_signatures(keep, delete) + methods = keep.contact_methods.to_a + delete.contact_methods.to_a + methods.map(&:kind).uniq.filter_map do |kind| + primary = methods.find { |method| method.kind == kind && method.primary? && method.contactable == keep } || + methods.find { |method| method.kind == kind && method.primary? } + signature(primary) + end + end + + def initialize(contactable, primary_signatures: []) + @contactable = contactable + @primary_signatures = primary_signatures + end + + def call + survivors = {} + merge_candidates.each do |contact_method| + signature = self.class.signature(contact_method) + if (survivor = survivors[signature]) + merge(contact_method, into: survivor) + else + survivors[signature] = contact_method + end + end + + settle_primary(survivors.values) + end + + private + + # Active before inactive, then the primary, then oldest — so a merge keeps the + # copy still in use. `#phone_number` reads only active phones, so letting an + # inactive duplicate win would leave the record looking phoneless. + def merge_candidates + @contactable.contact_methods.reload.sort_by do |contact_method| + [ contact_method.inactive? ? 1 : 0, contact_method.primary? ? 0 : 1, contact_method.id ] + end + end + + def merge(loser, into:) + carry_over(loser, into) + deduper.merge(into, loser) + end + + def carry_over(loser, survivor) + filled = CARRIED_COLUMNS.select do |column| + survivor.public_send(column).blank? && loser.public_send(column).present? + end + survivor.assign_attributes(filled.index_with { |column| loser.public_send(column) }) + survivor.filemaker_code = ContactMethod.join_filemaker_codes(survivor.filemaker_code, loser.filemaker_code) + survivor.save! if survivor.changed? + end + + def settle_primary(contact_methods) + contact_methods.each do |contact_method| + primary = @primary_signatures.include?(self.class.signature(contact_method)) + contact_method.update!(primary: primary) if contact_method.primary? != primary + end + end + + def deduper + @deduper ||= ModelDeduper.new(model_class: ContactMethod, logger: Rails.logger, dry_run: false) + end +end diff --git a/app/services/organization_services/import_from_filemaker.rb b/app/services/organization_services/import_from_filemaker.rb index e18d7e876e..7a0d81a523 100644 --- a/app/services/organization_services/import_from_filemaker.rb +++ b/app/services/organization_services/import_from_filemaker.rb @@ -19,8 +19,9 @@ module OrganizationServices # # Fields fill blanks only. Where the portal already holds a different value it keeps its # own, noted in #differences and as a comment on the organization. Also carried: - # * the organization's addresses (fm_addresses by OrgID), its phone onto its main - # address, and its email (fm_email_addresses) + # * the organization's addresses (fm_addresses by OrgID), its phones (fm_phone_numbers), + # and its email (fm_email_addresses) — each address and phone carrying its row's ID as + # its own filemaker_code # * each project's City/State with its Locality, District, SPA and supervisorial # district, onto the program's address in that city # * a merged-away project's code, onto the program it merged into @@ -162,9 +163,9 @@ def save(organization, row, label, attributes, also: nil) organization.filemaker_code = Organization.join_filemaker_codes(organization.filemaker_code, rows.map(&:fm_id)) kept = fill(organization, attributes) stamp(organization, row.data) - add_addresses(organization, @addresses.fetch(row.fm_id, []).map(&:data)) + add_addresses(organization, @addresses.fetch(row.fm_id, [])) rows.select { |source| source.is_a?(FmProject) }.each { |project| place_project(organization, project.data) } - kept += add_phones(organization, @phones.fetch(row.fm_id, []).map(&:data)) + add_phones(organization, @phones.fetch(row.fm_id, [])) fill_email(organization, @emails.fetch(row.fm_id, []).map { |email| email.data["EmailAddress"] }) return failed(organization, label) unless organization.save @@ -224,8 +225,8 @@ def import_agency(rolodex) organization ||= Organization.new(name:) kept = fill(organization, website_url: clip(data["WebSite"]), mission_vision_values: clip(data["Mission"])) stamp(organization, data) - add_addresses(organization, rolodex.addresses.map(&:data)) - kept += add_phones(organization, rolodex.phone_numbers.map(&:data)) + add_addresses(organization, rolodex.addresses) + add_phones(organization, rolodex.phone_numbers) fill_email(organization, [ data["EMail"] ]) unless names_a_person?(data) outcome = agency_outcome(organization) return failed(organization, label) unless organization.save @@ -341,15 +342,18 @@ def stamp(organization, data) organization.updated_by_id ||= @staff.user_id_for(data["ModifiedBy"]) || @import_user_id end - # An organization gathers addresses from every source, so it keeps the ones it has and - # takes the archive's others; the first becomes its primary when it has none. - def add_addresses(organization, rows) - rows.each do |fields| - attributes = @address_attributes.call(fields, address_type: "work") - next if attributes.nil? || held_address(organization, attributes) + # An organization gathers addresses from every source, so one it already has takes the + # archive row's code and the others are added; the first becomes its primary when it has none. + def add_addresses(organization, fm_addresses) + fm_addresses.each do |fm_address| + attributes = @address_attributes.call(fm_address.data, address_type: "work") + next if attributes.nil? + + held = held_address(organization, attributes) + next held.filemaker_code = Address.join_filemaker_codes(held.filemaker_code, fm_address.fm_id) if held primary = organization.addresses.none? { |address| address.primary? && !address.marked_for_destruction? } - organization.addresses.build(attributes.merge(primary:)) + organization.addresses.build(attributes.merge(primary:, filemaker_code: fm_address.fm_id)) end end @@ -403,22 +407,21 @@ def district_number(value) value.to_s[/\d+/]&.to_i end - # The portal keeps an organization's phone on its addresses, so each number the archive - # has that no address holds yet goes onto one without a phone, the primary first. One - # with nowhere left to go is returned as a difference to note. - def add_phones(organization, rows) - held = organization.addresses.filter_map { |address| phone_key(address.phone) } - numbers = rows.sort_by { |fields| flag?(fields["Primacy"]) ? 0 : 1 } - .filter_map { |fields| clip(fields["PhoneNumber"].to_s.strip) } - .uniq { |number| phone_key(number) }.reject { |number| held.include?(phone_key(number)) } - numbers.filter_map do |number| - address = organization.addresses.sort_by { |held_address| held_address.primary? ? 0 : 1 }.find { |held_address| held_address.phone.blank? } - if address - address.phone = number - next - end + # Each number lands on a phone record carrying its archive row's code — the one the + # organization already holds for that number, else a new one, the archive's primary first. + def add_phones(organization, fm_phones) + fm_phones.sort_by { |phone| flag?(phone.data["Primacy"]) ? 0 : 1 }.each do |phone| + number = clip(phone.data["PhoneNumber"].to_s.strip) + next if number.blank? + + held = organization.contact_methods.find { |method| phone_key(method.value) == phone_key(number) } + next held.filemaker_code = ContactMethod.join_filemaker_codes(held.filemaker_code, phone.fm_id) if held - [ "Phone", organization.addresses.filter_map(&:phone).to_sentence.presence || "no address to hold it", number ] + organization.contact_methods.build( + kind: "phone", value: number, filemaker_code: phone.fm_id, + contact_type: contact_type(phone.data["PhoneType"], allowed: ContactMethod::CONTACT_TYPES), + primary: organization.contact_methods.none?(&:primary?) + ) end end diff --git a/app/services/person_services/import_from_rolodex.rb b/app/services/person_services/import_from_rolodex.rb index c900f38daf..6d8d1ddf3c 100644 --- a/app/services/person_services/import_from_rolodex.rb +++ b/app/services/person_services/import_from_rolodex.rb @@ -19,6 +19,10 @@ module PersonServices # person is the agency itself, which the organizations import reads) # * failed — the Person didn't validate # + # Each address and phone the row brings carries its archive row's ID as its own + # filemaker_code, added to the one the person already holds when it's the same place + # or number. + # # An email field holding no address the person's email validation accepts ("ada@@x.org", # "ada at x dot org") or an address it rejects is left off — the row imports without it. # When that leaves the person with no email on file, the field's whole value is left as a @@ -387,26 +391,33 @@ def names(model, ids) # Every address the row answers to, not just the one it names as primary — # FmRolodex#addresses lists the primary first, so the first that can be built is the # person's primary and the rest ride along. An organization's address (a work address, - # or any address on an agency's row), one the person already has, and a primary they - # already hold are all left as they are; the organizations import places the first. + # or any address on an agency's row) and a primary they already hold are left as they + # are; the organizations import places the first. def build_addresses(person, rolodex) - held = person.addresses.map { |address| address_key(address.street_address, address.city) } attributes = rolodex.addresses.reject { |address| rolodex.organization_address?(address) } - .filter_map { |address| @address_attributes.call(address.data) } - .reject { |attrs| held.include?(address_key(attrs[:street_address], attrs[:city])) } - build_primary_first(person.addresses, attributes) + .filter_map { |address| @address_attributes.call(address.data)&.merge(filemaker_code: address.fm_id) } + attach(person.addresses, attributes) { |address| address_key(address[:street_address], address[:city]) } end def build_phones(person, rolodex) - held = person.contact_methods.map { |phone| phone_key(phone.value) } - attributes = rolodex.phone_numbers.filter_map { |phone| phone_attributes(phone.data) } - .reject { |attrs| held.include?(phone_key(attrs[:value])) } - build_primary_first(person.contact_methods, attributes) + attributes = rolodex.phone_numbers.filter_map { |phone| phone_attributes(phone) } + attach(person.contact_methods, attributes) { |phone| phone_key(phone[:value]) } end - def build_primary_first(association, attributes) + # Each archive row lands on one record carrying its code: the one the person already + # holds for it (the same address or number), or a new one — the first of which is + # primary when the person has none. + def attach(association, attributes) has_primary = association.any?(&:primary?) - attributes.each_with_index { |attrs, index| association.build(attrs.merge(primary: index.zero? && !has_primary)) } + attributes.each do |attrs| + held = association.find { |record| yield(record) == yield(attrs) } + if held + held.filemaker_code = held.class.join_filemaker_codes(held.filemaker_code, attrs[:filemaker_code]) + else + association.build(attrs.merge(primary: !has_primary)) + has_primary = true + end + end end def address_key(street, city) @@ -417,11 +428,12 @@ def phone_key(value) value.to_s.gsub(/\D/, "") end - def phone_attributes(fields) - number = clip(fields["PhoneNumber"]) + def phone_attributes(phone) + number = clip(phone.data["PhoneNumber"]) return if number.blank? - { kind: "phone", value: number, contact_type: contact_type(fields["PhoneType"], allowed: ContactMethod::CONTACT_TYPES) } + { kind: "phone", value: number, contact_type: contact_type(phone.data["PhoneType"], allowed: ContactMethod::CONTACT_TYPES), + filemaker_code: phone.fm_id } end def ids(people) diff --git a/app/services/person_services/reconcile_contact_methods.rb b/app/services/person_services/reconcile_contact_methods.rb deleted file mode 100644 index fb53a97cd1..0000000000 --- a/app/services/person_services/reconcile_contact_methods.rb +++ /dev/null @@ -1,92 +0,0 @@ -module PersonServices - # After a person merge, the generic merge moves both people's contact methods onto - # the keeper wholesale (contact_methods have no unique index), leaving identical - # duplicates and possibly two primary phones of the same kind. Merge contact methods - # that share the same kind + value into one, then settle a single primary per kind: - # the keeper's own pre-merge primary, or the deleted person's when the keeper had - # none. Runs in the people deduper's after_merge. - class ReconcileContactMethods - IDENTITY_COLUMNS = %w[kind value contact_type].freeze - - # Outside the identity, so a merge never drops the address link only one copy had. - CARRIED_COLUMNS = %w[address_id].freeze - - def self.signature(contact_method) - return unless contact_method - - IDENTITY_COLUMNS.map do |column| - value = contact_method.public_send(column).to_s.strip.downcase - # A row written before an untyped number said so holds a blank, so unknown and - # blank have to read alike or an otherwise identical pair won't fold. - value == ContactMethod::UNKNOWN_CONTACT_TYPE && column == "contact_type" ? "" : value - end - end - - # One preserved-primary signature per kind, read before the merge while the two - # people's contact methods can still be told apart: the keeper's own primary of - # that kind, else the deleted person's. - def self.primary_signatures(keep, delete) - methods = keep.contact_methods.to_a + delete.contact_methods.to_a - methods.map(&:kind).uniq.filter_map do |kind| - primary = methods.find { |method| method.kind == kind && method.primary? && method.contactable == keep } || - methods.find { |method| method.kind == kind && method.primary? } - signature(primary) - end - end - - def initialize(person, primary_signatures: []) - @person = person - @primary_signatures = primary_signatures - end - - def call - survivors = {} - merge_candidates.each do |contact_method| - signature = self.class.signature(contact_method) - if (survivor = survivors[signature]) - merge(contact_method, into: survivor) - else - survivors[signature] = contact_method - end - end - - settle_primary(survivors.values) - end - - private - - # Active before inactive, then the primary, then oldest — so a merge keeps the - # copy still in use. `Person#phone_number` reads only active phones, so letting - # an inactive duplicate win would leave the person looking phoneless. - def merge_candidates - @person.contact_methods.reload.sort_by do |contact_method| - [ contact_method.inactive? ? 1 : 0, contact_method.primary? ? 0 : 1, contact_method.id ] - end - end - - def merge(loser, into:) - carry_over(loser, into) - deduper.merge(into, loser) - end - - def carry_over(loser, survivor) - filled = CARRIED_COLUMNS.select do |column| - survivor.public_send(column).blank? && loser.public_send(column).present? - end - return if filled.empty? - - survivor.update!(filled.index_with { |column| loser.public_send(column) }) - end - - def settle_primary(contact_methods) - contact_methods.each do |contact_method| - primary = @primary_signatures.include?(self.class.signature(contact_method)) - contact_method.update!(primary: primary) if contact_method.primary? != primary - end - end - - def deduper - @deduper ||= ModelDeduper.new(model_class: ContactMethod, logger: Rails.logger, dry_run: false) - end - end -end diff --git a/app/views/organizations/_form.html.erb b/app/views/organizations/_form.html.erb index 7c415ba3b4..b489d72d6b 100644 --- a/app/views/organizations/_form.html.erb +++ b/app/views/organizations/_form.html.erb @@ -241,6 +241,20 @@ class: button_classes(:secondary_outline) %> +
    +

    Phones

    + <%= f.simple_fields_for :contact_methods do |sfi| %> + <%= render "contact_methods/contact_method_fields", f: sfi %> + <% end %> + + <%= link_to_add_association "➕ Add phone", + f, + :contact_methods, + partial: "contact_methods/contact_method_fields", + data: { association_insertion_node: "this" }, + class: button_classes(:secondary_outline) %> +
    +

    Affiliations

    diff --git a/app/views/organizations/show.html.erb b/app/views/organizations/show.html.erb index fe45c2e5b9..cac2beac2f 100644 --- a/app/views/organizations/show.html.erb +++ b/app/views/organizations/show.html.erb @@ -86,11 +86,7 @@ <% end %> <% if @organization.profile_show_phone? %> - <% phone = if @organization.addresses.loaded? - @organization.addresses.reject(&:inactive?).filter_map(&:phone).first - else - @organization.addresses.active.filter_map(&:phone).first - end %> + <% phone = @organization.phone_number %> <% if phone.present? %> <%= link_to phone, tel_href(phone), class: "hover:text-primary" %> diff --git a/db/migrate/20261025200004_add_filemaker_code_to_addresses_and_contact_methods.rb b/db/migrate/20261025200004_add_filemaker_code_to_addresses_and_contact_methods.rb new file mode 100644 index 0000000000..6cf9ddd2f1 --- /dev/null +++ b/db/migrate/20261025200004_add_filemaker_code_to_addresses_and_contact_methods.rb @@ -0,0 +1,11 @@ +class AddFilemakerCodeToAddressesAndContactMethods < ActiveRecord::Migration[8.1] + def up + add_column :addresses, :filemaker_code, :string + add_column :contact_methods, :filemaker_code, :string + end + + def down + remove_column :contact_methods, :filemaker_code, if_exists: true + remove_column :addresses, :filemaker_code, if_exists: true + end +end diff --git a/db/schema.rb b/db/schema.rb index 7a60730000..59fa4cf573 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -10,7 +10,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema[8.1].define(version: 2026_10_25_125134) do +ActiveRecord::Schema[8.1].define(version: 2026_10_25_200004) do create_table "action_text_mentions", charset: "utf8mb4", collation: "utf8mb4_unicode_ci", force: :cascade do |t| t.bigint "action_text_rich_text_id", null: false t.datetime "created_at", null: false @@ -89,6 +89,7 @@ t.integer "updated_by_id" t.string "zip_code", null: false t.boolean "invoice_address" + t.string "filemaker_code" t.index ["addressable_type", "addressable_id", "invoice_address"], name: "index_addresses_on_addressable_and_invoice_role" t.index ["addressable_type", "addressable_id"], name: "index_addresses_on_addressable" t.index ["created_by_id"], name: "index_addresses_on_created_by_id" @@ -459,6 +460,7 @@ t.datetime "updated_at", null: false t.integer "updated_by_id" t.string "value", null: false + t.string "filemaker_code" t.index ["address_id"], name: "index_contact_methods_on_address_id" t.index ["contactable_type", "contactable_id"], name: "index_contact_methods_on_contactable" t.index ["created_by_id"], name: "index_contact_methods_on_created_by_id" diff --git a/db/seeds/features/2026-10-25-filemaker-codes-on-addresses-and-phones.yml b/db/seeds/features/2026-10-25-filemaker-codes-on-addresses-and-phones.yml new file mode 100644 index 0000000000..e1bff6abbb --- /dev/null +++ b/db/seeds/features/2026-10-25-filemaker-codes-on-addresses-and-phones.yml @@ -0,0 +1,13 @@ +name: "FileMaker codes on imported addresses and phones" +area: people +display_status: admin_facing +released_on: 2026-10-25 +pr_number: 2860 +summary: >- + Addresses and phone numbers brought in from FileMaker — for people and + organizations — now carry the FileMaker record they came from, so each one + can be traced back to the archive. +pro_tips: + - "An address or phone's page in the FileMaker archive links back to the person or organization holding it." + - "When FileMaker has an address or number the record already had, the existing one takes the FileMaker code rather than getting a copy." + - "Merging duplicate addresses or phones keeps every FileMaker code they carried." diff --git a/db/seeds/features/2026-10-25-organization-phones.yml b/db/seeds/features/2026-10-25-organization-phones.yml new file mode 100644 index 0000000000..41c1996e56 --- /dev/null +++ b/db/seeds/features/2026-10-25-organization-phones.yml @@ -0,0 +1,14 @@ +name: "Phone numbers on organizations" +area: people +display_status: admin_facing +released_on: 2026-10-25 +pr_number: 2860 +summary: >- + Organizations now keep their own list of phone numbers, edited the same way + as a person's, with one marked primary. The FileMaker import brings an + organization's numbers in there. +pro_tips: + - "Add phones in the Phones section of the organization's edit page." + - "The profile shows the primary phone, or a phone saved on one of its addresses when it has none." + - "Merging two organizations keeps both sets of phones, folding a shared number into one." +action_path: "/organizations" diff --git a/docs/adr/0005-filemaker-code-archive-mapping.md b/docs/adr/0005-filemaker-code-archive-mapping.md index a7d8baa687..ea7c096496 100644 --- a/docs/adr/0005-filemaker-code-archive-mapping.md +++ b/docs/adr/0005-filemaker-code-archive-mapping.md @@ -10,7 +10,7 @@ source of truth across the board: staff edited records in both FileMaker and the portal, so for some data the portal is right and for some FileMaker is. The raw export lives in the read-only `fm_*` archive tables (one per FileMaker table, keyed by `fm_id`, every other field in a `data` JSON column; see `lib/tasks/fm_import.rake`). -Six app tables carry a `filemaker_code` column, read as a comma-separated set of +Eight app tables carry a `filemaker_code` column, read as a comma-separated set of codes (`FilemakerCoded`). The code is an archive `fm_id`, but nothing recorded *which* archive table each column's codes belong to. The dev seed (`db/seeds/dev/fm_archive.rb`) had to choose, so this records the choice. @@ -29,6 +29,8 @@ drive the archive pages' links. | `payments.filemaker_code` | `fm_payments` | `RecordID`, e.g. `D29186` | `filemaker_code_hint` on payment edit | | `workshops.filemaker_code` | `fm_workshops` | `WorkshopID` | `filemaker_code_hint` on workshop form | | `resources.filemaker_code` | none | `RecordID` of the source quotation, e.g. `Q10215` | resource form (editable, no archive link) | +| `addresses.filemaker_code` | `fm_addresses` | `AddrsID`, e.g. `1234` | `fm_archives#show` links back to the address's person or org | +| `contact_methods.filemaker_code` | `fm_phone_numbers` | `PhoneID`, e.g. `5678` | `fm_archives#show` links back to the phone's person or org | - **Organization → Project, or → Organization.** An app `Organization` is usually one art program site, which is a FileMaker *Project*. FileMaker's own Organization row is @@ -64,7 +66,7 @@ when a new export lands. | Archive table | FileMaker key | Id pattern | Range in prod | Rows | Portal column | |---|---|---|---|---|---| | `fm_activities` | `ActivityLogID` | 5 digits | `00011`–`30061` | 29,185 | | -| `fm_addresses` | `AddrsID` | 1–5 digits, unpadded | `1`–`30808` | 28,556 | | +| `fm_addresses` | `AddrsID` | 1–5 digits, unpadded | `1`–`30808` | 28,556 | `addresses` | | `fm_allocations` | `AllocRecID` | 4 digits | `0001`–`3508` | 3,098 | | | `fm_ancillary_data` | `RecordID` | 1–3 digits, unpadded | `1`–`559` | 556 | | | `fm_art_purchases` | `PurchaseID` | 4 digits | `0001`–`0226` | 226 | | @@ -95,7 +97,7 @@ when a new export lands. | `fm_payments` | `RecordID` | `D` + 5 digits | `D00001`–`D29263` | 27,220 | `payments` | | `fm_people_to_organizations` | `RecordID` | 1–4 digits, unpadded | `1`–`1128` | 1,128 | | | `fm_personnels` | `PrsnlRecID` | `PS` + 4–5 digits | `PS0028`–`PS10667` | 9,037 | `affiliations` | -| `fm_phone_numbers` | `PhoneID` | 2–5 digits, unpadded | `22`–`35946` | 34,040 | | +| `fm_phone_numbers` | `PhoneID` | 2–5 digits, unpadded | `22`–`35946` | 34,040 | `contact_methods` | | `fm_postal_codes` | `Zipcode` | 5-digit ZIP | `00501`–`99950` | 41,685 | | | `fm_program_sponsorships` | `RecordID` | `PSP` + 4 digits | `PSP0035`–`PSP0901` | 608 | | | `fm_projects` | `ProjectID` | `P` + 4 digits | `P0001`–`P2806` | 2,007 | `organizations` | diff --git a/docs/adr/0008-filemaker-organization-and-personnel-import.md b/docs/adr/0008-filemaker-organization-and-personnel-import.md index c58af6a93a..5580d2d3ef 100644 --- a/docs/adr/0008-filemaker-organization-and-personnel-import.md +++ b/docs/adr/0008-filemaker-organization-and-personnel-import.md @@ -136,8 +136,9 @@ Constraints already decided elsewhere: - **Conflicts:** a field on a record — an organization's website, mission, type — fills blanks and keeps the portal value where the archive disagrees, reported as a comment plus in the task output. A record itself comes in new instead: an affiliation whose dates differ, an - address the organization doesn't hold, and a phone number no address holds — placed on an - address without one, and noted only when none is free (decided 2026-10-25). + address the organization doesn't hold, and a phone number it doesn't hold — a phone + record (`contact_methods`) on the organization, as a person's are (decided 2026-10-25). + Each address and phone carries its archive row's ID as its own `filemaker_code` (ADR-0005). - **Out of scope:** event registrations (`fm_participants`, 68,497 rows), payments, funding, notes and activities — each its own pass. diff --git a/spec/models/organization_spec.rb b/spec/models/organization_spec.rb index ac4791c451..f1c9c359bc 100644 --- a/spec/models/organization_spec.rb +++ b/spec/models/organization_spec.rb @@ -13,6 +13,26 @@ it { should have_many(:users).through(:people) } it { should have_many(:reports) } it { should have_many(:workshop_logs) } + it { should have_many(:contact_methods).dependent(:destroy) } + end + + describe "#phone_number" do + let(:organization) { create(:organization) } + + it "prefers the primary active phone" do + organization.contact_methods.create!(kind: "phone", value: "555-0001") + organization.contact_methods.create!(kind: "phone", value: "555-0002", primary: true) + organization.contact_methods.create!(kind: "phone", value: "555-0003", primary: true, inactive: true) + + expect(organization.reload.phone_number).to eq("555-0002") + end + + it "falls back to a phone an active address holds" do + create(:address, addressable: organization, phone: "555-0004", inactive: true) + create(:address, addressable: organization, phone: "555-0005") + + expect(organization.reload.phone_number).to eq("555-0005") + end end describe 'validations' do diff --git a/spec/requests/dedupable_spec.rb b/spec/requests/dedupable_spec.rb index fd2c1f4dde..ecbf3acb81 100644 --- a/spec/requests/dedupable_spec.rb +++ b/spec/requests/dedupable_spec.rb @@ -415,6 +415,20 @@ expect(Organization.exists?(delete_rec.id)).to be true end + it "moves the duplicate's phones onto the keeper, folding a shared number into one" do + keep.contact_methods.create!(kind: "phone", value: "555-0100", primary: true, filemaker_code: "10") + delete_rec.contact_methods.create!(kind: "phone", value: "555-0100", primary: true, filemaker_code: "11") + delete_rec.contact_methods.create!(kind: "phone", value: "555-0199") + + post dedupe_perform_organizations_path, params: { + organization_to_delete_id: delete_rec.id, + organization_to_keep_id: keep.id + } + + expect(keep.contact_methods.order(:id).pluck(:value, :primary, :filemaker_code)) + .to eq([ [ "555-0100", true, "10, 11" ], [ "555-0199", false, nil ] ]) + end + it "combines both organizations' FileMaker codes onto the keeper" do keep.update!(filemaker_code: "FM-100") delete_rec.update!(filemaker_code: "FM-273") diff --git a/spec/requests/fm_archives_spec.rb b/spec/requests/fm_archives_spec.rb index 27486889fe..c17b4323db 100644 --- a/spec/requests/fm_archives_spec.rb +++ b/spec/requests/fm_archives_spec.rb @@ -335,6 +335,28 @@ expect(response.body).to include(ERB::Util.html_escape(workshop.title)) end + it "links an archived address back to the person holding it" do + FmAddress.create!(fm_id: "123") + person = create(:person) + create(:address, addressable: person, street_address: "1 Main St", filemaker_code: "123") + + get fm_archive_path("123", table: "addresses") + + expect(response.body).to include(person_path(person)) + expect(response.body).to include(ERB::Util.html_escape("#{person.name}: 1 Main St")) + end + + it "links an archived phone number back to the person holding it" do + FmPhoneNumber.create!(fm_id: "456") + person = create(:person) + person.contact_methods.create!(kind: "phone", value: "555-0100", filemaker_code: "456") + + get fm_archive_path("456", table: "phone_numbers") + + expect(response.body).to include(person_path(person)) + expect(response.body).to include(ERB::Util.html_escape("#{person.name}: 555-0100")) + end + it "omits the portal section when nothing carries the code" do FmRolodex.create!(fm_id: "00006") diff --git a/spec/requests/organizations_spec.rb b/spec/requests/organizations_spec.rb index 3db97c173d..4a03953189 100644 --- a/spec/requests/organizations_spec.rb +++ b/spec/requests/organizations_spec.rb @@ -477,6 +477,14 @@ expect(organization.reload.high_profile).to be(true) end + it "saves the organization's phones" do + organization = Organization.create!(valid_attributes) + patch organization_url(organization), params: { organization: { contact_methods_attributes: { + "0" => { kind: "phone", contact_type: "work", value: "555-0100", primary: "1" } + } } } + expect(organization.contact_methods.sole).to have_attributes(value: "555-0100", contact_type: "work", primary: true) + end + it "nests the organization under a parent" do parent = Organization.create!(valid_attributes.merge(name: "Roof Org")) organization = Organization.create!(valid_attributes) diff --git a/spec/services/address_deduper_spec.rb b/spec/services/address_deduper_spec.rb index 55110d858a..7a5bba855b 100644 --- a/spec/services/address_deduper_spec.rb +++ b/spec/services/address_deduper_spec.rb @@ -151,6 +151,15 @@ def address(primary: false, **overrides) expect(keep.reload.phone).to eq("555-9000") end + it "keeps every FileMaker code the merged addresses carried" do + keep = address(filemaker_code: "12") + address(filemaker_code: "7, 30") + + described_class.new(person).call + + expect(keep.reload.filemaker_code).to eq("12, 30, 7") + end + it "leaves the survivor's own phone alone when the duplicate also has one" do keep = address(phone: "555-1000") address(phone: "555-9000") diff --git a/spec/services/affiliation_services/import_from_filemaker_spec.rb b/spec/services/affiliation_services/import_from_filemaker_spec.rb index 5700a45b4d..f1fd15cd97 100644 --- a/spec/services/affiliation_services/import_from_filemaker_spec.rb +++ b/spec/services/affiliation_services/import_from_filemaker_spec.rb @@ -304,10 +304,24 @@ def stint(starts, ends = nil) affiliation = person.affiliations.sole expect(affiliation).to have_attributes(organization: umbrella, title: "Counselor", filemaker_code: nil) - expect(affiliation.organization_address).to have_attributes(addressable: umbrella, street_address: "1 Main St", address_type: "work") + expect(affiliation.organization_address).to have_attributes(addressable: umbrella, street_address: "1 Main St", address_type: "work", + filemaker_code: "40") expect(person.addresses).to be_empty end + it "adds the archive row's code to a work address the organization already has" do + held = create(:address, addressable: umbrella, street_address: "1 Main Street", city: "Pasadena", state: "CA") + FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", + "AddrsState" => "CA", "AddrsType" => "Work" }) + rolodex(OrgID: "ORG00001") + + import + + expect(umbrella.addresses.sole).to eq(held) + expect(held.reload.filemaker_code).to eq("40") + expect(person.affiliations.sole.organization_address).to eq(held) + end + it "places a work address on the person's only organization when the card names none" do create(:affiliation, person:, organization: program, title: "Facilitator") FmAddress.create!(fm_id: "40", data: { "RolodexID" => "00006", "AddrsLine1" => "1 Main St", "AddrsCity" => "Pasadena", @@ -326,7 +340,7 @@ def stint(starts, ends = nil) import - expect(person.addresses.sole).to have_attributes(street_address: "1 Main St", address_type: "work") + expect(person.addresses.sole).to have_attributes(street_address: "1 Main St", address_type: "work", filemaker_code: "40") end end diff --git a/spec/services/person_services/reconcile_contact_methods_spec.rb b/spec/services/contact_method_deduper_spec.rb similarity index 90% rename from spec/services/person_services/reconcile_contact_methods_spec.rb rename to spec/services/contact_method_deduper_spec.rb index 601bc8228c..72e55f773c 100644 --- a/spec/services/person_services/reconcile_contact_methods_spec.rb +++ b/spec/services/contact_method_deduper_spec.rb @@ -2,7 +2,7 @@ require "rails_helper" -RSpec.describe PersonServices::ReconcileContactMethods do +RSpec.describe ContactMethodDeduper do let(:person) { create(:person) } def contact_method(value: "555-1000", kind: "phone", **overrides) @@ -93,4 +93,13 @@ def address expect(keep.reload.address_id).to eq(kept_address.id) end + + it "keeps every FileMaker code the merged contact methods carried" do + keep = contact_method(filemaker_code: "22") + contact_method(filemaker_code: "35") + + described_class.new(person).call + + expect(keep.reload.filemaker_code).to eq("22, 35") + end end diff --git a/spec/services/organization_services/import_from_filemaker_spec.rb b/spec/services/organization_services/import_from_filemaker_spec.rb index c320e1ddd7..8254112062 100644 --- a/spec/services/organization_services/import_from_filemaker_spec.rb +++ b/spec/services/organization_services/import_from_filemaker_spec.rb @@ -31,9 +31,23 @@ def import website_url: "harbor.example.org", mission_vision_values: "Shelter for all", organization_type: "501c3/nonprofit", email: "info@harbor.example.org") expect(organization.addresses.sole).to have_attributes(street_address: "1 Main St", city: "Pasadena", zip_code: "91101", - address_type: "work", primary: true, phone: "555-0100", + address_type: "work", primary: true, filemaker_code: "1", locality: "LA County", district: "Foothills", la_service_planning_area: 3, la_supervisorial_district: 5) + expect(organization.contact_methods.sole).to have_attributes(kind: "phone", value: "555-0100", primary: true, filemaker_code: "2") + end + + it "adds the archive row's code to an address the organization already has" do + existing = create(:organization, filemaker_code: "ORG00001") + address = create(:address, addressable: existing, street_address: "1 Main St", city: "Pasadena", state: "CA") + fm_organization("ORG00001") + FmAddress.create!(fm_id: "1", data: { "OrgID" => "ORG00001", "AddrsLine1" => "1 Main Street", "AddrsCity" => "Pasadena", + "AddrsState" => "CA" }) + + import + + expect(existing.addresses.sole).to eq(address) + expect(address.reload.filemaker_code).to eq("1") end describe "organization type" do @@ -77,34 +91,33 @@ def import end describe "phones" do - def address(fm_id, street) - FmAddress.create!(fm_id:, data: { "OrgID" => "ORG00001", "AddrsLine1" => street, "AddrsCity" => "Pasadena", "AddrsState" => "CA" }) + def phone(fm_id, number, **data) + FmPhoneNumber.create!(fm_id:, data: { "OrgID" => "ORG00001", "PhoneNumber" => number }.merge(data.stringify_keys)) end - it "puts each number on an address with no phone yet, the primary first" do + it "brings each number in as a phone carrying its code, the primary first" do fm_organization("ORG00001") - address("1", "1 Main St") - address("2", "2 Main St") - FmPhoneNumber.create!(fm_id: "10", data: { "OrgID" => "ORG00001", "PhoneNumber" => "555-0101", "Primacy" => "1" }) - FmPhoneNumber.create!(fm_id: "11", data: { "OrgID" => "ORG00001", "PhoneNumber" => "555-0102" }) + phone("11", "555-0102") + phone("10", "555-0101", Primacy: "1", PhoneType: "Work") import - expect(Organization.sole.addresses.order(:id).pluck(:street_address, :phone)) - .to eq([ [ "1 Main St", "555-0101" ], [ "2 Main St", "555-0102" ] ]) + expect(Organization.sole.contact_methods.order(:id).pluck(:value, :contact_type, :primary, :filemaker_code)) + .to eq([ [ "555-0101", "work", true, "10" ], [ "555-0102", "unknown", false, "11" ] ]) end - it "notes a number no address is free to hold, keeping the ones on file" do + it "adds the code to a phone the organization already has, keeping its primary" do existing = create(:organization, filemaker_code: "ORG00001") - create(:address, addressable: existing, street_address: "1 Main St", city: "Pasadena", state: "CA", phone: "555-0100") + held = existing.contact_methods.create!(kind: "phone", value: "(555) 0100", primary: true) fm_organization("ORG00001") - FmPhoneNumber.create!(fm_id: "10", data: { "OrgID" => "ORG00001", "PhoneNumber" => "555-0199" }) + phone("10", "555-0100") + phone("11", "555-0199", Primacy: "1") - import = described_class.new.call + import - expect(existing.addresses.sole.phone).to eq("555-0100") - expect(existing.comments.sole.body).to include("Phone", "555-0100", "555-0199") - expect(import.differences.sole).to include("555-0199") + expect(existing.contact_methods.order(:id).pluck(:value, :primary, :filemaker_code)) + .to eq([ [ "(555) 0100", true, "10" ], [ "555-0199", false, "11" ] ]) + expect(held.reload.filemaker_code).to eq("10") end end diff --git a/spec/services/person_services/import_from_rolodex_spec.rb b/spec/services/person_services/import_from_rolodex_spec.rb index 6850e33d30..a919cc7473 100644 --- a/spec/services/person_services/import_from_rolodex_spec.rb +++ b/spec/services/person_services/import_from_rolodex_spec.rb @@ -436,10 +436,22 @@ def results results person = Person.find_by!(filemaker_code: "00060") - expect(person.addresses.order(:id).pluck(:street_address, :primary)) - .to eq([ [ "1 Main St", true ], [ "2 Second St", false ] ]) - expect(person.contact_methods.order(:id).pluck(:value, :primary)) - .to eq([ [ "555-0100", true ], [ "555-0200", false ] ]) + expect(person.addresses.order(:id).pluck(:street_address, :primary, :filemaker_code)) + .to eq([ [ "1 Main St", true, "A21" ], [ "2 Second St", false, "A20" ] ]) + expect(person.contact_methods.order(:id).pluck(:value, :primary, :filemaker_code)) + .to eq([ [ "555-0100", true, "P21" ], [ "555-0200", false, "P20" ] ]) + end + + it "folds two archive rows for one address into one address carrying both codes" do + FmAddress.create!(fm_id: "A22", data: { "RolodexID" => "00061", "AddrsLine1" => "1 Main St", + "AddrsCity" => "Los Angeles", "AddrsState" => "CA" }) + FmAddress.create!(fm_id: "A23", data: { "RolodexID" => "00061", "AddrsLine1" => "1 Main Street", + "AddrsCity" => "Los Angeles", "AddrsState" => "CA" }) + rolodex("00061", EMail: "ada@example.com") + + results + + expect(Person.find_by!(filemaker_code: "00061").addresses.sole.filemaker_code).to eq("A22, A23") end it "reads the address and phone tables once, not once per row" do @@ -706,6 +718,22 @@ def run_events(run) = Ahoy::Event.where("JSON_UNQUOTE(JSON_EXTRACT(properties, ' .to eq([ [ "1 Main St", true ], [ "2 Second St", false ] ]) expect(person.contact_methods.pluck(:value, :primary)).to eq([ [ "555-0100", true ] ]) end + + it "adds the archive's codes to the address and phone the person already holds" do + address = create(:address, addressable: person, street_address: "1 Main St", city: "Los Angeles", state: "CA") + phone = person.contact_methods.create!(kind: "phone", value: "(555) 0100") + FmAddress.create!(fm_id: "A72", data: { "RolodexID" => "00074", "AddrsLine1" => "1 Main Street", + "AddrsCity" => "Los Angeles", "AddrsState" => "CA" }) + FmPhoneNumber.create!(fm_id: "P72", data: { "RolodexID" => "00074", "PhoneNumber" => "555-0100" }) + rolodex("00074", EMail: "ada@example.com") + + described_class.new.call + + expect(person.addresses.sole).to eq(address) + expect(address.reload.filemaker_code).to eq("A72") + expect(person.contact_methods.sole).to eq(phone) + expect(phone.reload.filemaker_code).to eq("P72") + end end it "links an existing person with the same name and email instead of creating one" do From 66020cf071417edb2d3bbe2f6113268412322542 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Sun, 25 Oct 2026 17:16:02 -0400 Subject: [PATCH 25/25] Edit org phones in the details card, above the phones on addresses MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Main's org edit restyle listed address phones in the details card, and this branch's org phone records got their own Phones section further down — two blocks both called Phones. Like the person form, the details card now edits the organization's own numbers and lists the address phones beneath them. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- app/views/organizations/_form.html.erb | 50 +++++++++++++------------- spec/requests/organizations_spec.rb | 16 +++++++-- 3 files changed, 39 insertions(+), 29 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 4b340c5eec..0bce588edd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -49,7 +49,7 @@ This codebase (Rails 8.1) | Directory | Purpose | Count | |---|---|---| | `app/models/` | ActiveRecord models | ~146 files | -| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~155 files | +| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~161 files | | `app/jobs/` | SolidQueue background jobs | 6 files | | `app/models/concerns/` | Shared model modules | 25 concerns | diff --git a/app/views/organizations/_form.html.erb b/app/views/organizations/_form.html.erb index b489d72d6b..da9e236fe7 100644 --- a/app/views/organizations/_form.html.erb +++ b/app/views/organizations/_form.html.erb @@ -96,20 +96,32 @@ input_html: { class: "w-full rounded-md border-gray-300 shadow-sm focus:ring-blue-500 focus:border-blue-500" } %> -
    +
    Phones
    - <% phones = f.object.decorate.address_phones %> - <% numbered = f.object.addresses.size > 1 %> - <% if phones.any? %> -
      - <% phones.each do |number, phone| %> -
    1. <% if numbered %><%= number %>. <% end %><%= phone %>
      2. - <% end %> -
    - <% else %> -

    None yet.

    + <%= f.simple_fields_for :contact_methods do |sfi| %> + <%= render "contact_methods/contact_method_fields", f: sfi %> + <% end %> + + <%= link_to_add_association "➕ Add phone", + f, + :contact_methods, + partial: "contact_methods/contact_method_fields", + data: { association_insertion_node: "this" }, + class: button_classes(:secondary_outline) %> + + <% address_phones = f.object.decorate.address_phones %> + <% if address_phones.any? %> + <% numbered = f.object.addresses.size > 1 %> +
    +
    On addresses
    +
      + <% address_phones.each do |number, phone| %> +
    • <% if numbered %><%= number %>. <% end %><%= phone %>
      • + <% end %> +
    +

    Saved per <%= link_to (numbered ? "address number" : "address"), "#addresses", class: "underline" %>

    +
    <% end %> -

    Saved per <%= link_to (numbered ? "address number" : "address"), "#addresses", class: "underline" %>

    @@ -241,20 +253,6 @@ class: button_classes(:secondary_outline) %>
    -
    -

    Phones

    - <%= f.simple_fields_for :contact_methods do |sfi| %> - <%= render "contact_methods/contact_method_fields", f: sfi %> - <% end %> - - <%= link_to_add_association "➕ Add phone", - f, - :contact_methods, - partial: "contact_methods/contact_method_fields", - data: { association_insertion_node: "this" }, - class: button_classes(:secondary_outline) %> -
    -

    Affiliations

    diff --git a/spec/requests/organizations_spec.rb b/spec/requests/organizations_spec.rb index 4a03953189..d97c433bcb 100644 --- a/spec/requests/organizations_spec.rb +++ b/spec/requests/organizations_spec.rb @@ -271,18 +271,30 @@ end describe "GET /edit" do - it "lists address phones in the details, numbered by address when there are several" do + it "lists address phones under the phone editor, numbered by address when there are several" do organization = create(:organization) create(:address, addressable: organization, phone: "123-123-1234") create(:address, addressable: organization, phone: "123-332-1234") get edit_organization_url(organization) - phones = Capybara.string(response.body).find("ol", text: "123-123-1234") + phones = Capybara.string(response.body).find("#phones") + expect(phones).to have_text("On addresses") expect(phones).to have_css("li", text: "1. 123-123-1234") expect(phones).to have_css("li", text: "2. 123-332-1234") end + it "edits the organization's own phone numbers beside them" do + organization = create(:organization) + organization.contact_methods.create!(kind: "phone", value: "555-0100") + + get edit_organization_url(organization) + + phones = Capybara.string(response.body).find("#phones") + expect(phones).to have_field(with: "555-0100") + expect(phones).to have_link("➕ Add phone") + end + it "leaves focus alone so in-page links keep their scroll position" do get edit_organization_url(create(:organization))