Skip to content

Shared address translation for imports and a one-time cleanup - #2829

Merged
maebeale merged 10 commits into
mainfrom
maebeale/address-translation
Oct 25, 2026
Merged

maebeale merged 10 commits into
mainfrom
maebeale/address-translation

Conversation

@maebeale

@maebeale maebeale commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 suggested review level: 5 Inspect 🔬 rewrites existing address + form-answer data; changes two importers

Countries and states arrive as free text from FileMaker, the LMS registration sheet, and pre-dropdown portal forms ("USA", "California", "CA - California", a province with no country), so one place shows up as several and the new dropdowns can't preselect them — one shared translator now fixes that for every import and for the data already in the portal.

  • AddressTranslation — country → a dropdown name; US state → USPS code; address type; locality.
    • A blank country is read from the state: a US state → United States, ON → Canada, England → United Kingdom.
    • Lookup tables only, never guessed (same rule as TagTranslation, ADR-0007).
    • A country it can't place is cleared and reported, so every stored country is a dropdown name or blank. The old value stays in the Ahoy log.
    • An unreadable state is kept and reported: state is required, and regions outside the US are legitimate.
  • Imports — FmAddressAttributes (moved here from Import FileMaker organizations, programs, and affiliations #2788) builds every FileMaker address through it. Today that's the people import; after Import FileMaker organizations, programs, and affiliations #2788 rebases, the organizations and affiliations imports too. The LMS registration importer uses it as well. FileMaker's own fm_* tables stay raw.
    • The LMS import preview lists values the dropdowns don't offer (country left blank, state kept as typed), and the completion notice counts them.
  • Existing data — data:translate_addresses (dry run by default) rewrites address countries/states and country/state form answers.
    • It fills a locality only where it's Unknown.
    • Every change is an autochange.* Ahoy event. Running it again changes nothing.
  • No save hook — the dropdowns own new data.
  • Form answers — country/state answers are now the one documented exception to "stored verbatim" (ADR-0009).

Prod runbook

  1. Deploy, then bin/rails data:translate_addresses (dry run by default). Read the changes and the "nothing translates" list. Every run saves its report to storage and prints the read-back command.
  2. Pick an import user in admin Settings (Imports) if none is set, then run DRY_RUN=false bin/rails data:translate_addresses. Every change is credited to that user.
  3. Merge this PR, rebase Import FileMaker organizations, programs, and affiliations #2788, then run the FileMaker organizations import. It translates as it goes.
  4. Run step 2 again to catch anything since.

🤖 Generated with Claude Code

@maebeale maebeale changed the title WIP: Shared address translation for imports and cleanup Shared address translation for imports and a one-time cleanup Oct 24, 2026
@maebeale
maebeale marked this pull request as ready for review October 24, 2026 22:50
maebeale added a commit that referenced this pull request Oct 24, 2026
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs
#2829 merged and a rebase before it runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
maebeale added a commit that referenced this pull request Oct 24, 2026
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs
#2829 merged and a rebase before it runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
maebeale and others added 7 commits October 24, 2026 19:46
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
FileMaker and LMS imports carried free-text states and countries ("California",
"USA", "CA - California", provinces in the state field) that split one place into
several and don't preselect in the dropdowns.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ogging each change

Portal data from before the dropdowns holds the same free-text spellings the imports
did; one idempotent pass (before and after the FileMaker imports) brings it in line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…down one

Portal addresses and country answers should only hold dropdown values; the old text
stays in the Ahoy log, the fm_* archive, or the import report.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A blanked country or unreadable state shouldn't land silently; the preview lists them
before importing and the notice counts them after.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every FileMaker import (people now, organizations and affiliations next) needs the same
row-to-address reading; keeping it beside AddressTranslation keeps all address handling
in one place.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@maebeale
maebeale force-pushed the maebeale/address-translation branch from 3f57286 to 467fd73 Compare October 24, 2026 23:47
maebeale and others added 3 commits October 24, 2026 19:51
A console-only report scrolls away; the list of values nothing translates is what
someone fixes by hand afterwards, so every run keeps a copy in storage.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every rewrite should name who made it; the same import user fm:import_people credits
is the one to use, and a real run won't proceed without one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@maebeale

Copy link
Copy Markdown
Collaborator Author

@jmilljr24 i'm merging now to check it out in staging

@maebeale
maebeale merged commit 0f3faef into main Oct 25, 2026
3 checks passed
@maebeale
maebeale deleted the maebeale/address-translation branch October 25, 2026 00:16
maebeale added a commit that referenced this pull request Oct 25, 2026
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@maebeale
maebeale restored the maebeale/address-translation branch October 25, 2026 00:58
@maebeale
maebeale deleted the maebeale/address-translation branch October 25, 2026 00:59
maebeale added a commit that referenced this pull request Oct 25, 2026
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
maebeale added a commit that referenced this pull request Oct 25, 2026
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
maebeale added a commit that referenced this pull request Oct 25, 2026
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
maebeale added a commit that referenced this pull request Oct 25, 2026
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
maebeale added a commit that referenced this pull request Oct 25, 2026
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
maebeale added a commit that referenced this pull request Oct 25, 2026
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
maebeale added a commit that referenced this pull request Oct 25, 2026
* Start the FileMaker organizations and affiliations import

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Import FileMaker organizations, programs, and affiliations

The Rolodex pass brought people over but deliberately left out everything
tying a person to an organization, so imported people have no jobs, no
facilitator history, and no work addresses. This is the second pass ADR-0008
describes: organizations and projects become (nested) orgs, personnel and
people-to-organizations rows become affiliations, and each Rolodex card's
org, title and work addresses land where they belong.

Agency cards were skipped wholesale by the people pass, losing the contact
person many of them name; those people now come through, while the card's
addresses and website stay with the agency's org.

Raw-value parsing and address building move out of ImportFromRolodex so all
three passes read the archive the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Record that every FileMaker organization comes over

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Read Personnel roles and org types as production records them

The archive's real values showed no Personnel role says Facilitator —
FileMaker calls a facilitator a Leader — so every assignment would have
landed as a job and no program would read as running. Liaisons and
assistants support a program without facilitating it, so they stay jobs.

Organization types were already close to the portal's own four; a word
none of them covers is Other with the word kept, not dropped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Import a FileMaker Leader as a primary-contact Facilitator

Prod's affiliations already record the old app's leaders this way — the
January 2026 position-to-title conversion and #2321 folded Leader into a
Facilitator who is the program's primary contact — so FileMaker's leaders
should arrive looking the same.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Give FileMaker liaisons and assistants prod's January shape

Prod's existing affiliations show every Liaison as a primary contact, and
the January 2026 conversion gave each liaison or assistant a Facilitator
affiliation over the same dates beside their role. FileMaker's rows now
arrive the same way, so the two systems' people read alike.

A row prod still titles Leader is the same facilitation, so it links
rather than gaining a Facilitator twin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Import FileMaker liaisons as primary-contact facilitators, noting the role

Prod records a liaison the way it records a leader — a Facilitator who is a
primary contact — so FileMaker's liaisons arrive that way too. Folding both
into Facilitator would lose the word FileMaker used, so each affiliation
gets a comment saying how FileMaker listed the person at import.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Import every FileMaker Personnel role as a Facilitator

Assistants are an AWBW program role, not a job title, and prod holds no
Assistant-titled rows — the old app's assistants became Facilitators who
aren't primary contacts. FileMaker's arrive the same way, with the role
kept in a comment like leaders' and liaisons'.

A leader or liaison matching a Facilitator row the portal already had now
turns its primary contact on, saying so in that comment, so the existing
row ends up the shape a fresh import would have made.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Keep a FileMaker program on record when its Personnel row names no one

Some Personnel rows name a project but no person. Skipping them would leave
an organization with no trace that its program ran, so when nothing else
affiliates anyone there, a placeholder person, AWBW Placeholder, holds that
history. It's read last so a real person always wins.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Note the AWBW Placeholder fallback in AGENTS.md

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Mark primary contact only where FileMaker says so; match by stretch

A blank or unrecognized role says nothing about who the program's contact
was, so only a FileMaker Leader or Liaison is one now.

A person can facilitate at one organization over several separate
stretches, so a Personnel row links an existing Facilitator row only when
their periods overlap — otherwise it would fold a 2015 assignment into a
2024 one. Blank dates read as open-ended, except an ended row with no end
date, which stays at its start so it can't reach a later stretch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Expect affiliations to point into people-to-organizations too

The import gives job affiliations a FileMaker people-to-organizations code,
so the spec that lists each model's archive tables needs it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Take address handling out of the organizations import; use the shared translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Fold leftover Leader affiliations into Facilitator from Data health

Prod still holds 111 rows titled Leader, which count toward no program
status. The January conversion already gave 109 of them a Facilitator twin
with the same dates, leaving the Leader row as a redundant copy; the other
2 have nothing beside them. One Data health fix merges each copy into its
twin (keeping its FileMaker codes) or renames a lone one, so no duplicate
remains, and lists any that overlap a Facilitator row with different dates
for a person to settle.

The stretch-of-time test the import uses moves into AffiliationStint so
both read a person's separate stints the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Review Leader affiliations row by row instead of one fold button

One button that merges, renames and deletes 111 rows at once asks an admin
to trust changes they can't see. The Leader affiliations page works like
attendance reconciliation: each row shows its Facilitator twin and offers
merge, rename, rename and end, delete, or leave, with a suggestion
preselected, and a preview lists every change before anything is made.
Data health now reports the rows and links to the page.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Only report Leader affiliations; leave the cleanup to person edit

Changing a person's affiliations should happen where an admin can see the
whole person, not from a bulk page. Data health lists each row still titled
Leader beside the Facilitator row that covers it, if any, and opens the
person's edit page at that affiliation, with the back link returning to the
check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Show whether a Leader row's person already facilitates there

Settling a Leader row by hand starts with one question: does this person
already have a Facilitator affiliation at the organization? Each row now
answers it with a badge, lists those Facilitator rows and how their
stretch meets the Leader row's, and notes when they facilitate elsewhere.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Report every old-app role title in Data health, not just Leader

The January conversion copied all of the old app's roles — Leader, Liaison,
Assistant and a blank default — into affiliation titles, and none of them
counts toward program status; prod still has 53 Liaison rows beside the 111
Leader ones. The check now lists all of them, and the import matches any of
those titles as a Facilitator so it doesn't add a twin beside a Liaison row.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Recount services and rake tasks after main dropped the address translation task

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Land a Personnel row with no imported project at the person's own organization

27 Personnel rows name no project, and none of their agreements names one
either, so the import skipped them and the program history went with them.
The person's Rolodex card still says where they worked, so the row lands
there — creating the organization from its typed name when nothing matches.
A row whose only person is an agency's own card is treated as naming no
one, so AWBW Placeholder keeps that program's history.

Features entries move to today, the date this ships.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Bring differing FileMaker affiliations and phones in as their own records

An affiliation whose dates differ from the portal's is a separate stretch
of facilitation, not a disagreement about one, so folding it into the
portal's row and noting the difference lost the history. It now comes in as
its own affiliation; only one whose dates agree (or fill each other's blanks)
is linked. An organization's phone numbers each go onto an address that has
none, so a second number isn't dropped; one with no free address is still
noted on the organization.

The agency-card path is pinned end to end: the card becomes an organization
with its work address, and the person it names is affiliated there at that
address.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Pin that an undated affiliation on either side updates the other

The rule was already in the matching; a spec and the docs now say so, so a
later change to date matching can't quietly start duplicating them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Bring agency contact people in from the organizations import

The people import already ran and its people have been cleaned up, so it
won't be re-run — but it skipped every agency card, so the contact person a
card names never came in and couldn't be affiliated with the agency. The
organizations import now runs the Rolodex import over agency cards alone as
its first step, leaving everyone else as the cleanup left them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* FileMaker codes on imported addresses and phones; phone records on organizations (#2860)

* Start: FileMaker codes on imported addresses and phones

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Trace imported addresses and phones back to their FileMaker rows

People carried their Rolodex code, but the addresses and phones the import
brought along didn't, so the archive couldn't say which of its rows had landed
or where. Each now carries its AddrsID / PhoneID, an address or number the
person already held takes the code instead of a copy, and merges keep every code.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Give organizations phone records, and trace their FileMaker addresses and phones

The organizations import squeezed an org's numbers onto its addresses, dropping
any with no free address and leaving no record to carry the FileMaker code.
Orgs now keep phones as contact methods like people, edited on the org form and
folded on merge, so each imported number and address carries its archive ID.
ReconcileContactMethods moves to ContactMethodDeduper since orgs share it now.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Edit org phones in the details card, above the phones on addresses

Main's org edit restyle listed address phones in the details card, and this
branch's org phone records got their own Phones section further down — two
blocks both called Phones. Like the person form, the details card now edits
the organization's own numbers and lists the address phones beneath them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant