Skip to content

Import FileMaker organizations, programs, and affiliations - #2788

Open
maebeale wants to merge 19 commits into
mainfrom
maebeale/affiliations-organizations-import
Open

maebeale wants to merge 19 commits into
mainfrom
maebeale/affiliations-organizations-import

Conversation

@maebeale

@maebeale maebeale commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 suggested review level: 5 Inspect 🔬 new bulk import that writes orgs, affiliations and addresses across the whole archive

Needs #2829 merged, then a rebase. Address handling (FmAddressAttributes, the contact-type table, country/state translation) moved to #2829; this branch calls it. Expect small conflicts in ImportFromRolodex and AGENTS.md — keep this side's structure plus #2829's unmatched_addresses.

Imported people still have no jobs, facilitator history, or work addresses; this second FileMaker pass (ADR-0008) brings in the organizations and who worked or facilitated where.

What is the goal of this PR and why is this important?

  • fm:import_organizations runs after fm:import_people. Dry run by default; undo with fm:undo_import RUN=…

  • Orgs + programs: FM organizations and projects become portal orgs

    • An org with one project stays one org, carrying both codes
    • An org with several projects becomes a parent, with each program nested under it
  • Affiliations: from three sources

    • Personnel → matches how prod already records the old app's roles:
      • Every role → Facilitator, with a comment saying how FileMaker listed them
      • Only Leader or Liaison → primary contact; a matched existing Facilitator row gets it turned on
      • Assistant, blank or anything else → not primary contact
      • Links an existing Facilitator row only for the same stretch (overlapping dates), so separate stints stay separate
      • A row naming a project but no person → "AWBW Placeholder" stands in, only where the org has no other affiliations
      • An existing prod row titled with an old-app role (Leader, Liaison, …) links instead of getting a duplicate
    • People-to-organizations → job affiliation
    • The Rolodex card's org + Title → job affiliation
  • Work addresses: go onto the org, and the person's affiliation there points at them

    • With no org to land on, the address stays with the person
  • Agency cards: the contact person they name now comes in as a person

    • The card's website and addresses fill the agency's org
  • Leftover old-app role rows: a new Data health check lists every affiliation still titled Leader, Liaison, Assistant or "default" (prod: 111 Leader, 53 Liaison) for hand cleanup

    • Each row badges whether the person already has a Facilitator row at that org, and lists those rows
    • Each name opens the person's edit page at that affiliation; nothing changes them automatically

How did you approach the change?

  • Every FM org comes over, even ones nothing points at
  • Personnel: only Status "Active" stays open, from the production values
  • Org type: FileMaker's words map onto the 4 portal types; any other word ("School") is Other with the word as its description
  • Each source is matched by filemaker_code first, then exact name, so a rerun changes nothing
  • Fill blanks only. Where the portal disagrees, it keeps its value and leaves a comment
  • Shared archive parsing moved out of ImportFromRolodex (FilemakerValues, FmAddressAttributes)
  • ADR-0008 now records these choices. Its open questions carry the draft's provisional answers

Anything else to add?

Decisions to confirm before the real run:

  • Reconcile: the reconcile page treats imported Facilitator rows as hand-entered, so it can end an open one when the person no-shows a training. Since Reconcile cancelled training registrants before the event #2803, cancelling a registration can end one too, even before the event. Every change still goes through the preview page. Leaving as is unless we exempt PS…-coded rows
  • Not carried: AgreementStatus, Department, project ProjectType

Separately, I noticed fm:import_people wraps a real run in one outer transaction, so its per-batch commits are really savepoints. The new task commits each batch for real.

🤖 Generated with Claude Code

@maebeale maebeale changed the title Import FileMaker organizations, projects, and affiliations Import FileMaker organizations, programs, and affiliations Oct 22, 2026
BATCH_SIZE = 1_000
# Personnel Status values that mean the person still staffs the program. Anything else
# reads as ended — over-reporting active programs would skew program status.
ACTIVE_STATUSES = %w[active current].freeze

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 From Claude: This is a guess. No real Personnel Status values were available locally. Check the dry run's "Personnel Status values" tally before the real run, because every other value ends the affiliation.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 From Claude: Settled from the production values: only "Active" (1,774 of 8,907) stays open.

@maebeale
maebeale force-pushed the maebeale/affiliations-organizations-import branch 3 times, most recently from 86aa950 to 7f03d8e Compare October 24, 2026 15:34
@maebeale
maebeale marked this pull request as ready for review October 24, 2026 15:35
@maebeale
maebeale force-pushed the maebeale/affiliations-organizations-import branch 2 times, most recently from 5f30788 to dd933cd Compare October 25, 2026 00:19
maebeale and others added 19 commits October 25, 2026 05:40
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Rolodex pass brought people over but deliberately left out everything
tying a person to an organization, so imported people have no jobs, no
facilitator history, and no work addresses. This is the second pass ADR-0008
describes: organizations and projects become (nested) orgs, personnel and
people-to-organizations rows become affiliations, and each Rolodex card's
org, title and work addresses land where they belong.

Agency cards were skipped wholesale by the people pass, losing the contact
person many of them name; those people now come through, while the card's
addresses and website stay with the agency's org.

Raw-value parsing and address building move out of ImportFromRolodex so all
three passes read the archive the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The archive's real values showed no Personnel role says Facilitator —
FileMaker calls a facilitator a Leader — so every assignment would have
landed as a job and no program would read as running. Liaisons and
assistants support a program without facilitating it, so they stay jobs.

Organization types were already close to the portal's own four; a word
none of them covers is Other with the word kept, not dropped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Prod's affiliations already record the old app's leaders this way — the
January 2026 position-to-title conversion and #2321 folded Leader into a
Facilitator who is the program's primary contact — so FileMaker's leaders
should arrive looking the same.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Prod's existing affiliations show every Liaison as a primary contact, and
the January 2026 conversion gave each liaison or assistant a Facilitator
affiliation over the same dates beside their role. FileMaker's rows now
arrive the same way, so the two systems' people read alike.

A row prod still titles Leader is the same facilitation, so it links
rather than gaining a Facilitator twin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… role

Prod records a liaison the way it records a leader — a Facilitator who is a
primary contact — so FileMaker's liaisons arrive that way too. Folding both
into Facilitator would lose the word FileMaker used, so each affiliation
gets a comment saying how FileMaker listed the person at import.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Assistants are an AWBW program role, not a job title, and prod holds no
Assistant-titled rows — the old app's assistants became Facilitators who
aren't primary contacts. FileMaker's arrive the same way, with the role
kept in a comment like leaders' and liaisons'.

A leader or liaison matching a Facilitator row the portal already had now
turns its primary contact on, saying so in that comment, so the existing
row ends up the shape a fresh import would have made.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Some Personnel rows name a project but no person. Skipping them would leave
an organization with no trace that its program ran, so when nothing else
affiliates anyone there, a placeholder person, AWBW Placeholder, holds that
history. It's read last so a real person always wins.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A blank or unrecognized role says nothing about who the program's contact
was, so only a FileMaker Leader or Liaison is one now.

A person can facilitate at one organization over several separate
stretches, so a Personnel row links an existing Facilitator row only when
their periods overlap — otherwise it would fold a 2015 assignment into a
2024 one. Blank dates read as open-ended, except an ended row with no end
date, which stays at its start so it can't reach a later stretch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The import gives job affiliations a FileMaker people-to-organizations code,
so the spec that lists each model's archive tables needs it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… translator

FmAddressAttributes and the contact-type table now live with AddressTranslation (#2829),
so every FileMaker import stores the dropdowns' countries and states. This branch needs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Prod still holds 111 rows titled Leader, which count toward no program
status. The January conversion already gave 109 of them a Facilitator twin
with the same dates, leaving the Leader row as a redundant copy; the other
2 have nothing beside them. One Data health fix merges each copy into its
twin (keeping its FileMaker codes) or renames a lone one, so no duplicate
remains, and lists any that overlap a Facilitator row with different dates
for a person to settle.

The stretch-of-time test the import uses moves into AffiliationStint so
both read a person's separate stints the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One button that merges, renames and deletes 111 rows at once asks an admin
to trust changes they can't see. The Leader affiliations page works like
attendance reconciliation: each row shows its Facilitator twin and offers
merge, rename, rename and end, delete, or leave, with a suggestion
preselected, and a preview lists every change before anything is made.
Data health now reports the rows and links to the page.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Changing a person's affiliations should happen where an admin can see the
whole person, not from a bulk page. Data health lists each row still titled
Leader beside the Facilitator row that covers it, if any, and opens the
person's edit page at that affiliation, with the back link returning to the
check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Settling a Leader row by hand starts with one question: does this person
already have a Facilitator affiliation at the organization? Each row now
answers it with a badge, lists those Facilitator rows and how their
stretch meets the Leader row's, and notes when they facilitate elsewhere.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The January conversion copied all of the old app's roles — Leader, Liaison,
Assistant and a blank default — into affiliation titles, and none of them
counts toward program status; prod still has 53 Liaison rows beside the 111
Leader ones. The check now lists all of them, and the import matches any of
those titles as a Facilitator so it doesn't add a twin beside a Liaison row.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@maebeale
maebeale force-pushed the maebeale/affiliations-organizations-import branch from 4d917d5 to 3c3e011 Compare October 25, 2026 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant