Classify import exceptions into named buckets (grouped summary + detail), instead of one flat fail file

[bosd]

The idea

Today a failed import produces one flat fail file. In practice the failures are not one thing — they are several distinct populations, each needing a different decision by a different person. Sorting them into named buckets is what actually moves a migration forward.

This is not speculative. On a live migration project the buckets emerged by hand as ~22 CSVs in a review/ directory, built one at a time as each problem surfaced. That accidental taxonomy is the requirement; this issue is about making Fluvo produce it deliberately.

The taxonomy that actually emerged

Generalised from real migration work (part/product data, ~40k affected rows):

Excel mangling — source data damaged before Fluvo ever saw it

• Scientific notation: a 12-digit part number arrives as 1,1253E+11
• Trailing-zero loss: 0100 → 100
• Normalisation collisions: two distinct codes become identical once cleaned
• Case-only duplicates

Duplicates

• Duplicate internal references within the source
• References colliding with records already loaded in the target — a different decision from an intra-source duplicate, because one side is already live

Needs a client decision — nothing is technically wrong; someone has to choose

• Dormant records: archive, restore, or keep
• Records missing a business prerequisite (e.g. no supplier) — migrate anyway or exclude

Blocked on missing reference data

• Unresolved vendor/partner references
• Records referencing a category or code that does not exist
• Reference records that exist but carry no name

Company-dependent conflicts

• The same field carrying different values per company, where the source has only one (see Multi-company: warn on unset company, and import company-dependent fields per company #255)

The organising principle: bucket by who decides, not by what is wrong

This is the part that makes it more than a categorised error log, and it should drive the design:

Owner	Examples	What the tool does
Fluvo can fix	scientific notation, trailing zeros, case normalisation	Propose the correction, show it, apply on confirmation
The client must decide	dormant records, records without a supplier, which duplicate wins	Present the choice; do not guess
Blocked on missing data	unresolved vendors, missing categories	Ask for the missing input

A client should only ever be shown the middle row. Mixing the three is why a raw fail file is unusable as a client-facing artifact.

Volume forces aggregation, not just filtering

Real numbers from that project: the duplicate-reference detail file is 40,340 rows; the trailing-zero file is 454 KB. Neither can go in front of a human.

But the grouped view of the same data is 222 groups — and that is reviewable.

So a bucket is two artifacts, not one:

• a grouped summary (one row per decision, human-sized), and
• the detail rows behind each group (machine-sized, for the fix pass).

This is the same constraint already recorded for the staging portal — heavy lifting in Polars, only what needs a human goes to the review layer.

Scope

v1 — core, free:

• A review/ output directory with one grouped file + one detail file per bucket, and a buckets.json manifest: bucket id, title, decision owner, row count, group count, detail path.
• Stable bucket ids so a bucket can be tracked across re-runs of the same migration.
• Detectors for the taxonomy above, run as part of preflight and after a failed load. Several already exist in embryo — the @lang/@company checks, the Silent corruption: a dict written to a translated field is stringified, not rejected #274 dict guard, the Nightly e2e red on all 4 Odoo versions since 2026-08-19 — company guard aborts an existing groupby test #291 company guard — and should register into the same bucket system rather than each inventing its own output.
• Reuse the normalisation work from fluvo compare: diff Odoo against a spreadsheet (and against another Odoo) #298: the logic that detects "Excel mangled this" for a comparison is the same logic that populates the mangling buckets. Do not implement it twice.
• Buckets are read-only reports. Applying a decision is a separate, explicit step.

Out of scope here: the visual review layer (staging portal / Flowfile — PLAN phase 5, #288) and the client-facing summary report, which is a premium artifact and is tracked privately.

Why this belongs in core

It is part of doing an import well — Fluvo already writes fail files, and this is those fail files made useful. It is also the substrate the premium reporting sits on, so it needs to exist and be good regardless of what gets sold on top.

Acceptance

• Re-running a migration reproduces the same bucket ids for the same underlying problems.
• A bucket with tens of thousands of detail rows still yields a grouped summary a person can read in one screen.
• Every existing detector routes into the bucket system; none keeps a private output format.