Adding a `required` field with a default to a table that has rows fails on PostgreSQL: `ADD COLUMN ... NOT NULL` runs before the default, and the plan calls it `safe`

[rrrodzilla]

Summary

Adding a required field that has a default to a schema whose table already has rows fails on PostgreSQL with column "<field>" of relation "<Table>" contains null values, and the schema is not migrated. The plan calls the step [safe].

The cause is that AddField adds the column as NOT NULL with no default, and sets the default in a second statement. The default never fills the existing rows. CREATE TABLE puts the DEFAULT inline, so a fresh table works and the bug only appears on a table that already has data.

Both ways of declaring a default fail:

• default("draft") (the literal modifier): the SET DEFAULT runs too late.
• @default("0") (the CEL rule): no column default is emitted at all. The rule only runs on writes, so existing rows have no value.

Two related gaps come from the same cause:

• Making an existing optional field required emits only ADD REQUIRED (SET NOT NULL), even when the field has a default. It fails the same way if any row is NULL. MigrationStep::BackfillRequired exists and both backends generate SQL for it, but the differ never emits it.
• Adding an optional field with default(...) leaves every existing row NULL, for the same ordering reason. An inline DEFAULT would fill them.

Version checked

v0.45.0 release binary (x86_64 Linux, PostgreSQL build), source at tag v0.45.0 (10ec403), PostgreSQL 16.

Reproduction

v1/widget.schema:

schema Widget {
    name: text required
}

v2/widget.schema:

schema Widget {
    name:     text required
    status:   enum("draft", "live") required default("draft")
    priority: integer required @default("0")
}

schemaforge apply v1
# insert one row, via the API or directly:
#   INSERT INTO "Widget"(id, name) VALUES ('widget_01', 'one');
schemaforge migrate v2
schemaforge apply v2

migrate shows:

Widget (2 steps, safe)
  1. ADD field 'status' [safe]
  2. ADD field 'priority' [safe]

apply fails:

  Widget           UPDATE (2 steps)  [safe]
error: backend error: migration step failed (ADD field 'status'): error returned from database: column "status" of relation "Widget" contains null values

With only priority (the @default case), it fails the same way on ADD field 'priority'.

The other two gaps, on the same one-row table:

• Adding status: enum("draft", "live") default("draft") (optional) applies, but the existing row has status = NULL.
• Then changing it to required default("draft") plans ADD REQUIRED on 'status' [requires_confirmation], and apply --force fails with column "status" of relation "Widget" contains null values.

Expected

• Adding a required field with a literal default(...) to a table with rows succeeds, and existing rows get the default.
• Adding a required field with no usable backfill value is refused at plan time with a clear message, for example the existing MigrationError::RequiredFieldWithoutDefault, instead of failing halfway through apply. The plan should not call it [safe].
• Making a field required backfills NULLs from its default before SET NOT NULL, or it is refused at plan time.

Actual

The DDL fails at apply time, after a plan that said safe. Applying the change needs hand-written SQL: create the column with the same type, check constraint and default before apply. ADD COLUMN IF NOT EXISTS then skips it.

Source

• crates/schema-forge-postgres/src/codegen.rs:52-93 (AddField): the column is added as ADD COLUMN IF NOT EXISTS ... NOT NULL (:57-70), and the default follows as a separate ALTER COLUMN ... SET DEFAULT (:72-81). PostgreSQL fills existing rows only from a DEFAULT given in the ADD COLUMN itself.
• crates/schema-forge-postgres/src/codegen.rs:383-404 (field_to_column_def, used by CREATE TABLE): the default is inline here, which is why new tables work.
• crates/schema-forge-core/src/migration.rs:896-925 (emit_add_field) emits a bare AddField. :796-817 (diff_field_modifiers) emits a bare AddRequired. Nothing constructs BackfillRequired (:205), although codegen.rs:204-211 and the SurrealDB codegen render it. MigrationError::RequiredFieldWithoutDefault (:987) is also never returned.
• crates/schema-forge-core/src/migration.rs:259: AddField is always classified Safe, including a required field.
• crates/schema-forge-dsl/src/parser.rs:905-914: @default("expr") becomes a CEL rule (FieldAnnotation::Default), not a FieldModifier::Default, so no column default is ever emitted for it.

Suggested fix

1. In AddField, emit the literal default inline: ADD COLUMN IF NOT EXISTS "f" <type> <checks> DEFAULT <literal> NOT NULL. This fixes the default(...) case and the optional-field backfill.
2. For a required field with no literal default:
   • if it has an @default whose expression is constant (no field, now, or principal references), evaluate it once and use the result as the backfill;
   • otherwise add the column nullable, emit BackfillRequired when a value is known, then AddRequired;
   • if there is no value, return RequiredFieldWithoutDefault at plan time. Classify the step requires_confirmation, not safe.
3. In diff_field_modifiers, when a field becomes required and has a default, emit BackfillRequired before AddRequired.
4. Add codegen tests next to add_field_with_default / add_field_required for required + default(...), plus an integration test that migrates a table with one row.