fix: make deployment reconciliation latest-wins

[vmelikyan]

Summary

This changes deployment reconciliation from one long, environment-wide serialized lock to a durable latest-wins model.

Previously, successive commits could queue behind an older deployment. A slow A could block B and C long enough to exhaust the 120 lock-attempt limit, and old work could potentially publish after a newer commit.

Now:

• Every accepted deployment intent advances desiredGeneration.
• observedGeneration records the latest completed generation.
• Pending repository/service scopes are durably coalesced in acceptedRefs.
• New generations can start while older builds are still running.
• Superseded work may finish its current expensive operation, but authority checks prevent it from entering the next major phase or overwriting newer deployment state.
• Native Kubernetes/Helm mutations use short renewable authority locks instead of holding one lock across the entire reconciliation.
• Static environments retain repository-selective behavior; one upstream repository change does not redeploy the entire environment.

Migration 031 adds the generation/mailbox fields and a partial index for builds with pending reconciliation work.

Follow-up reliability fixes

• AuthorityLockLostError during Native Helm deployment is now treated as retryable infrastructure loss. It no longer publishes a false DEPLOY_FAILED, GitHub failure event, comment, or failure metric.
• Generic reconciliation exceptions use BullMQ’s existing attempt count:
  • Intermediate attempts retry without publishing terminal failure.
  • The final attempt publishes one authority-fenced ERROR and advances observedGeneration.
  • This prevents a permanently failing generation from being resurrected by the recovery sweep forever.
• Configuration errors retain their existing immediate terminal behavior.

E2E validation

Forced A → B → C race

Used vmelikyan/lc-test PR #803 with a controllable BuildKit barrier.

• A and B were held inside image builds.
• C started while both A and B were still physically running.
• C completed its build, deployed, and became live before A or B finished.
• After releasing the barrier, A and B completed their existing image builds but created no deployment Jobs.
• C remained the deployed revision.
• No exceeds 120 attempts to lock the resource build deployment errors occurred.

Static environment selectivity

Used vmelikyan/lc-test-2 PR #56 with two services sourced from different repositories.

Verified:

• A tracked push to lc-test-3/main rebuilt and deployed only lc-test-3.
• The environment-page row-level Redeploy action rebuilt only the selected service.
• A tracked push to lc-test/main selected only the Helm service.
• While the first repository’s build was deliberately blocked, a push to the second repository started its build immediately.
• The authoritative generation retained both repositories’ pending work.
• The superseded first pass stopped after its current build and did not deploy stale output.
• A final post-fix smoke reached generation 9/9; only lc-test-3 received new build/deploy Jobs, while the Helm service’s SHA, run UUID, timestamp, and Jobs remained unchanged.

The local Native Helm fixture still encounters the pre-existing worker kubeconfig issue and attempts to contact localhost:8080. Repository selection was verified despite that unrelated deployment failure.

Automated validation

• Full lint passed.
• 349 Jest suites passed.
• 4,153 Jest tests passed.
• 55 workspace-gateway tests passed.
• 4,208 passing tests total.
• Server TypeScript check passed.
• Independent reviews found no race, fencing, or retry-boundary blockers.

pnpm build currently fails during Next.js page-data collection for an unrelated AI-admin route. The same failure reproduces on clean origin/main and was not introduced by this branch.