fix: apply dynamodb gsi changes one index per stack update

[Ignacio Nistal (ignacionistal)]

fix: apply DynamoDB GSI changes one index per stack update

Type of Change*

• New feature
• Bug fix
• Documentation update
• Refactoring
• Hotfix
• Security patch
• UI/UX improvement

Summary

• UAT from staging failed because CloudFormation tried to add two GSIs (PollStatus_Index and WorkflowExternalId_Index) to the existing laboratory-run-table in one update. DynamoDB allows only one GSI create or delete per UpdateTable.
• pnpm run deploy now runs deploy-dynamodb-gsi-waves before the final cdk deploy. If an existing table would change more than one GSI, it applies those indexes one CloudFormation update at a time, then the normal deploy lands the last remaining index.
• Environments that already have the desired indexes (or only need one more) are a no-op. Brand-new tables are unchanged (CreateTable may define many GSIs).

Deploy

You only start one pnpm build-and-deploy or one CI job. That job may run two sequential CloudFormation updates internally (one GSI per update). You do not need to re-trigger CI.
The job will take longer because each GSI must become ACTIVE before the next update. If GitHub OIDC hits the 1-hour role limit during backfill, bump role-duration-seconds.

Testing*

• Merge to staging and let cicd-release-quality-uat run once
• Confirm logs show an intermediate GSI wave (add PollStatus_Index) then the final deploy adding WorkflowExternalId_Index
• Confirm laboratory-run-table ends with both new indexes ACTIVE
• Confirm new tables (laboratory-s3-access-table, laboratory-data-tagging-table, workflow-run-preset-table) still create successfully
• After UAT is unblocked, get this onto development as well so quality/prod pick up the same guard

Impact

• Deploy time: a UAT (or any existing-env) rollout that adds two GSIs will run two CloudFormation updates in one job, so the back-end deploy will take longer. Each GSI must reach ACTIVE before the next update. Later deploys that do not change GSIs are unchanged.
• Runtime: no application behaviour change. Both indexes were already in the table definition; this only makes CloudFormation apply them legally.
• Dependencies: none. No new packages.
• Other environments: no-op if the table already has the desired GSIs (e.g. quality, if those indexes were applied one at a time). Fresh environments still create the table with all GSIs in one CreateTable.

Additional Information

• This is an AWS/DynamoDB UpdateTable limit, not a bug in our table schema. CloudFormation cannot add two GSIs in a single stack update on an existing table.
• The Node 20 AWS SDK warning in the UAT logs is unrelated and was not the failure.
• The failed UAT stack rolled back to UPDATE_ROLLBACK_COMPLETE; the next deploy can proceed.
• GitHub OIDC role-duration-seconds is currently 3600. If GSI backfill is slow, that session may expire mid-job and will need a bump.
• After UAT is unblocked, this should also land on development so quality/prod get the same guard for future multi-GSI merges.

Checklist*

• No new errors or warnings have been introduced.
• All tests pass successfully and new tests added as necessary.
• Documentation has been updated accordingly.
• Code adheres to the coding and style guidelines of the project.
• Code has been commented in particularly hard-to-understand areas.

[diegomayorga-dept]

PR Review: fix: apply dynamodb gsi changes one index per stack update (#925)

Summary

• 1 blocker, 7 major, 1 minor
• Top concerns: the fix silently skips tables in nested stacks (defeats its own purpose for those tables), the 418-line orchestrator has zero test coverage of its wave/failure-restore logic, and an aborted mid-wave deploy can ship app code that queries a GSI the table doesn't have yet

Correctness (1)

major — packages/back-end/scripts/deploy-dynamodb-gsi-waves.ts:226

listAssemblyTemplates/loadDesiredTables only reads the two top-level stack template files from manifest.json ({prefix}-main-back-end-stack and {prefix}-easy-genomics-api-stack). It never opens .nested.template.json files, so any DynamoDB table defined inside a nested stack (e.g. authentication-log-table in auth-nested-stack.ts, workflow-schema-table in aws-healthomics-nested-stack.ts) is invisible to the "desired" side. Meanwhile loadCurrentGsis does recurse into nested stacks for the deployed side. A nested-stack table needing 2+ GSI changes will still hit CloudFormation's "one GSI per update" error — the exact bug this PR exists to fix — for any table not in those two hardcoded top-level stacks.

Suggestion: Parse nested-stack templates recursively for the desired side too, mirroring the recursion already done for current state via listNestedStackPhysicalIds.

Tests (1)

blocker — packages/back-end/scripts/deploy-dynamodb-gsi-waves.ts:232

The entire 418-line orchestrator has zero test coverage, including the actual mechanism this PR ships: the MAX_WAVES loop, the backup/patch/cdk deploy/restore/cleanup sequence, and — most importantly — the try/finally restore path when an intermediate cdk deploy throws. None of the failure-recovery logic is exercised.

Suggestion: Extract/inject the CloudFormation client and spawnSync call so the wave loop (including the backup-restore-on-failure path and MAX_WAVES guard) can be tested with mocks; add unit tests for the pure helpers (isStackMissingError, manifest parsing, cloneTemplates).

Breaking Changes (2)

major — .projenrc.ts:39

The PR body itself flags that GitHub OIDC role-duration-seconds (currently 3600s) may need bumping because the deploy now runs multiple sequential cdk deploy invocations, but no change in this diff actually raises it. A slow GSI backfill in CI can expire credentials mid-wave, aborting the pipeline in a half-applied state.

Suggestion: Bump role-duration-seconds in this PR, or add a safeguard so credential expiry surfaces as a contained error rather than a partial GSI rollout.

major — packages/back-end/scripts/deploy-dynamodb-gsi-waves.ts:472

Each intermediate wave deploys the full desired template (new Lambda code, IAM, etc.) with only the in-flight table's GSIs truncated. If the process aborts after wave 1 succeeds but before the final deploy runs, already-deployed application code that queries a not-yet-created index (e.g. WorkflowExternalId_Index) goes live against a table still missing it — breaking that query path in production until someone reruns the deploy.

Suggestion: Gate app code reading a new GSI behind a runtime check until the index is confirmed ACTIVE, or sequence infra-only GSI waves strictly ahead of the app-code deploy.

Simplification (3)

major — packages/back-end/scripts/deploy-dynamodb-gsi-waves.ts:195

resolveDeployEnv() is a near-verbatim copy of the same function already in preflight-deletion-protection.ts (which this script's own deploy pipeline runs immediately before it) — same branching, same error shape, different message prefix.

Suggestion: Extract to a shared scripts/lib/deploy-env.ts helper used by both scripts.

major — packages/back-end/scripts/deploy-dynamodb-gsi-waves.ts:423

The pipeline generalizes to N tables / N waves (with template backup/restore per wave) to solve a problem described as exactly two simultaneous GSI additions on one table — most of the 734 new lines serve generality the described failure mode doesn't need.

Suggestion: Unless 3+ simultaneous GSI changes are a real expected case, a much smaller script scoped to "detect >1 GSI diff, patch one index in, one intermediate deploy" would cover the described bug.

minor — packages/back-end/scripts/deploy-dynamodb-gsi-waves.ts:300

listNestedStackPhysicalIds's pagination loop and isStackMissingError's "does not exist" detection duplicate logic already written inline in preflight-deletion-protection.ts.

Suggestion: Move both into a shared scripts/lib/ helper.

PR Hygiene (2)

major — PR

Title has no ticket reference (fix: ... conventional-commit style instead of the repo's ticket-key-prefix convention).

Suggestion: Prefix with the relevant ticket key, e.g. <TICKET-ID> Apply DynamoDB GSI changes one index per stack update.

major — PR

All three "Testing*" checklist items are unchecked, yet the top-level checklist claims "All tests pass successfully and new tests added as necessary" is checked — direct contradiction given UAT hasn't run yet.

Suggestion: Uncheck the top-level "tests pass" item until staging/UAT verification is actually done, or check off the Testing steps once confirmed.

Security (0)

No findings ≥80 confidence. Caveat: the security subagent's run was flagged by an internal safety classifier — "SECURITY WARNING: This subagent performed actions that may violate security policy. Reason: Blocked by classifier. Review the subagent's actions carefully before acting on its output." — before returning its (empty) findings array. This is likely a false positive from the agent reading AWS deploy-script/credential-handling code as part of its assigned task (checking IAM scope, credential logging, table-name validation), not evidence of an actual problem in the PR. Treat the empty result with lower confidence than the other six dimensions — a manual second look at credential/IAM handling in deploy-dynamodb-gsi-waves.ts is warranted rather than taking the [] at face value.

---