fix: apply dynamodb gsi changes one index per stack update

[Ignacio Nistal (ignacionistal)]

fix: apply DynamoDB GSI changes one index per stack update

Type of Change*

• New feature
• Bug fix
• Documentation update
• Refactoring
• Hotfix
• Security patch
• UI/UX improvement

Summary

• UAT from staging failed because CloudFormation tried to add two GSIs (PollStatus_Index and WorkflowExternalId_Index) to the existing laboratory-run-table in one update. DynamoDB allows only one GSI create or delete per UpdateTable.
• pnpm run deploy now runs deploy-dynamodb-gsi-waves before the final cdk deploy. If an existing table would change more than one GSI, it applies those indexes one CloudFormation update at a time, then the normal deploy lands the last remaining index.
• Environments that already have the desired indexes (or only need one more) are a no-op. Brand-new tables are unchanged (CreateTable may define many GSIs).

Deploy

You only start one pnpm build-and-deploy or one CI job. That job may run two sequential CloudFormation updates internally (one GSI per update). You do not need to re-trigger CI.
The job will take longer because each GSI must become ACTIVE before the next update. If GitHub OIDC hits the 1-hour role limit during backfill, bump role-duration-seconds.

Testing*

• Merge to staging and let cicd-release-quality-uat run once
• Confirm logs show an intermediate GSI wave (add PollStatus_Index) then the final deploy adding WorkflowExternalId_Index
• Confirm laboratory-run-table ends with both new indexes ACTIVE
• Confirm new tables (laboratory-s3-access-table, laboratory-data-tagging-table, workflow-run-preset-table) still create successfully
• After UAT is unblocked, get this onto development as well so quality/prod pick up the same guard

Impact

• Deploy time: a UAT (or any existing-env) rollout that adds two GSIs will run two CloudFormation updates in one job, so the back-end deploy will take longer. Each GSI must reach ACTIVE before the next update. Later deploys that do not change GSIs are unchanged.
• Runtime: no application behaviour change. Both indexes were already in the table definition; this only makes CloudFormation apply them legally.
• Dependencies: none. No new packages.
• Other environments: no-op if the table already has the desired GSIs (e.g. quality, if those indexes were applied one at a time). Fresh environments still create the table with all GSIs in one CreateTable.

Additional Information

• This is an AWS/DynamoDB UpdateTable limit, not a bug in our table schema. CloudFormation cannot add two GSIs in a single stack update on an existing table.
• The Node 20 AWS SDK warning in the UAT logs is unrelated and was not the failure.
• The failed UAT stack rolled back to UPDATE_ROLLBACK_COMPLETE; the next deploy can proceed.
• GitHub OIDC role-duration-seconds is currently 3600. If GSI backfill is slow, that session may expire mid-job and will need a bump.
• After UAT is unblocked, this should also land on development so quality/prod get the same guard for future multi-GSI merges.

Checklist*

• No new errors or warnings have been introduced.
• All tests pass successfully and new tests added as necessary.
• Documentation has been updated accordingly.
• Code adheres to the coding and style guidelines of the project.
• Code has been commented in particularly hard-to-understand areas.