Repository navigation
Conversation
Add a GitHub Actions OIDC provider and trust the deployer role from the terraform-test environment, so CI assumes it without AWS access keys. Stop managing the DNS-account role in bootstrap. It is maintained by hand in account 011713309463 and referenced by ARN, so bootstrap no longer needs credentials outside the application account. Add a workflow that plans iac/app on pushes to dev and applies only on a manual admin-triggered run.
Plan test infrastructure from the iac branch as well as dev, so the workflow can be exercised before it reaches the default branch.
Drop the workflow_dispatch condition on the apply step so pushes to iac deploy directly. Restore the admin-gated condition before merging to dev.
Split the workflow into a plan job and an apply job. The plan job writes the plan to the run summary and uploads it, and the apply job applies that saved artifact behind the terraform-test environment, so reviewers approve the plan they have read. Plans run in a separate terraform-plan environment without reviewers, and the deployer trust policy accepts both. Scope the deployer's IAM role actions to the application role prefix and service-linked roles instead of every role in the account. Move bootstrap state into the existing S3 bucket now that it exists. Remove the footer donate link, which opened a modal that does not exist.
Give GitHub Actions its own OIDC deployer role limited to the application stack, and require a permissions boundary on every role the stack creates. The existing deployer role is now for local applies only.
Rename the workflow to terraform-deploy, run it only on pushes to iac, assume tcf-github-terraform-deployer, and ship the Lambda zip built during plan with the plan artifact so apply can use it.
Run the Terraform plan and apply pipeline as tcf-terraform-deployer through GitHub OIDC, so CI uses the role the DNS account already trusts. Rename the GitHub role to tcf-github-deployer and scope it to code deploys: push iac-test images, register task definitions, run release tasks, and update iac-test services. aws.yml now assumes it through OIDC from the aws-deploy environment and targets the iac-test stack instead of using access keys. Require the iac-test-role-boundary permissions boundary on every role tcf-terraform-deployer creates or changes, limit PassRole to ECS tasks and Lambda, and deny removing role boundaries.
Run aws.yml on pushes to master and iac instead of after the CI workflow completes. workflow_run jobs always run as the default branch with its copy of the workflow, which the prod environment rejects. The ci job now calls ci.yml against the pushed commit, and the deploy job waits for approval in the prod environment before deploying, mirroring the Terraform pipeline. Assume tcf-github-deployer in the account from the prod environment's AWS_ACCOUNT_ID variable, and trust the prod environment from bootstrap.
Ignore task_definition changes on the ECS service so a Terraform apply no longer rolls the service back to the test image after a code deploy. Terraform still manages the task definition family, which code deploys copy when they register a new revision.
Make iac-test-app tags immutable except buildcache, so a commit SHA always points at the image built for it while the build cache tag can still move. Expire untagged images after a day and keep the newest 10 tagged images. Skip the image build in aws.yml when the commit SHA is already in ECR, so re-running a deploy reuses the image instead of failing on the immutable tag, and let tcf-github-deployer describe images for that check.
Saved plans embed a full copy of state, so the tfplan artifacts on this public repository exposed every secret in state to anyone signed in to GitHub. The plan job now stores the plan and Lambda zip under plans/<run_id>/ in the state bucket, and the apply job reads them from there and deletes them afterwards. Encrypt the state bucket by default with a dedicated KMS key using S3 Bucket Keys, deny requests that do not use TLS, expire old app state versions after 30 days, and expire saved plans after a day. Drop encrypt = true from the backends, since without a KMS key it makes Terraform write SSE-S3 and override the bucket default.
EC2 rejects apostrophes in security group rule descriptions, which failed the apply that creates the CloudFront VPC origin ingress rule.
Legacy prod regularly bursts to 16 ACU and averages 2-4 ACU through enrollment, which a db.t3.micro cannot serve. Replace the RDS instance with an Aurora PostgreSQL 18.6 Serverless v2 cluster that scales from 0 to 16 ACU and pauses after five idle minutes, and point the database credentials secret at the cluster writer endpoint.
The web service ran a single task in one subnet, so enrollment traffic had nowhere to go. Track average CPU and memory to scale between ecs_min_count and ecs_max_count tasks, spread tasks across both public subnets, and leave desired_count to the autoscaler.
The release task ran invalidate_cachalot tcf_website, which empties the query cache for every table and sends all traffic to the database at once. During enrollment that load spike took the site down for 30-60 seconds per deploy. Cachalot already invalidates each table a migration changes, since CACHALOT_INVALIDATE_RAW catches the ALTER, CREATE, DROP and data statements that migrate runs, and the release task shares the web tasks' Redis cache.
This reverts commit e6c42b4, "feat(iac): switch the database to Aurora Serverless v2". The November 2025 outage came from exhausted database connections behind RDS Proxy, not from fixed instance capacity, and Aurora never paused under steady bot traffic, so it costs more off-peak.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves production onto Terraform-managed AWS infrastructure (
iac/), deployed from GitHub Actions through OIDC roles instead of static AWS keys, and adds scheduled database dumps to S3.Infrastructure (
iac/app)tcf-prod(project_name=tcf,environment=prod)tcf-terraform-state-099933383052) with KMS encryption and lockfileorigin.<domain>record; ACM cert gains it as a SAN for CloudFront → ALB TLStask_definition/desired_countso code deploys and autoscaling own themAWS_ELB_URLandAWS_CLOUDFRONT_URLenv varsbuildcache)@virginia.eduemailstcf-prod-role-boundarypermissions boundarypg_dumptask → encrypted S3 bucket (cron(0 4 * * ? *)America/New_York)terraform applyno longer shows a diff every runBootstrap (
iac/bootstrap, new)Separate Terraform project that creates:
tcf-terraform-deployer, the Terraform apply roletcf-github-deployer, the code deploy role, scoped to ECR/ECStcf-prod-role-boundarypermissions boundarySee
iac/bootstrap/README.mdfor setup.CI/CD
terraform-deploy.yml(new): plan → upload saved plan to S3 → apply that exact planterraform-plan/terraform-prodenvironmentsaws.ymlmaster,iac) and runs CI as a reusable workflowprodenvironment with a concurrency locktcf-github-deployervia OIDC (no more access-key secrets)tcf-prod-*cluster/service/repo/task familyinvalidate_cachalot(stops flushing the whole query cache on every deploy)ci.yml: addsworkflow_callApp / local dev
Dockerfile, gunicorn bind, compose port mapping)aws-cliinstead ofmcscripts/prod_dump.shrewritten--latest.env.example.gitignoreignores Terraform state,.terraform/, and*.tfvarsNotes
tcf-terraform-dnsin the DNS account is managed by hand and is not in Terraform.