fix(gke): clear stale taints from active node pools

[plural-copilot]

Summary

• Reconcile the parity-designated active GKE node pool with GKE's explicit empty-taint update after module.gke completes.
• Keep platform.plural.sh/pending=upgrade:NoSchedule on the genuinely inactive/draining pool.

Incident and invariant

GKE can retain a taint as an effective taint when the module serializes an empty active-pool taint list. That leaves the active pool stale-tainted, makes trivy-system/node-collector unschedulable, and causes vulnerability scan jobs to fail. The transition invariant is now explicit: after every completed target-version/active-pool transition, the designated active pool has no taints; the inactive pool retains its protection taint.

Validation

• terraform init -backend=false and terraform validate using Docker Terraform 1.8.5 (passed; existing plural_cluster.kubeconfig deprecation warning only)
• Verified the GKE CLI supports --node-taints, including an explicit empty value
• git diff --check passed
• terraform fmt -check was run; it reports pre-existing formatting drift in five Terraform files, intentionally not included in this focused change

Rollback

Revert this commit to remove the post-transition GKE empty-taint reconciliation. The original desired configuration remains: the active pool is configured empty and the inactive pool carries the pending-upgrade taint. Reverting removes only the explicit GKE API reconciliation for stale effective taints.

Plural Stack: gke-fleet-cluster-dev

[plural-copilot]

This PR was generated by the codex Plural Agent Runtime. Here's some useful information you might want to know to evaluate the ai's perfomance:

Name	Details
💬 Prompt	Create one focused PR in this repository to fix the confirmed GKE blue/green node-pool stale taint defect....
🔗 Run history	View run history

[plural-copilot]

Plural has generated a plan for this PR, full details here

Plan Details

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.gke.google_container_cluster.primary has changed
  ~ resource "google_container_cluster" "primary" {
        id                                       = "projects/pluralsh-test-384515/locations/us-central1/clusters/gke-fleet-dev"
        name                                     = "gke-fleet-dev"
      ~ node_version                             = "1.34.9-gke.1322001" -> "1.34.10-gke.1328000"
        # (38 unchanged attributes hidden)

      ~ node_pool {
            name                        = "default-pool"
          ~ version                     = "1.34.9-gke.1322001" -> "1.35.7-gke.1222000"
            # (7 unchanged attributes hidden)

            # (4 unchanged blocks hidden)
        }
      ~ node_pool {
            name                        = "blue"
          ~ node_count                  = 1 -> 0
          ~ version                     = "1.34.10-gke.1236000" -> "1.34.10-gke.1328000"
            # (6 unchanged attributes hidden)

          ~ autoscaling {
              ~ min_node_count       = 1 -> 0
                # (4 unchanged attributes hidden)
            }

          ~ node_config {
              ~ effective_taints            = [
                  + {
                      + effect = "NO_SCHEDULE"
                      + key    = "platform.plural.sh/pending"
                      + value  = "upgrade"
                    },
                ]
                tags                        = [
                    "gke-gke-fleet-dev",
                    "gke-gke-fleet-dev-blue",
                ]
                # (22 unchanged attributes hidden)

                # (5 unchanged blocks hidden)
            }

            # (3 unchanged blocks hidden)
        }
      ~ node_pool {
            name                        = "green"
          ~ node_count                  = 0 -> 1
          ~ version                     = "1.34.10-gke.1236000" -> "1.35.8-gke.1036000"
            # (6 unchanged attributes hidden)

          ~ autoscaling {
              ~ min_node_count       = 0 -> 1
                # (4 unchanged attributes hidden)
            }

          ~ node_config {
              ~ effective_taints            = [
                  - {
                      - effect = "NO_SCHEDULE"
                      - key    = "platform.plural.sh/pending"
                      - value  = "upgrade"
                    },
                ]
                tags                        = [
                    "gke-gke-fleet-dev",
                    "gke-gke-fleet-dev-green",
                ]
                # (22 unchanged attributes hidden)

                # (5 unchanged blocks hidden)
            }

            # (3 unchanged blocks hidden)
        }

        # (28 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # plural_cluster.this will be updated in-place
  ~ resource "plural_cluster" "this" {
        id             = "5e1553c4-aa35-49c6-aae6-7a12551a0341"
      ~ kubeconfig     = {
          ~ token                  = (sensitive value)
            # (2 unchanged attributes hidden)
        }
        name           = "gke-fleet-dev"
        tags           = {
            "fleet" = "gke-fleet"
            "tier"  = "dev"
        }
        # (8 unchanged attributes hidden)
    }

  # terraform_data.clear_active_node_pool_taints will be created
  + resource "terraform_data" "clear_active_node_pool_taints" {
      + id               = (known after apply)
      + triggers_replace = [
          + "green",
          + "1.35",
        ]
    }

  # module.gke.google_container_node_pool.pools["blue"] will be updated in-place
  ~ resource "google_container_node_pool" "pools" {
        id                          = "projects/pluralsh-test-384515/locations/us-central1/clusters/gke-fleet-dev/nodePools/blue"
        name                        = "blue"
      ~ version                     = "1.34.10-gke.1328000" -> "1.34"
        # (10 unchanged attributes hidden)

        # (6 unchanged blocks hidden)
    }

  # module.gke.google_container_node_pool.pools["green"] will be updated in-place
  ~ resource "google_container_node_pool" "pools" {
        id                          = "projects/pluralsh-test-384515/locations/us-central1/clusters/gke-fleet-dev/nodePools/green"
        name                        = "green"
      ~ version                     = "1.35.8-gke.1036000" -> "1.35"
        # (10 unchanged attributes hidden)

        # (6 unchanged blocks hidden)
    }

Plan: 1 to add, 3 to change, 0 to destroy.

[plural-copilot]

Plural AI has generated a summary of what this plan entails here

Plan Summary

Summary

This plan reconciles the existing regional GKE cluster gke-fleet-dev in project pluralsh-test-384515 and updates its blue/green node-pool strategy. It will add a one-time terraform_data action to clear the upgrade taint from the active green pool, update the Plural cluster record's sensitive kubeconfig token, and set the node-pool Kubernetes minor versions to blue=1.34 and green=1.35. There are no resource destructions or cluster replacements.

Blast Radius

The scope is limited to the existing development GKE cluster gke-fleet-dev and its two worker pools in us-central1; no production environment or unrelated projects are shown. The main operational impact is a GKE node-pool version reconciliation, which can trigger node upgrades/replacements and pod eviction/rescheduling according to GKE's upgrade behavior. blue is configured as the drained pool with minimum size 0, while green is the active pool with minimum size 1 and no scheduling taint. Applications using only blue capacity, lacking disruption tolerance, or incompatible with Kubernetes 1.35 could experience interruption. The local command also requires the Terraform runner to have permission to update GKE node-pool taints.

Critical Systems

• The regional GKE cluster gke-fleet-dev in us-central1, including workloads scheduled on its node pools.
• The Plural cluster registration/control-plane integration, because its kubeconfig token is updated.
• Any workloads that depend on the blue/green node-pool labels, taints, capacity, or Kubernetes version compatibility.

Notable Changes

• 1 resource created: terraform_data.clear_active_node_pool_taints, which runs a local gcloud beta container node-pools update command to remove all node taints from the active green pool.
• 3 resources updated in place: the plural_cluster.this kubeconfig token, the blue node pool version (1.34.10-gke.1328000 to 1.34), and the green node pool version (1.35.8-gke.1036000 to 1.35).
• The refreshed state shows out-of-band changes already occurred: blue is at zero nodes and tainted platform.plural.sh/pending=upgrade:NoSchedule, while green is at one node and has had its upgrade taint removed. Applying will make Terraform's recorded configuration consistent with that blue/green transition.
• 0 destroys: the cluster, network, node pools, and other listed infrastructure are not being deleted or replaced.

Safety Assessment

Conditionally safe to apply, but verify before approval. The plan has a relatively contained blast radius and no destroys, and it appears to complete a deliberate blue/green upgrade to Kubernetes 1.35 on the active green pool. However, Terraform explicitly detected out-of-band changes, and the plan will reconcile node-pool versions and run an imperative gcloud command outside normal resource management. Before applying, confirm that the displayed drift is intentional, that green is healthy and has sufficient capacity, that workloads tolerate rescheduling and Kubernetes 1.35, and that the Terraform identity can run the taint-clearing command. Also confirm the sensitive Plural kubeconfig-token update is expected. Do not apply blindly if the out-of-band node scaling or pool transition was an emergency/manual operation.