flannel migration: document removing leftover flannel iptables rules#2768
flannel migration: document removing leftover flannel iptables rules#2768stitrace wants to merge 1 commit into
Conversation
The live-migration controller removes the flannel daemonset and deletes the flannel network devices, but it does not remove the iptables chains flannel programs (FLANNEL-POSTRTG in nat, FLANNEL-FWD in filter). These survive the migration and the FLANNEL-POSTRTG masquerade rule keeps SNAT-ing cross-node pod-to-pod traffic to the node tunnel IP, which silently breaks NetworkPolicy after migration. Add a cleanup step (with a reboot alternative) so operators can remove the leftover rules.
✅ Deploy Preview for calico-docs-preview-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview succeeded!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
@stitrace Thanks for the submission. Someone will take a look at this together with projectcalico/calico#12920 |
|
Is this an alternative to projectcalico/calico#12920 ? With the automated removal in the linked PR, presumably we don't need manual steps? |
|
Not quite an alternative — they're complementary. #12920 (projectcalico/calico#12920) automates the cleanup for migrations performed with a Calico version that includes the fix, so going forward the manual steps won't be needed. However, the leftover chains don't clean themselves up: any cluster that already migrated with an earlier version still has the stale Flannel iptables rules sitting there, and anyone migrating on an older release will keep hitting this. For those users the docs are the only discoverable fix. I'd suggest trimming this PR down to a short note along the lines of: "migrations performed with Calico vX.Y+ remove these rules automatically; if you migrated with an earlier version, clean them up manually as follows" — with the version filled in once the fix ships in a release. Happy to rework the PR that way if it sounds good. |
|
Got it. Also, just a thought - for a lot of folks simply doing a rolling update of cluster nodes will be an easier way to handle this rather than scripting running these commands across a whole cluster. Might be worth taking that approach? |
3 participants
Description
The live-migration guide tells you to delete the migration controller once migration completes, but it does not mention that flannel leaves iptables rules behind on every node.
The migration controller removes the flannel daemonset and deletes the flannel network devices (
flannel.<vni>,cni0), but it does not remove the iptables chains flannel programs:FLANNEL-POSTRTG(nat) andFLANNEL-FWD(filter). These survive the migration.The masquerade rule in
FLANNEL-POSTRTGkeeps SNAT-ing cross-node pod-to-pod traffic to the node's tunnel IP. This is invisible until you useNetworkPolicy: the SNAT'd source no longer matches pod-selector rules, so Calico's default-deny drops the traffic. Symptom after an otherwise successful migration: cross-node connections to policy-selected pods silently time out, while same-node traffic keeps working.This PR adds a cleanup step after "Delete the migration controller" with an idempotent flush command (legacy + nft backends) and a reboot alternative.
Notes
calico/...); maintainers may want to backport to the versioned snapshots.Reproduction
After migrating, on any node:
nft list chain ip nat POSTROUTINGstill showsjump FLANNEL-POSTRTGwith non-zero counters; a cross-node listener sees the client's source as the sender node's tunnel IP (<block>.0) rather than the pod IP, and NetworkPolicy-selected pods become unreachable cross-node.