Repository navigation
internal/db: Close incoming dqlite connections after a failed bootstrap or join - #856
Open
claudiubelu wants to merge 5 commits into
Open
claudiubelu wants to merge 5 commits into
claudiubelu wants to merge 5 commits into
Conversation
…tstrap or join When Bootstrap or Join fails, the reverter closes the dqlite app but keeps the accept channel open. The go-dqlite reader ranging over it keeps receiving connections from the remaining cluster members and panics when it cannot dial the closed app's local socket, crashing the daemon before the join failure cleanup can run. Rotate the accept channel before closing the dqlite app, like Restart already does. Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com>
…ffline
On a failed join, the cluster member handling the cleanup resets the
joiner through PUT /cluster/{name}.
That endpoint required an open database, which a joiner whose dqlite
join failed never has, so the reset was refused and the joiner could
not be initialized again until its daemon was restarted.
Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com>
Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com>
…led bootstrap or join After a failed bootstrap or join, the reverter closes the dqlite app and leaves a fresh accept channel without a consumer. Accept then holds every incoming dqlite connection for 10s before closing it, as it does during a Restart, although no dqlite app will consume it. If the failed joiner remains in the dqlite cluster, e.g. because its join failure cleanup did not remove it, the remaining members wait on it while probing for the dqlite leader. The membership consistency check probes one member at a time within 5s, so it fails with "no available dqlite leader server found" whenever the failed joiner is probed first, blocking token generation, joins and removals. Heartbeats are delayed by 10s as well. Close the accept channel without replacing it after a failed bootstrap or join, so that Accept closes incoming connections right away, and create a new one on the next Bootstrap or Join. Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com>
DialDqlite only applies the context to the TCP and TLS dial. The HTTP upgrade request that follows waits for the response without any limit, so a remote member that never answers it, e.g. because it has no dqlite app, blocks the dial past the dqlite leader probe and dial timeouts. Set the connection deadline once the context is done, so that the upgrade is aborted as well. Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After a failed bootstrap or join, the daemon keeps serving its HTTPS endpoint without a dqlite app. Since #850,
Acceptholds every incoming dqlite connection for 10s before closing it. That makes sense during aRestart, but here no dqlite app will ever consume the connection.If the failed joiner remains in the dqlite cluster (e.g.: its join failure cleanup was refused, see #774), the remaining members wait on it when probing for the dqlite leader:
WithConcurrentLeaderConns(1)).DialDqliteonly applies the context to the TCP/TLS dial, not to the HTTP upgrade, so the probe of the failed joiner blocks until it closes the connection 10s later.no available dqlite leader server found, and heartbeats are delayed by 10s.This PR:
Acceptcloses incoming connections right away.Restartand the daemon start-up still hold them as before.DialDqliteonce the context is done.Reproducer: bootstrap
c1, then makec2fail to join with an invalidpatch.global.sqlwhilec1is stillPENDING(without #774, the cleanup leavesc2behind). Before this PR, everytokens addonc1fails after 10.2s; with it, they succeed in ~40ms. Unit tests cover both changes; theDialDqlitetest hangs without the fix.Depends-On: #850