SOLR-13136: Create new shards in CONSTRUCTION state so queries don't fail during shard creation

[nick-boss-tech]

🤖 AI text below 🤖 (posted on behalf of Nick Shanin)

https://issues.apache.org/jira/browse/SOLR-13136

What happens today

A newly created shard becomes visible for queries before its replicas are ready, so queries routed to it during creation can fail with "no servers hosting shard".

What this change does

CreateShardCmd now creates the shard in CONSTRUCTION state, waits for its replicas to become ACTIVE, and only then flips the shard to active. If creation fails partway, the partial shard is cleaned up instead of being left behind. The cleanup re-reads the cluster state, so replicas already visible there are deleted along with the slice, instead of being left behind as orphaned cores. The cleanup also runs for more failures than before: on base it ran only when adding replicas failed with an AssignmentException, while now any exception from adding replicas, waiting for them to become active, or applying buffered updates triggers it. The cleanup is best effort: the state it reads is this node's local view, so a replica the Overseer has written but this node has not seen yet is missed, as before; DeleteShardCmd waits one minute for the replica deletions and then removes the slice regardless; and if the cleanup itself fails, that failure is caught and logged at WARN, the slice stays in CONSTRUCTION, and the caller sees only the original create error. One consequence to be aware of: if a new shard's replicas do not become active before the create's replica wait times out, the create now fails and the half-created shard is deleted, so the create must be retried, where the previous behavior activated the shard anyway even when its replicas were not ready.

For the unit test, ShardResponse.setException is now public, so the test can build a failed response; PR #5002 makes the same visibility edit.

Proof

Gate record for this branch at head e6f3f18929f: tidy clean, Error Prone compile clean, :solr:core:check -x test green. The tests below were run with Gradle (:solr:core:test) at that head, and run against base production code as well (verified 2026-10-04).

• CreateShardConstructionTest.testNewShardStaysConstructionUntilReplicasActive (1/1) watches the collection state while a shard is being created and queries throughout, asserting the shard stays in CONSTRUCTION until its replicas are ACTIVE and that no query fails with "no servers hosting shard". On base it fails because base marks the shard ACTIVE as soon as its replicas are added, before they are ACTIVE.
• CreateShardCmdTest.testReplicaWaitFailureDeletesShardInsteadOfActivating drives the replica wait into its timeout and asserts the slice is deleted rather than activated. On base it fails because base has no such wait and activates the shard anyway; the only cleanup on base ran on an AssignmentException from adding replicas, with the cluster state snapshot taken before they were added, which lists no replicas to delete. This case also fails against this PR's own previous activate-in-finally behavior, which is how the reversal described below was pinned down.
• CreateShardCmdTest.testAddReplicaFailureDeletesShardAndReplica (the class is 3/3 at this head) drives the first cleanup block, the one around adding replicas: the replica is registered in the cluster state, creating its core then fails, and the test asserts the cleanup deletes that replica and the slice, never activates the shard, and that the caller sees the original add failure. On base it fails at the replica assertion: base caught only an AssignmentException from adding replicas, this failure is not one, and the updates offered on base stop at the createshard and the addreplica, with nothing deleted.
• CreateShardCmdTest.testBufferedUpdatesFailureDoesNotFailCreate pins the buffered-updates policy described below. Its failure on base is vacuous: base never sends REQUESTAPPLYUPDATES, so the failure the test answers never occurs there, and that base result says nothing about the policy.

A choice to check

This PR takes the position that a shard whose replicas do not become active before the create's wait times out should not be activated. The new slice is deleted and the create fails, so it can be retried.

What that looks like in production: the slice being deleted was created by this same call, sat in CONSTRUCTION state the whole time, and never served a query; its cores hold at most an in-progress copy, and the collection's existing shards are untouched. A guard on whether the slice already existed means a pre-existing shard can never be deleted by this path. Deleting a half-created slice on failure is also what this command already did when adding a replica failed; this change extends that pattern to the wait that follows. The old behavior instead reported success and activated the shard with its replicas not ready, after which queries routed to it failed or returned partial results, and the broken state stayed until an operator cleaned it up by hand.

The cost of the new behavior: on a slow but healthy cluster, a create that would eventually have succeeded can time out, fail, and need a retry, where before it limped in degraded. The alternative is to activate the shard anyway so it is not left unusable in CONSTRUCTION; an earlier version of this PR implemented exactly that, and this version reverses the decision. In outcome the alternative lands close to where the old code ended up, an active shard whose replicas are not ready, though the old code had no construction state and no wait in this flow at all. Was deleting and failing the right call?

Open policy question, flagged for reviewers

Before activation, CreateShardCmd asks the new shard's leader to apply any updates buffered during construction (REQUESTAPPLYUPDATES). The core refuses that request when it is not buffering, which is expected and harmless. Today the code logs any failure of that step and continues to activation: it does not distinguish the expected not-buffering refusal from a genuine apply failure, and activation proceeds either way. The alternative is to treat a non-refusal failure like the other readiness failures in this flow and abort creation, deleting the new shard. The current code keeps the log-and-continue behavior; flagging it here so reviewers can say whether the stricter policy is wanted.

The current policy is pinned by CreateShardCmdTest.testBufferedUpdatesFailureDoesNotFailCreate, named in Proof above. One limit of that test: it answers the request with a stand-in refusal message ("Core is not buffering updates"), while the real refusal from RequestApplyUpdatesOp is "Core not in buffering state". Nothing reads the message today, but the stricter policy would have to tell the expected refusal from a genuine failure by that text.

Limits

• Coverage is the collection-create flow exercised by the tests above. Shard splits and restores do not call CreateShardCmd, so they are unaffected by this change: SplitShardCmd issues its own CREATESHARD state update and sets the new shards' state itself, and RestoreCmd builds its shards through CreateCollectionCmd and marks them CONSTRUCTION with its own update.
• The buffered-updates policy in the open question above is undecided by design; this PR keeps the current log-and-continue behavior until reviewers weigh in.
• The wait for a new shard's replicas uses a fixed 600-second timeout that callers cannot change, because the create message never carries a timeout value. That timeout is spent on two waits in a row, one for the replicas to appear and one for them to become active, so a create whose replicas never become active can block for up to 20 minutes before it fails and the shard is deleted.

Changelog: changelog/unreleased/SOLR-13136.yml (type fixed).

AI assistance

AI agents assisted with research, implementation, review, and drafting. Nick Shanin directed the work and takes responsibility for this contribution.