SOLR-11431: Report 503 instead of 500 for a core that fails to initialize

[nick-boss-tech]

🤖 AI text below 🤖 (posted on behalf of Nick Shanin)

https://issues.apache.org/jira/browse/SOLR-11431

What happens today

A core that fails to initialize answers requests with a 500: SolrCoreInitializationException is built with SERVER_ERROR. During leader election, PeerSync treats that 500 as a failed version request, so a replica whose core never came up can fail the election for its shard. The election is built to tolerate a replica that cannot answer: SyncStrategy creates the election PeerSync with cantReachIsSuccess = true, so an unreachable replica counts as success and is reconciled later by recovery. A core that failed to initialize still has its index and transaction log on disk and may hold the most up-to-date copy; it is simply another replica that cannot answer right now, and a 404 from it is already tolerated. The status is also wrong in itself: the core is unavailable, not mishandling the request.

What this change does

• SolrCoreInitializationException now reports 503 (SERVICE_UNAVAILABLE) instead of 500. A stale comment in CoreContainer is corrected to match, and the three TestCoreContainer expectations move from 500 to 503.
• PeerSync needs no new tolerance: it already counts a 503 or 404 answer to a GET_VERSIONS request as success during leader election when cantReachIsSuccess is set. The only PeerSync edit extracts that existing check as the test-visible predicate isToleratedLeaderElectionException, with the same conditions and the same truth table, and PeerSyncLeaderElectionTest pins the predicate directly. A 500 still fails, and failures on GET_UPDATES requests still fail.
• A new PeerSyncFailedCoreResponseTest drives real ShardResponse objects through PeerSync.handleResponse (the response class is final, so it is built, not mocked): the failed-core 503 and the 404 count as success, the 500 does not, and nothing is tolerated for GET_UPDATES or with cantReachIsSuccess off. Making that wiring test possible is the reason handleResponse becomes package-visible and two ShardResponse setters (setException, setShardAddress) become public: the test has to construct real responses and pass them through the handler.

Proof

Verified at head 84a2aa8 on 2026-10-04.

• TestCoreContainer: 25 tests, 0 failures, 3 skipped at this head, including the three expectations that move from 500 to 503.
• Base comparison: the same TestCoreContainer against base production code fails in testCoreInitFailuresFromEmptyContainer and testCoreInitFailuresOnReload with "expected:<503> but was:<500>", so the status change itself has a fails-on-base proof.
• PeerSyncFailedCoreResponseTest: 5 of 5 pass. A fails-on-base run is not possible for the new PeerSync tests: they exercise isToleratedLeaderElectionException and a test-visible handleResponse, both added by this PR, so they do not compile against the base tree.
• The branch's earlier validation also ran PeerSyncLeaderElectionTest and TestLazyCores green, with tidy and Error Prone clean.

Limits

• The 503 change applies wherever SolrCoreInitializationException surfaces, and other consumers change behavior with it. SolrCmdDistributor.checkRetry retries a forwarded update on 404, 403, or 503, so an update forwarded to a replica whose core failed to initialize, a non-retried 500 on base, is now retried. CloudSolrClient treats a 503 like a 404 in its stale-state retry and expires the cached collection state, so a request routed to a failed-init core now triggers that refresh where the 500 did not. Both fit an unavailable core, but they are behavior changes and are stated here rather than left to be found.
• The most operator-visible of these is diagnostic. ResponseUtils.getErrorInfo attaches the stack trace to the error body only for code 500 (or below 100) and logs "500 Exception" at ERROR only for those codes, so a request to a core that failed to initialize now returns the message without the trace and is not logged at ERROR there. The message itself still names the init failure and its cause.
• No end-to-end test shows a core that failed to initialize answering a /get version request with 503. The PeerSync tests construct the exception and the responses locally; the status mapping is covered at the container level by TestCoreContainer. The election failure described above is likewise a reading of the code, not a reproduced one: no test, on base or at this head, elects a leader with a failed core in the shard.
• Interaction with SOLR-12998: Return 503 for recovery on a still-loading core and retry only that response #5010 (SOLR-12998): that retry matches a 503 whose message contains the still-loading suffix (" is still loading"), not any 503. A failed-init core's message is "not available due to init failure: " followed by the underlying cause, which does not carry that suffix on its own, so this 503 does not trigger the retry today; a cause whose own text happened to contain the suffix would be retried. If that matching ever loosens to any 503, the two changes would interact.
• The tolerance stays narrow: version requests only, and only under cantReachIsSuccess.

Changelog: changelog/unreleased/SOLR-11431.yml

AI assistance

AI agents assisted with research, implementation, review, and drafting. Nick Shanin directed the work and takes responsibility for this contribution.