Skip to content

DAOS-19087 pool: Recov conts in all reints/extends - #19078

Open
liw wants to merge 2 commits into
release/2.8from
liw/always-recover-conts-2.8
Open

liw wants to merge 2 commits into
release/2.8from
liw/always-recover-conts-2.8

Conversation

@liw

@liw liw commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

To avoid leaking container children during reintegration, perform
container recovery in all reintegration modes. Not addressed by this
simple change is the extra delay for a POOL_REINT request.

Fix the backoff-less retrying of -DER_AGAINs from container recovery. It
has been seen to generate 11 million log messages within 2 minutes.

Fix an ilog_modify log message that can't be controled by DD_MASK,
making it unnecessarily harder to turn on debug messages.

Fix segfault

#0  crt_hg_bulk_transfer at src/cart/crt_hg.c:2253
#1  crt_bulk_transfer at src/cart/crt_bulk.c:306
#2  ds_pool_recov_cont_handler at src/pool/srv_target.c:3203
#3  crt_handle_rpc at src/cart/crt_rpc.c:1756

that happens when reintegrating a target on the PS leader, likely caused
by NULL crt_rpc_priv.crp_hg_hdl. The approach is to use the existing
bulk mechanism offered by CaRT for CoRPCs. (The CaRT bug with local bulk
transfers is not addressed by this fix.)

Restore the pre-#18651 behavior that aborts a pool reintegrate operation upon unknown target.

Test-tag: pr rebuild dmg_negative_test

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

liw added 2 commits September 16, 2026 08:58
To avoid leaking container children during reintegration, perform
container recovery in all reintegration modes. Not addressed by this
simple change is the extra delay for a POOL_REINT request.

Fix the backoff-less retrying of -DER_AGAINs from container recovery. It
has been seen to generate 11 million log messages within 2 minutes.

Fix an ilog_modify log message that can't be controled by DD_MASK,
making it unnecessarily harder to turn on debug messages.

Fix segfault

    #0  crt_hg_bulk_transfer at src/cart/crt_hg.c:2253
    #1  crt_bulk_transfer at src/cart/crt_bulk.c:306
    #2  ds_pool_recov_cont_handler at src/pool/srv_target.c:3203
    #3  crt_handle_rpc at src/cart/crt_rpc.c:1756

that happens when reintegrating a target on the PS leader, likely caused
by NULL crt_rpc_priv.crp_hg_hdl. The approach is to use the existing
bulk mechanism offered by CaRT for CoRPCs. (The CaRT bug with local bulk
transfers is not addressed by this fix.)

Signed-off-by: Li Wei <liwei@hpe.com>
Restore the pre-#18651 behavior that aborts a pool reintegrate operation
upon unknown target.

Test-tag: pr rebuild dmg_negative_test
Signed-off-by: Li Wei <liwei@hpe.com>
@github-actions

Copy link
Copy Markdown

Ticket title is 'Rebuild and container IV activities may result in orphan container children'
Status is 'In Progress'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-19087

@github-actions github-actions Bot added the priority Ticket has high priority (automatically managed) label Sep 16, 2026
@liw liw added the clean-cherry-pick Cherry-pick from another branch that did not require additional edits label Sep 16, 2026
@liw
liw marked this pull request as ready for review September 16, 2026 01:56
@liw
liw requested review from a team as code owners September 16, 2026 01:56
@liw
liw requested review from Nasf-Fan and liuxuezhao September 16, 2026 01:57
@liw

liw commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

We'll also need to add a backport of #19106 here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clean-cherry-pick Cherry-pick from another branch that did not require additional edits priority Ticket has high priority (automatically managed)

Development

Successfully merging this pull request may close these issues.

3 participants