DAOS-19616 placement: derive CURRENT layouts from PRE and POST - #19088
Conversation
Generating CURRENT directly can mis-handle cascading shard relocation. Build POST first and generate PRE only when transition state affects placement. Derive CURRENT from both layouts while preserving readable PRE targets. Add POST peers for drain, reintegration, and collateral relocation. Use POST directly for failed PRE targets and unaffected objects. Handle missing rebuild spares without invalid target lookups. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
|
Ticket title is 'FIO crc mismatch detected on 4x node IB config while running HSN cable pulls test' |
github-actions
Bot
added
the
priority
|
Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19088/1/testReport/ |
|
Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19088/1/testReport/ |
The derived CURRENT layout keeps the readable PRE_REBUILD target of a
shard and adds the POST_REBUILD target as a write-only peer, so the
same target can legitimately be the readable primary of one shard and
the rebuild destination of another shard of the same object while an
extension is in flight: PRE remaps the shard from a reintegrating
target to a spare, and POST, placing on the extended map, picks the
same spare for a second, failed shard. The old code produced the same
pair, with the destination hidden as a peer behind a bogus primary.
PLACEMENT 18 now rejects readable duplicates only.
Port the placement UTs which pin the relocation behaviour:
no_stale_read_source cascading relocation on a spare
extension_preserves_read_sources NEW / new-UP extension, RP and EC
classes, one or two groups, v1/v2
no_hidden_peer_source every migration destination is
written, every readable target
was written before the failure
relocation_peer_alloc_failure pl_map_extend() allocation failure
is propagated (fault injection)
They assert that every healthy PRE_REBUILD read source stays in the
CURRENT write set, that no target is readable twice in a group and
that the single group layout matches. Test pool maps now give targets
fseq 1, as gen_pool_buf() does, so that a reintegrated target is not
mistaken for a freshly added one.
Build and run from the repository root:
scons --config=force --jobs=16 \
install/bin/jump_pl_map install/lib64/libdaos.so
./utils/run_utest.py --suite_filter='^placement$' --sudo=no
All 33 placement cases pass on top of the derived CURRENT layout with
no further placement change. Native sweeps over 985,600 random and
120,000 exhaustive/extension cases found no PRE read source missing
from the CURRENT write set.
Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
|
Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19088/1/execution/node/1266/log |
| }; | ||
|
|
||
| static bool | ||
| comp_may_change_layout(const struct pool_component *comp) |
There was a problem hiding this comment.
If the PR is refreshed, it may be worth adding the following comment. Returning false for DOWNOUT does not mean its remapped spare cannot differ between PRE and POST; another DOWN , DRAIN , or UP target can still change the remap sequence. This takes me sometime to think.
/*
* Return true if this component can make PRE_REBUILD and POST_REBUILD
* diverge.
*
* Returning false for DOWNOUT does not mean that a shard remapped from a
* DOWNOUT target must have the same spare in both layouts. Its spare may
* still change when another DOWN, DRAIN, or UP target affects the remap
* sequence. Once such a transitional component is encountered, the complete
* PRE_REBUILD and POST_REBUILD layouts, including DOWNOUT shards, must be
* compared.
*/
| D_ASSERT(post_affected == NULL || gen_mode == POST_REBUILD); | ||
| layout->ol_ver = allow_version; | ||
| if (post_affected != NULL) | ||
| *post_affected = false; |
There was a problem hiding this comment.
minor, post_affected the caller already init as false, so seems need not set it again here.
| static bool | ||
| comp_may_change_layout(const struct pool_component *comp) | ||
| { | ||
| return comp->co_status != PO_COMP_ST_UPIN && comp->co_status != PO_COMP_ST_DOWNOUT; |
There was a problem hiding this comment.
When I read the code, was think about PO_COMP_ST_NEW status.
By checking some code, NEW cannot reach this func because it should be already filtered out.
But maybe it could be a little bit clearer to write as -
return comp->co_status == PO_COMP_ST_UP || comp->co_status == PO_COMP_ST_DOWN || comp->co_status == PO_COMP_ST_DRAIN;
There was a problem hiding this comment.
Yes, I think we need to have some cleanups for the master version of this patch, for example, PL_HAS_PEER can be removed.
Let's keep the 2.8 version as now and see if it can pass CI.
6 participants
Generating CURRENT directly can mis-handle cascading shard relocation. Build POST first and generate PRE only when transition state affects placement. Derive CURRENT from both layouts while preserving readable PRE targets. Add POST peers for drain, reintegration, and collateral relocation. Use POST directly for failed PRE targets and unaffected objects. Handle missing rebuild spares without invalid target lookups.
Steps for the author:
After all prior steps are complete: