DAOS-19644 cart: Add multiprovider design document by frostedcmos · Pull Request #19092 · daos-stack/daos

frostedcmos · 2026-09-16T16:16:08Z

Add multiprovider design doc

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

- Add multiprovider design doc Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

github-actions · 2026-09-16T16:16:32Z

Ticket title is 'Add multi-provider design document to github in .md format'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-19644

soumagne · 2026-09-16T20:08:39Z

You should move that doc to docs in my opinion instead of having it in the root directory. If you move it to docs/design that should not be visible in the main website unless someone adds a link to the page.

kjacque

Just a few comments.

Michael-Hennecke · 2026-09-17T11:33:08Z

You should move that doc to docs in my opinion instead of having it in the root directory. If you move it to docs/design that should not be visible in the main website unless someone adds a link to the page.

/design is the agreed location to put these, not under /docs. we don't want the designs to show up on the docs website (even if they aren't linked to the navigation bar, they would still be there...)

Michael-Hennecke · 2026-09-17T11:49:43Z

+
+## 1. Summary
+
+DAOS and CaRT currently assume a single fabric provider per deployment path. That limitation makes it difficult to run one DAOS deployment across multiple network fabrics or to support separate client populations that must reach the same servers through different providers.


"per deployment path" is confusing, this sounds like there may be multiple "deployment paths" within a single DAOS system? I believe that is impossible, and I would reword this as "per DAOS system".
And "difficult to run one DAOS deployment" should probably be "impossible to access one DAOS system"?

Michael-Hennecke · 2026-09-17T11:51:32Z

+
+This design adds initial multi-provider support with one **primary** provider and one **secondary** provider. The primary provider remains the normal server execution and server-to-server communication path. The secondary provider is introduced as a client ingress path: requests received on the secondary provider are forwarded to the appropriate primary execution target inside the engine.
+
+The result is a single DAOS deployment that can serve multiple client populations across different providers or network paths without requiring a separate DAOS instance per fabric.


DAOS deployment -> DAOS system
i would delete "without requiring a separate DAOS instance per fabric". such a thing would essentially bifurcate a single DAOS system into two disjoint ones ... we cannot have two engines (with two different fabric interfaces) access the same NVMe storage...

Michael-Hennecke · 2026-09-17T11:54:22Z

+
+*Figure 1. Example topology with a primary CXI client cluster and a secondary TCP client cluster.*
+
+![Example topology with same provider on different client clusters](multi-provider-design-doc-assets/image2.png)


texts in the images are rendering really badly - change font or increase resolution?

i will see what i can do to make them better, copilot struggled few times to convert originals from the document

Michael-Hennecke · 2026-09-17T11:56:31Z

+3. Preserve the existing primary-provider server execution model.
+4. Minimize client-side changes required to use the secondary provider.
+5. Keep the design extensible so future work can support more than two providers.
+


any goals for an admin node to talk to the servers using IP addresses that are configured on the same physical interface that's used for the secondary provider? or do we restrict admin nodes to the fabric of the primary provider (should be acceptable, just explicitly spell out the goal/non-goal)

Tagging @kjacque for this question above

The admin node talks exclusively over the control plane network, and from the control plane perspective, we use TCP over whatever route will allow us to reach an MS replica's IP. So it could in theory be routed over the secondary provider's network if that allows us to get to the MS replica. You could even bind the control plane to the secondary interface if desired. Not sure what the potential consequences of that are.

Michael-Hennecke · 2026-09-17T11:59:11Z

+CRT_PROVIDER_SECONDARY=1
+```
+
+That should also be representable through CaRT initialization options so that the role choice is not environment-variable only.


role -> provider ?

This is a bit different from the provider. You could in theory have the same provider as primary and secondary, so this would be the selector of 'primary' vs 'secondary' roles.

Michael-Hennecke · 2026-09-17T12:02:15Z

+- `port`
+
+The configuration should also define the number of secondary ingress endpoints, for example `secondary_provider_endpoints`, with an initial default of `1`.
+


it's unclear if a future extension to more than two providers would establish one primary and multiple secondary providers, or if the "ordered list" approach make the 3rd a tertiary provider, and what that difference would actually be?
implementation is OOS here, but we may want to make the wording/parameter naming more clear in such examples.

Will clarify, current thinking is that in future we might have 1 primary and multiple secondary providers, each secondary provider is identified by the unique secondary provider index. There would be no difference in treatment between secondary providers

Michael-Hennecke · 2026-09-17T12:12:59Z

+Secondary-provider support changes the DAOS data plane, not the management-plane transport.
+
+The control-plane components, including `daos_server` and `daos_agent`, continue to communicate over the management network using gRPC over TCP. All server and client nodes therefore still need access to a shared management TCP network so that the agent can communicate with the server even when the data-plane provider selected for the client is not TCP.
+


This warrants more discussion, as we cannot require the existence of a physical TCP network for management traffic that is separate from the (primary or secondary) provider network.
In scenarios where management traffic uses an "IPoverIB" TCP connection over the high-speed fabric, we need to spell out what is supported as access_points in daos_agent.yml and as hostlist in daos_control.yml (and i assume we only want to support IP addresses on the primary fabric for mgmt_svc_replicas in daos_server.yml?)

Tagging @kjacque for this as well

From the control plane perspective, all we need is a network that can access the IP addresses listed as access_points/mgmt_svc_replicas in the config. Whether that is running over the primary or secondary fabric, or some third management network, isn't really the control plane's business, as long as there is some route from each client cluster to the MS IPs.

Michael-Hennecke · 2026-09-17T12:17:18Z

+- Validate binding-bulk flows where control and data paths differ.
+
+## 6. Compatibility and Interoperability
+


we should spell out the expected incompatibilities here. my understanding is that the wire protocol changes in a way that makes it incompatible with earlier DAOS versions?

will do, the main incompatibility is in bulks, where multiprovider support requires additional data to be transferred with each bulk handle identifying 'provider index' on which such bulk was originally created.

Michael-Hennecke · 2026-09-17T12:18:50Z

+
+- Missing secondary-provider metadata.
+- Mismatched provider ordering across configuration fields.
+- Bulk transfer attempted across incompatible providers.


Mismatched provider ordering across servers/engines?

not sure if this one needs to be called out separately. with two providers this would be same as mismatch of primary providers across servers and is not multiprovider specific

Michael-Hennecke

see inline comments...

Doc-only: true Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

kjacque

My concerns have been addressed. LGTM

DAOS-19644 cart: Add multiprovider design document

47ac485

- Add multiprovider design doc Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

frostedcmos requested a review from a team as a code owner September 16, 2026 16:16

frostedcmos requested review from Michael-Hennecke, NiuYawei and kjacque September 16, 2026 16:21

kjacque reviewed Sep 16, 2026

View reviewed changes

Michael-Hennecke reviewed Sep 17, 2026

View reviewed changes

frostedcmos added 2 commits September 17, 2026 23:15

- Incorportated review feedback

3e660c2

Doc-only: true Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

- extra feedback

45812e3

Doc-only: true Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>

frostedcmos requested review from Michael-Hennecke, jgmoore-or, kjacque and soumagne September 18, 2026 15:31

kjacque approved these changes Sep 18, 2026

View reviewed changes


		## 1. Summary

		DAOS and CaRT currently assume a single fabric provider per deployment path. That limitation makes it difficult to run one DAOS deployment across multiple network fabrics or to support separate client populations that must reach the same servers through different providers.


		This design adds initial multi-provider support with one primary provider and one secondary provider. The primary provider remains the normal server execution and server-to-server communication path. The secondary provider is introduced as a client ingress path: requests received on the secondary provider are forwarded to the appropriate primary execution target inside the engine.

		The result is a single DAOS deployment that can serve multiple client populations across different providers or network paths without requiring a separate DAOS instance per fabric.


		Figure 1. Example topology with a primary CXI client cluster and a secondary TCP client cluster.

		![Example topology with same provider on different client clusters](multi-provider-design-doc-assets/image2.png)

		- `port`

		The configuration should also define the number of secondary ingress endpoints, for example `secondary_provider_endpoints`, with an initial default of `1`.

		Secondary-provider support changes the DAOS data plane, not the management-plane transport.

		The control-plane components, including `daos_server` and `daos_agent`, continue to communicate over the management network using gRPC over TCP. All server and client nodes therefore still need access to a shared management TCP network so that the agent can communicate with the server even when the data-plane provider selected for the client is not TCP.

		- Validate binding-bulk flows where control and data paths differ.

		## 6. Compatibility and Interoperability

Conversation

frostedcmos commented Sep 16, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

soumagne commented Sep 16, 2026

Uh oh!

kjacque left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Michael-Hennecke commented Sep 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michael-Hennecke Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michael-Hennecke Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michael-Hennecke left a comment

Choose a reason for hiding this comment

Uh oh!

kjacque left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Sep 16, 2026 •

edited

Loading

Michael-Hennecke Sep 17, 2026 •

edited

Loading

Michael-Hennecke Sep 17, 2026 •

edited

Loading