[multicast] viona CTRL_RX/CTRL_MAC receive filtering + softnpu egress lock scoping + dep updates by zeeshanlakhani · Pull Request #1093 · oxidecomputer/propolis

zeeshanlakhani · 2026-03-30T01:41:28Z

viona: CTRL_RX and CTRL_MAC table receive filtering

This work extends the minimal VIRTIO_NET_F_CTRL_RX support from #1125 to cover CTRL_RX promiscuity/all-multicast controls and CTRL_MAC tableset handling. Propolis records the guest-supplied unicast and multicast MAC tables. When viona interface version 7 is available (illumos-gate patchset in review), it installs the multicast table on the in-kernel device via the new VNA_IOC_SET_MAC_FILTERS ioctl. The unicast table is used to decide when the device must fall back to full promiscuity, since the kernel MAC client already carries the primary unicast address and does not install additional unicast filters.

Filter state is cleared on device reset and carried across live migration via VionaStateV1. The payload is exported unconditionally, matching the import behavior on master since #1125, which takes the payload without consulting negotiated features. Import still tolerates a missing payload from sources predating VionaStateV1. The device narrows to classified delivery once the driver's first MAC_TABLE_SET since reset is accepted (empty or not). Until then, the all-multicast lower bound holds, because some drivers, including illumos vioif, negotiate CTRL_RX but never actually manage a filter table. That marker is carried across migration as an optional multicast_table_managed field, where older sources omit it (the target infers it from non-empty tables) and older targets ignore it. Table installation is tracked independently of promiscuity, so guest promiscuity toggles leave installed classified flows in place rather than unplumbing and replumbing them accordingly.

On falcon links attached to softnpu, promiscuity is pinned at construction rather than derived from guest filter state. A host without ALL_VLAN support pins to full promiscuity, because classified unicast delivery would drop transit frames for the emulated fabric. Other falcon VMs are no longer forced promiscuous.

softnpu: release pipeline lock before egress I/O

Every port worker and the guest Tx path serialize on the pipeline mutex. Previously, the guard was held across both the dlpi and virtio egress, meaning that one worker blocked in egress I/O stalled the others. This is especially costly for multicast, where a single input frame fans out to several ports for replication.

This change scopes the lock to pipeline evaluation (only). The egress packets borrow from the input frame rather than the pipeline, so they remain valid after the guard is dropped. Lock acquisition moves from the callers into process_external_packet and process_guest_packet, which now take the mutex directly.

deps and callouts

softnpu is pinned to its zl/multicast branch pending upstream.
oxide-tokio-rt is bumped to match omicron.
Kernel-side filter installation uses viona interface version 7 support as per illumos-gate change 775 (stlouis#986, in review). Without it, the device remains at its all-multicast fallback.
The headers ref is temporarily pinned to Gerrit change 775 (refs/changes/75/775/4) so that the viona-api header-check can verify the new VNA_IOC_SET_MAC_FILTERS FFI definitions (against the new viona version). We need to restore to stlouis once 775 integrates.

This wires up the softnpu ASIC backend to support multicast end-to-end by translating DPD's sidecar.p4 table operations into sidecar-lite.p4's simplified P4 pipeline. ## AsicMulticastOps We replace the stubbed AsicMulticastOps implementation (which returned "OperationUnsupported" for group creation and port addition with in-memory group tracking via McGroupData, following the tofino_stub pattern. Group membership is used by the table translation layer to build port bitmaps for sidecar-lite's Replicate extern. Ports >= 128 are rejected at add time to match sidecar-lite's 128-bit bitmap width. ## Table translation (asic/src/softnpu/table.rs) We map sidecar.p4 table names to sidecar-lite equivalents and translate action parameters where the designs differ for emulation. All multicast action arms are gated with #[cfg(feature = "multicast")]. ## References - [softnpu #183](oxidecomputer/softnpu#183) - [propolis #1093](oxidecomputer/propolis#1093) - [p4rs #240](oxidecomputer/p4#240) - [sidecar-lite #152](oxidecomputer/sidecar-lite#152) - tokio: 1.50 (due to softnpu) - oxide-tokio-rt: 0.1.3 (following-up from tokio's move to 1.50)

This wires up the softnpu ASIC backend to support multicast end-to-end by translating DPD's sidecar.p4 table operations into sidecar-lite.p4's simplified P4 pipeline. ## AsicMulticastOps We replace the stubbed AsicMulticastOps implementation (which returned "OperationUnsupported" for group creation and port addition with in-memory group tracking via McGroupData, following the tofino_stub pattern. Group membership is used by the table translation layer to build port bitmaps for sidecar-lite's Replicate extern. Ports >= 128 are rejected at add time to match sidecar-lite's 128-bit bitmap width. ## Table translation (asic/src/softnpu/table.rs) We map sidecar.p4 table names to sidecar-lite equivalents and translate action parameters where the designs differ for emulation. ## Fix update semantics We fix an issue with stale in-memory sources after update with absent source list(s). `modify_group_external` wrote back `new.sources.or(existing)` to the in-memory record while `update_external_tables` in the same call canonicalized `new.sources` directly to rewrite the P4 source-filter entries. A `None` update therefore cleared the P4 table but left a stale `Some([...])` in memory. All multicast action arms are gated with #[cfg(feature = "multicast")]. ## References and Deps - [softnpu #183](oxidecomputer/softnpu#183) - [propolis #1093](oxidecomputer/propolis#1093) - [p4rs #240](oxidecomputer/p4#240) - [sidecar-lite #152](oxidecomputer/sidecar-lite#152) - tokio: 1.50 (due to softnpu) - oxide-tokio-rt: 0.1.3 (following-up from tokio's move to 1.50) - bump oxide-tokio-rt to 0.1.4, tokio to 1.52.1 (matching Omicron)

Pin softnpu and p4rs to their zl/multicast branches for multicast support in the emulated ASIC, and bump oxide-tokio-rt to 0.1.4.

This work extends the minimal `VIRTIO_NET_F_CTRL_RX` support from [1125](#1125) to the full control-queue receive filtering surface, including promiscuity levels and guest-supplied unicast/multicast MAC tables being applied to the in-kernel device via the new [VNA_IOC_SET_MAC_FILTERS](https://code.oxide.computer/c/illumos-gate/+/775) ioctl. Filter state is cleared on device reset and carried across live migration via `VionaStateV1`. The payload is exported unconditionally, matching the import behavior on master since #1125, which takes the payload without consulting negotiated features. Import still tolerates a missing payload from sources predating VionaStateV1. The device narrows to classified delivery only once the guest installs a non-empty multicast table. An empty table holds the device at its all-multicast default, where some drivers (illumos vioif among them) negotiate `CTRL_RX` but never manage a filter table, and narrowing on an empty table would silently drop their multicast. The virtio spec permits the resulting unrequested traffic, but not the loss. On falcon links, a host without ALL_VLAN support now falls back to full promiscuity rather than all-multicast, since classified unicast delivery would drop transit frames for the emulated fabric. When viona interface version 7 (kernel-side MAC filter table support in st.louis) is available as per https://code.oxide.computer/c/illumos-gate/+/775 (in review), then this is fully lit up.

Every port worker and the guest Tx path serialize on the pipeline mutex. Previously, the guard was held across both the dlpi and virtio egress, meaning that one worker blocked in egress I/O stalled the others. This is especially costly for multicast, where a single input frame fans out to several ports for replication. This change makes the lock scoped to pipeline evaluation only. The egress packets borrow from the input frame rather than the pipeline, so they remain valid after the guard is dropped. Lock acquisition moves from the callers into `process_external_packet` and `process_guest_packet`, which now take the mutex directly. The latter function returns a bool so the guest read loop can stop early when no P4 program is loaded or the lock is poisoned, preserving the prior break semantics.

FelixMcFelix

Thanks Zeeshan; hopefully these comments makes sense, and thanks for the extra tests around the actual handling of control queue messages. I am partly looking at this from the angle of minimising the amount of things we need to do in viona/MAC, and from keeping as many (guest ⨉ test environment) pairs out of promiscuous mode as is doable.

FelixMcFelix · 2026-07-13T17:00:02Z

 libloading = "0.7"
 p9ds = { git = "https://github.com/oxidecomputer/p9fs" }
-softnpu = { git = "https://github.com/oxidecomputer/softnpu" }
+softnpu = { git = "https://github.com/oxidecomputer/softnpu", branch = "zl/multicast" }


Calling this out to reset to main before merge.

FelixMcFelix · 2026-07-13T17:15:54Z

+
+    /// Notification that the virtio portion of the device has been reset.
+    ///
+    /// This fires for guest-initiated status resets as well as full device
+    /// resets, after the generic virtio state and virtqueues have been reset.
+    ///
+    /// Devices should use this to restore any state tied to feature
+    /// negotiation, since the next driver may negotiate a different feature
+    /// set.
+    fn device_reset(&self) {}


I think this is already covered by Lifecycle::reset()?

Not quite as Lifecycle::reset() covers a full device reset, but a guest writing 0 to the virtio status register calls set_status() -> virtio_reset() without invoking Lifecycle::reset(). That path is used for driver unload/reload and must clear the state negotiated by the previous driver.

The existing notification is only queue_change(VqChange::Reset), called once per queue while the virtio state lock is held. device_reset(), on the other hand, provides one device-level callback after the generic state and queues are reset and the lock is released.

FelixMcFelix · 2026-07-20T12:39:48Z

+        // A guest whose multicast table is empty has not demonstrated that
+        // it manages its own filtering. illumos vioif negotiates CTRL_RX and
+        // forwards promiscuity toggles over the control queue, but its
+        // multicast entry point is a noop, so narrowing to classified
+        // delivery after a promiscuity cycle (snoop on/off) would silently
+        // drop its multicast traffic. Unrequested traffic is permitted by


I think this is more likely an upstream illumos/vioif driver bug -- if it's advertising that feature flag, then it should be tracking the MAC addresses it needs and telling the device. I don't think any driver should expect to receive more packets than it asks for.

I also don't know that we can always rely on compliant guests requesting at least one multicast MAC -- I guess it's a reasonable proxy since 224.0.0.1, ff01::1 and ff02::1 are often automatically joined on e.g. Linux, and we don't want to leave existing illumos guests worse off. But I don't know the full diversity of guest behaviours here and if any of them misbehave in the same way as illumos.

Agreed on both counts. Illumos vioif deliberately acknowledges multicast joins without programming the device and relies on hypervisor over-delivery, so this is a compatibility workaround for driver behavior rather than a virtio guarantee. The spec also permits an empty multicast table, so non-empty membership is not a reliable signal.

I'll make the distinction command-based instead, by retaining the all-multicast floor until the driver sends its first accepted MAC_TABLE_SET. After any accepted table update‚ we know the driver is managing the table, thus we can remove the compatibility floor and derive the receive configuration from its requested state. Reset will clear that marker, and migration will preserve it. This is essentially what I said above in #1093 (comment).

Changes aside, @FelixMcFelix, should we file the bug upstream? Is that something you want me to do?

I've opened https://www.illumos.org/issues/18280.

Here are the follow-ups from review of the CTRL_RX/CTRL_MAC receive filtering work: - Withhold classified delivery until the first accepted MAC_TABLE_SET since reset. illumos vioif negotiates CTRL_RX but never programs the multicast table, so the all-multicast lower bound holds until the driver demonstrates that it can manage its own filtering. - We now carry that marker across migration as an optional payload field (multicast_table_managed). Older sources omit it and the target infers from non-empty tables, while older targets ignore it. - Track table installation independently of promiscuity (mac_filters_dirty), so guest promiscuity toggles no longer unplumb and replumb the MAC client's classified flows. Failed installs and clearings stay dirty and are reissued on a later application. - We scope the falcon promiscuity pin to softnpu-attached links rather than forcing every falcon VM promiscuous. - Return `Result<(), NoP4Program>` from softnpu's `process_guest_packet` instead of just a bool. - Write MAC filter entries directly into `vmf_mcast` rather than collecting them into an intermediate vector. - Bump HEADER_CHECK_REF to gerrit patchset 4.

zeeshanlakhani mentioned this pull request Mar 30, 2026

[multicast] wire up softnpu backend to multicast table ops oxidecomputer/dendrite#242

Merged

zeeshanlakhani self-assigned this Apr 17, 2026

zeeshanlakhani force-pushed the zl/multicast branch 3 times, most recently from bad9d65 to 784de5e Compare April 18, 2026 03:36

zeeshanlakhani mentioned this pull request Jul 3, 2026

[multicast] default the multicast feature and wire SoftNPU end-to-end oxidecomputer/dendrite#224

Open

zeeshanlakhani force-pushed the zl/multicast branch from 75518ad to 0de0c5b Compare July 11, 2026 08:07

zeeshanlakhani changed the title ~~[multicast] dep updates~~ [multicast] viona CTRL_RX/CTRL_MAC receive filtering + softnpu egress lock scoping + dep updates Jul 11, 2026

zeeshanlakhani force-pushed the zl/multicast branch from 0de0c5b to a9779dc Compare July 12, 2026 15:51

zeeshanlakhani requested a review from FelixMcFelix July 12, 2026 15:51

zeeshanlakhani marked this pull request as ready for review July 12, 2026 15:51

zeeshanlakhani force-pushed the zl/multicast branch from a9779dc to 95c195a Compare July 12, 2026 16:26

zeeshanlakhani added 3 commits July 12, 2026 18:07

[multicast] dep updates

b0c16f1

Pin softnpu and p4rs to their zl/multicast branches for multicast support in the emulated ASIC, and bump oxide-tokio-rt to 0.1.4.

zeeshanlakhani force-pushed the zl/multicast branch from 95c195a to cd0e261 Compare July 12, 2026 18:23

zeeshanlakhani mentioned this pull request Jul 16, 2026

Multicast e2e oxidecomputer/omicron#9912

Open

FelixMcFelix reviewed Jul 20, 2026

View reviewed changes

zeeshanlakhani requested a review from FelixMcFelix July 22, 2026 02:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[multicast] viona CTRL_RX/CTRL_MAC receive filtering + softnpu egress lock scoping + dep updates#1093

[multicast] viona CTRL_RX/CTRL_MAC receive filtering + softnpu egress lock scoping + dep updates#1093
zeeshanlakhani wants to merge 4 commits into
masterfrom
zl/multicast

zeeshanlakhani commented Mar 30, 2026 •

edited

Loading

Uh oh!

FelixMcFelix left a comment

Uh oh!

FelixMcFelix Jul 13, 2026

Uh oh!

FelixMcFelix Jul 13, 2026

Uh oh!

zeeshanlakhani Jul 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FelixMcFelix Jul 20, 2026

Uh oh!

zeeshanlakhani Jul 20, 2026 •

edited

Loading

Uh oh!

zeeshanlakhani Jul 21, 2026

Uh oh!

FelixMcFelix Jul 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

zeeshanlakhani commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

viona: CTRL_RX and CTRL_MAC table receive filtering

softnpu: release pipeline lock before egress I/O

deps and callouts

Uh oh!

FelixMcFelix left a comment

Choose a reason for hiding this comment

Uh oh!

FelixMcFelix Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

FelixMcFelix Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FelixMcFelix Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

FelixMcFelix Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zeeshanlakhani commented Mar 30, 2026 •

edited

Loading

zeeshanlakhani Jul 20, 2026 •

edited

Loading