Adding possibility of pinning libe3 to specific CPU cores by stefanomaxenti · Pull Request #82 · wineslab/libe3

stefanomaxenti · 2026-09-20T18:57:44Z

Problem

E3Config::io_thread_affinity has existed since the real-time data path landed, but e3_config_t never exposed it. The C API had no way to reach the setting — and the OAI integration is the only C consumer — so libe3's inbound, outbound and report loops always ran wherever the scheduler happened to put them.

Change

The core is now reachable from C, as a pair of fields:

int pin_io_threads;      /* 0 = don't pin (the zero-initialized default) */
int io_thread_affinity;  /* core to pin to, when pin_io_threads != 0 */

Two fields instead of one, because there is no spare value to mean "off". Other optional fields in this struct use -1 or 0 to mean "keep the default", but a CPU core has no such spare value: 0 is a real core, same as any other. So pin_io_threads decides whether to pin at all, and io_thread_affinity says which core. That also keeps the safe behaviour as the default — a zeroed e3_config_t has pin_io_threads at 0 and pins nothing. With a single field, that same zeroed struct would have meant "pin to core 0".

Also: apply_thread_config() no longer swallows failures but returns a message.

A requested core outside the process's cpuset is the expected failure mode on an isolated-core host, and discarding the return value made a missed pin look identical to a working one. Affinity and niceness both stay best-effort, but they now log what happened — a warning naming the core and the error on failure, an info line on success.

Compatibility

e3_config_t grows by two fields, so sizeof(e3_config_t) changes. Downstream consumers must be rebuilt, not just relinked — an old caller would pass a short struct and libe3 would ead pin_io_threads past the end of it. VERSION is bumped to 0.2.1.

Assisted-by: Claude:claude-opus-5

Summary

adding possibility to pin the libe3 threads to CPU cores - useful for guaranteed performance and isolation

Type of change

Linked issue

Closes #

Mandatory test checklist

These mirror what CI (.github/workflows/pr-tests.yml) enforces. All boxes must be ticked before review.

./build_libe3 -c -d build -j $(nproc) -r -t passes (Release build + tests)
./build_libe3 -c -d build -j $(nproc) -g -t passes (Debug build + tests)
cd build && ctest --output-on-failure is clean
MPMC queue benchmark (./build/test_bench_mpmc_queue) shows no regression vs main
VERSION bumped per SemVer if the public API or ABI changed
If public headers under include/ were touched, ./build_libe3 --docs renders without new Doxygen warnings
If new build dependencies were added, they are installed by ./build_libe3 -I (update the script if needed)
If the libe3.pc interface changed, downstream consumers (dApp-openairinterface5g) still link cleanly

CI checklist

CI posts a single CI report comment on this PR once every workflow has finished; it carries the
verdict, a per-workflow table and the benchmark/E2E detail. Confirm against that comment:

The report's verdict is green for the head commit
Unit Tests is green (Debug + Release matrix on ubuntu-latest)
Commit policy is green (trailers + linear history + each commit builds/tests independently)
MPMC Queue Benchmark shows no regression (only runs when include/libe3/mpmc_queue.hpp changes)

Twin-repo coordination

libe3 is paired with dapps and dApp-openairinterface5g. We do not accept patches that break or reduce compatibility with the twin repositories.

This PR does not change the E3 wire protocol or public ABI, OR a paired PR exists in each affected twin repo (link below).

Paired PR(s): TBD but this adds a new feature that can be controlled by the OAI config file. A new PR for that will be published.

Workflow confirmation

My branch is a linear, fast-forward-able descendant of main (rebased if main moved), with no merge commits. (See CONTRIBUTING.md § Pull Request Process.)
Every commit builds and passes tests on its own (atomic, git bisect-safe) with a descriptive message.
I have read and followed CONTRIBUTING.md.

github-actions · 2026-09-20T19:03:23Z

CI report — `f2329e0` — ✅ all checks passed

Workflow	Result	Time	Run
Commit policy	✅ success	1m53s	#110
E2E dApp Integration	✅ success	2m05s	#119
E2E Topologies (multi-dApp / multi-RAN)	✅ success	3m13s	#117
Full-loop Latency Benchmark	✅ success	1m14s	#118
Unit Tests	✅ success	4m40s	#145
latrec portability	⏭️ not triggered (paths filter)	—	—
MPMC Queue Benchmark	⏭️ not triggered (paths filter)	—	—

E2E Topologies (multi-dApp / multi-RAN)

zmq/ipc

✅ 1 RAN - 1 dApp: indications=5
- dapp peer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
- dapp1 peer=t12 ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- dapp2 peer=t12 ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
- dapp peer=t2a ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.2 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- dapp peer=t2b ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

zmq/tcp

✅ 1 RAN - 1 dApp: indications=5
- dapp peer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
- dapp1 peer=default ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- dapp2 peer=default ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
- dapp peer=default ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- dapp peer=off100 ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

E2E dApp Integration

✅ posix/ipc

dApp exit: 0
Indications received: 7

✅ posix/tcp

dApp exit: 0
Indications received: 7

✅ zmq/ipc

dApp exit: 0
Indications received: 7

✅ zmq/tcp

dApp exit: 0
Indications received: 7

Full-loop Latency Benchmark

Full-loop latency

Full-loop latency benchmark (N=1008 after 50 warmup)

All values in microseconds (μs). Link: zmq, transport: ipc, encoding: ASN.1 APER.

#	Description	Tags	mean	p50	p99	max
1	Collect indication data	`RECORD_BEGIN` to `ENCODE_E3SM_BEGIN`	0.12	0.11	0.34	0.50
2	Create & encode indication	`ENCODE_E3SM_BEGIN` to `ENCODE_E3SM_DONE`	0.94	0.88	1.82	2.06
3	Encode E3AP (indication)	`EMIT_ENTER` to `ENQUEUE`, then `DEQUEUE` to `ENCODE_E3AP_DONE`	3.29	3.36	4.54	8.06
4	Queuing (indication)	`ENQUEUE` to `DEQUEUE`	6.73	6.77	16.32	27.14
5	Delivery (indication)	`ENCODE_E3AP_DONE` to `SEND_DONE`	0.33	0.27	1.60	15.86
6	E3 wire (RAN -> dApp)	`SEND_DONE` to `RECV`	52.89	53.42	64.60	85.75
7	Decode E3AP (indication)	`RECV` to `DECODE_E3AP_DONE`	1.70	1.46	5.65	16.26
8	libe3 dispatch (indication)	`DECODE_E3AP_DONE` to `DELIVER_BEGIN`	0.12	0.12	0.19	0.28
9	Decode indication	`DELIVER_BEGIN` to `DECODE_E3SM_DONE`	0.60	0.55	0.88	1.43
10	Process data	`DECODE_E3SM_DONE` to `ENCODE_E3SM_BEGIN`	0.04	0.04	0.05	0.08
11	Create & encode control	`ENCODE_E3SM_BEGIN` to `ENCODE_E3SM_DONE`	0.36	0.37	0.49	0.62
12	Encode E3AP (control)	`EMIT_ENTER` to `ENQUEUE`, then `DEQUEUE` to `ENCODE_E3AP_DONE`	3.82	3.73	5.37	23.71
13	Queuing (control)	`ENQUEUE` to `DEQUEUE`	17.77	19.12	25.96	33.42
14	Delivery (control)	`ENCODE_E3AP_DONE` to `SEND_DONE`	5.32	5.28	7.35	30.27
15	E3 wire (dApp -> RAN)	`SEND_DONE` to `RECV`	50.66	50.46	60.59	70.78
16	Decode E3AP (control)	`RECV` to `DECODE_E3AP_DONE`	2.86	2.88	4.13	27.64
17	libe3 dispatch (control)	`DECODE_E3AP_DONE` to `DECODE_E3SM_BEGIN`	0.29	0.29	0.42	0.65
18	Decode & handle control	`DECODE_E3SM_BEGIN` to `DECODE_E3SM_DONE`	0.42	0.39	0.55	0.81
Total	Total round-trip		148.72	150.50	169.68	191.66

ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.

These numbers are measured inside a GitHub Actions container and should be treated as an upper bound on E3AP's and the library's own latency, not a representative deployment measurement.

Ready to merge (fast-forward only)

A maintainer can land the reviewed commits with:

git fetch origin
git checkout main && git merge --ff-only f2329e060585d0031cd5a81ff87a59c8706e179c && git push origin main

Head: f2329e060585d0031cd5a81ff87a59c8706e179c (branch c-api-io-thread-affinity). If --ff-only fails as non-fast-forward, the branch must be rebased on the latest main.

One comment per PR, rewritten in place once every workflow for f2329e0 finished.

Thecave3

Minor requests, good work overall

Thecave3 · 2026-09-20T19:00:40Z

keep these comments but let's have them trimmed a little bit

Thecave3 · 2026-09-20T19:02:30Z

+                                 << affinity << ": " << std::strerror(rc)
+                                 << " (thread left unpinned)";
+        } else {
+            E3_LOG_INFO(LOG_TAG) << "Pinned " << role << " thread to core " << affinity;


I'd probably use debug to reduce the logs but I am not 100% convinced of my own opinion, how does AERIAL and OAI usually do for these kinds of messages?

I see the value in our logs to have it always shown

It's just a single message at startup.

Thecave3 · 2026-09-20T19:03:24Z

            if (config->io_threads != 0) cfg.io_threads = config->io_threads;
            if (config->log_level >= 0) cfg.log_level = config->log_level;
            if (config->log_path) cfg.log_path = config->log_path;
+            // Gated by the flag, not by a sentinel: core 0 is a real target,


i'd remove the comment and set the same format adopted above unless the size requires a newline for formatting

E3Config has carried io_thread_affinity since the real-time data path landed, but e3_config_t never exposed it, so the C API -- and therefore the OAI integration, its only C consumer -- had no way to reach it. The inbound, outbound and report loops ran wherever the scheduler put them. Expose it as a pair: pin_io_threads gates the setting and io_thread_affinity carries the core. The other optional fields use an in-band sentinel (-1 or 0 meaning "keep the default"), which does not work here because every core index including 0 is a legitimate target; with a flag, a zero-initialized e3_config_t means "do not pin" rather than "pin to core 0". That distinction matters on a host with isolated cores, where core 0 is typically the housekeeping core the RAN most wants to stay off. While here, stop swallowing the outcome in apply_thread_config(). A requested core outside the process's cpuset is the expected failure on such a host, and discarding the return made a missed pin look exactly like a working one. Both the affinity and the niceness call stay best-effort, but now say what happened. Assisted-by: Claude:claude-opus-5

stefanomaxenti requested a review from Thecave3 as a code owner September 20, 2026 18:57

Thecave3 requested changes Sep 20, 2026

View reviewed changes

Thecave3 force-pushed the c-api-io-thread-affinity branch from 8f5946f to f2329e0 Compare September 22, 2026 15:23

Thecave3 merged commit f2329e0 into main Sep 22, 2026
25 checks passed

Thecave3 deleted the c-api-io-thread-affinity branch September 22, 2026 16:30

Navigation Menu

Adding possibility of pinning libe3 to specific CPU cores - #82

Adding possibility of pinning libe3 to specific CPU cores#82
Thecave3 merged 1 commit into
mainfrom
c-api-io-thread-affinity

stefanomaxenti commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026 •

edited

Loading

zmq/ipc

zmq/tcp

✅ posix/ipc

✅ posix/tcp

✅ zmq/ipc

✅ zmq/tcp

Full-loop latency

Full-loop latency benchmark (N=1008 after 50 warmup)

Uh oh!

Thecave3 left a comment

Uh oh!

Thecave3 Sep 20, 2026

Uh oh!

Thecave3 Sep 20, 2026

Uh oh!

stefanomaxenti Sep 22, 2026

Uh oh!

Thecave3 Sep 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

stefanomaxenti commented Sep 20, 2026

Problem

Change

Compatibility

Summary

Type of change

Linked issue

Mandatory test checklist

CI checklist

Twin-repo coordination

Workflow confirmation

Uh oh!

github-actions Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report — f2329e0 — ✅ all checks passed

zmq/ipc

zmq/tcp

✅ posix/ipc

✅ posix/tcp

✅ zmq/ipc

✅ zmq/tcp

Full-loop latency

Full-loop latency benchmark (N=1008 after 50 warmup)

Ready to merge (fast-forward only)

Uh oh!

Thecave3 left a comment

Choose a reason for hiding this comment

Uh oh!

Thecave3 Sep 20, 2026

Choose a reason for hiding this comment

Uh oh!

Thecave3 Sep 20, 2026

Choose a reason for hiding this comment

Uh oh!

stefanomaxenti Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

Thecave3 Sep 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Sep 20, 2026 •

edited

Loading

CI report — `f2329e0` — ✅ all checks passed