[nuk3s]

What this changes

On Linux, /metrics reports no process memory at all, so you can't see the server's own host memory grow. On a GB10 that matters, because the CPU and GPU share one memory pool. Host memory a rank adds counts against the same limit as the weights and the KV pool.

This PR reads /proc/self/status at scrape time and adds five gauges to /metrics and /v1/metrics:

Gauge	Source
tensorfold:process_resident_bytes	VmRSS
tensorfold:process_resident_anon_bytes	RssAnon: heap, stacks and private maps
tensorfold:process_resident_file_bytes	RssFile: the binary, libraries and mapped files
tensorfold:process_resident_shmem_bytes	RssShmem: System V shared memory, tmpfs maps and shared anonymous maps
tensorfold:process_resident_peak_bytes	VmHWM: the kernel's lifetime peak

What's in it:

• zig/src/server/process_resident.zig (new): reads the file into a 4 KiB stack buffer, no allocation.
• zig/src/server/metrics.zig: residentGauges writes the gauges after processGauges, in the live and the parked scrape.
• docs/memory-diagnostics.md: lists the new gauges.

The macOS footprint gauges don't change.

Why

I run DeepSeek-V4.1-Flash at TP=2 on two GB10s. Rank 0 and rank 1 are the two engine processes, one per GB10. Twice, a memory watchdog outside the engine killed rank 0, 29 and 40 minutes after ready, and nothing told me which part of its memory grew. With this reader in my fork build it was right there: RssShmem rose 512 MiB and RssAnon 420 MiB in the 12 minutes after ready. The RssShmem jump lines up with NCCL's pinned buffers for its RDMA connections.

Receipt

No inference path changes, so exactness and speed are n/a. The server tests go from 796 to 803, the metrics.zig module alone from 11/11 to 18/18, and the goldens don't move. On a real scrape each gauge was within 0.2 MiB of /proc/<pid>/status, and on a GB10 the /metrics values matched an outside sampler.

Full test details and what I didn't cover

More on the change:

• A line that's missing or doesn't parse leaves its gauge out, never zero, the same way the footprint gauges are left out off macOS.
• process_resident.zig sits next to process_memory.zig, which keeps the macOS footprint reader.
• process_footprint_bytes and process_footprint_peak_bytes stay macOS only. The resident set has no device or driver memory, so the help text and the doc say it isn't a physical footprint (CUDA: retain pinned peaks and scoped process memory receipts #865).
• Replies, /health, /memory and the macOS output don't change.

More on why:

• CONTRIBUTING.md asks for memory receipts with the peak process footprint and the device and pinned allocations. On Linux a CUDA receipt only gets device_memory_bytes and pinned_memory_bytes (Report the CUDA backend's pinned host memory in /metrics #541).
• tools/memory_receipts.py reads VmRSS and VmHWM, but only for a child process it starts, and without the split. The growth only shows up in /proc.
• My setup: 1M context, 4 lanes. The DeepSeek-V4.1 CUDA path on main is still experimental (DeepSeek V4.1: the lane backend — one stream per slot on the lane core #827), so this ran on bertholomus/TensorFold deepseek-v41-zig at 629c524a.
• In the first run, MemAvailable on rank 0's machine went from 4.8 GiB at ready to 2.5 GiB at the kill. ps showed rank 0's RSS at 4.0 and 4.3 GiB at the two kills, but nothing showed the RSS at ready.
• The RssShmem jump happened inside one 30 s sample, at the first KV transfer between the ranks, when NCCL connected new queue pairs (its RDMA connections). GPUDirect RDMA was off, so NCCL's buffers were pinned host memory.
• A later run connected those queue pairs before ready. There the same 512 MiB was already in the first sample after the load and didn't move after ready. So I'm fairly sure the jump is those buffers.

Environment: main at 8c36359d, and this branch at 462e50de. x86_64 Linux (Fedora 43, kernel 7.2), Zig 0.17.0, no GPU. Also aarch64-linux-musl under qemu-aarch64-static.

Exactness: n/a. No inference path changes. The reader only runs on a scrape.

Decode speed and prompt speed: n/a, same reason.

Tests run. Each command's result on main and on this branch:

Command	Before (8c36359d)	After (462e50de)
zig build test-server-cpu --summary all	5/5 steps; 801/801 passed (server 796, json 5)	5/5 steps; server 803/803 passed, json 5/5 (cached, unchanged)
zig build test-golden --summary all	322/335 equal, 13 known, 0 unexpected; 26/26	322/335 equal, 13 known, 0 unexpected; 26/26
metrics.zig module alone (zig test with engine_api and lanes)	11/11	18/18
the same, aarch64-linux-musl under qemu		18/18
the same, aarch64-macos, x86_64-macos and x86_64-windows-gnu, --test-no-exec		builds
python3 zig/tests/server/test_memory_http.py on the CPU fake server	6/6	6/6
python3 tools/zig/lean_check.py, then lean_new.py	428 problems	428 problems, 0 new

• zig fmt --check and git diff --check are clean.
• 7 new tests. In process_resident.zig: status lines parse to bytes; a kernel with 64 KiB pages still reports kB; a missing line, another unit, a bad number or a u64 overflow is unknown, never zero; a repeated line keeps its first value; this process's own status reads and parses. In metrics.zig: the gauges carry only what the kernel reports and no footprint; on Linux a live and a parked render carry them and no process_footprint_bytes; off Linux neither render has them.
• The tests catch wrong changes: VmRSS sent as process_footprint_bytes fails 2 of 18 module tests, dropping the process_resident_bytes line fails 2 of 18, and a parked scrape without the gauges fails 1 of 18. A parser that takes any unit fails 1 of 5, and so does one that keeps the last repeated line.
• Real Linux scrape: the CPU fake server's /metrics and /v1/metrics have the five gauges and no process_footprint_bytes. Each value was within 0.2 MiB of the server's /proc/<pid>/status, read right after the scrape.
• GB10: the same parser and reader (with other field names) in the fork build above. A sampler outside the engine read rank 0's /proc/<pid>/status every 30 s, and the /metrics row is a scrape in between. Values in MiB:

UTC	Source	VmRSS	RssAnon	RssFile	RssShmem	VmHWM
16:05:20	/proc/<pid>/status	3,821	1,684	238	1,898	5,207
16:05:28	/metrics	3,819.4	1,682.8	238.0	1,898.6	5,207.7
16:05:50	/proc/<pid>/status	3,818	1,682	238	1,898	5,207

Not run, and why:

• Metal and macOS: no Mac here. The macOS gauges and their code don't change, and the module builds for macOS.
• CUDA on main: I didn't serve a model on a GPU with this branch. The GB10 numbers come from the fork build.
• Windows: builds only. The reader returns null off Linux.
• Scrape cost under load: not measured. Each scrape opens and reads one procfs file.
• Rank 1 of a TP pair serves no /metrics, so these gauges don't cover its memory.
• Driver memory is in neither the resident set nor device_memory_bytes, so these gauges can't show growth there.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. No inference path changes.
• No precision traded for speed. No bits change.
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. n/a: no GPU kernel changes.
• Prompt processing is no slower than the last release. No prompt path changes.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thanks, @nuk3s. This is well scoped, and the TP=2 GB10 investigation in the description makes the case for it clearly.

• The reader maps exactly VmRSS, RssAnon, RssFile, RssShmem and VmHWM. It requires the kernel's kB unit,
  checks for overflow, and leaves a gauge out rather than report a false zero
  (zig/src/server/process_resident.zig:14, :38).
• It reads into a bounded stack buffer, handles procfs reporting size 0, and returns null off Linux before opening
  anything (process_resident.zig:47-53).
• Live and parked scrapes both carry the gauges (zig/src/server/metrics.zig:126, :183). Their names are distinct
  from the macOS footprint gauges. The help text and docs/memory-diagnostics.md:46 say plainly that the resident set
  isn't a physical footprint.

Against 8c36359d5, fmt, lean (no new findings) and git diff --check are clean. The metrics module passes 18/18 and
builds for aarch64-linux-musl and x86_64-windows-gnu. We fed one fixed procfs sample through both versions: main omits
the five gauges, and this branch emits them in both live and parked scrapes. The server and golden steps show no new
failures. We didn't run a live Linux scrape ourselves, so your Linux and GB10 receipts cover that part.

This will land shortly after the 1.1.7 release. Thank you!

[ashhart]

Landed on main as 49e8b84, with a changelog line. Thank you!

[nuk3s]

Thank you for merging it so quickly!

[ashhart]

This is in TensorFold 1.1.8, released today. Thank you, @nuk3s!