Repository navigation
[Benchmark] 2x RTX 3060 (layer split), Xeon E5-2697 v2 without AVX2, Swift 1.5 IQ2_XS (SC117 abliterated), 0.1.45 #3389
Description
Hardware
- GPU: 2x NVIDIA GeForce RTX 3060, 12 GiB each, used as a two-card layer split (
--layer-split auto, both cards) - CPU: Intel Xeon E5-2697 v2 @ 2.70 GHz (dual socket, 12 cores each = 24 cores / 48 threads). No AVX2, no AVX-512 — the engine is the locally compiled "avx"-floor build for older CPUs
- RAM: 127.9 GiB | OS: Windows 11 (10.0.26200) | driver 610.88 | CUDA 13.3
- Strata: project 0.1.45, commit 477ac13; engine 0.1.45 (source build for this CPU)
Model and configuration
- swift-1.5-iq2_xs — the SC117 abliterated (uncensored) Swift 1.5 build of Qwen3.8-Flash-Next, IQ2_XS
- Context 128000, q4_0 KV, MTP draft (
--spec 4), vision on, parallel 1 - Tuned engine args (unchanged from the install):
--kv-resident 32768,--prefill auto:32768,--conversation-cache-mib 8192,--pipeline-windows 2,STRATA_ARENA_PIN_GIB=26 - Measured with
tools/strata_bench.py(3 runs per test, medians with ranges); thinking off, temp 0
Results (median, range)
| Test | Value |
|---|---|
| short chat decode | 54.0 tok/s (46.7–59.9) |
| short chat TTFT | 0.85 s (0.76–0.97) |
| 4K prompt, fresh | 569.9 tok/s (564.9–579.7), TTFT 7.11 s |
| 16K prompt, fresh | 820.0 tok/s (814.8–824.1), TTFT 19.65 s |
| agent decode (follow-up turns) | 50.2 tok/s (40.9–63.1) |
| agent conversation-cache hit | 68.8 % |
Full per-turn agent table, prompts and engine log excerpt are in the README this README is pasted from (result folder bench/results-v045-baseline/2026-10-19-community-rtx-3060-swift-1-5-iq2-xs/).
Why this setup matters
- No AVX2 CPU: the older-CPU "avx" engine floor still reaches ~54 tok/s decode and ~570/820 tok/s on 4K/16K prompts with a two-card layer split — useful data point for other AVX2-less servers.
- Abliterated model: this is the SC117 refusal-reduced (uncensored) Swift 1.5 build, so prompt/decode numbers are for the abliterated weights, not the stock Swift.
- It is the same GPU pair as Community benchmark on NVIDIA GeForce RTX 3060 + NVIDIA GeForce RTX 3060 #2608, but on different CPU/OS/model/engine, so not a duplicate.
Activity
github-actions commentedon Oct 20, 2026
Thanks for the issue. To keep the tracker readable, every item needs an area prefix in the title and the parts of the issue form filled in. This one is missing:
Edit the issue (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment
/reopen.Details: CONTRIBUTING.md
[-]Community benchmark: 2x RTX 3060 (layer split), Xeon E5-2697 v2 without AVX2, Swift 1.5 IQ2_XS (SC117 abliterated), 0.1.45[/-][+][Benchmark] 2x RTX 3060 (layer split), Xeon E5-2697 v2 without AVX2, Swift 1.5 IQ2_XS (SC117 abliterated), 0.1.45[/+]github-actions commentedon Oct 20, 2026
Thanks, this follows the format now.
sudoeste commentedon Oct 20, 2026
Can you make a A/B test against this? https://github.com/eddoursul/Strata/tree/custom
The setup won't give a valid cfg, you have to adapt the cfg in examples.
Btw, on my tests IQ3_S runs faster (???) than iq2_xs on dual 3060. Don't understand why
CaptRauti commentedon Oct 20, 2026
Thanks @sudoeste! Two replies:
On IQ3_S vs IQ2_XS on dual 3060: that matches a hunch we had on this machine too. We've been running IQ2_XS (SC117 abliterated) but we do have an IQ3_S parked on a second drive. I'd be curious to A/B it once more now that we're on 0.1.45 — if it really is faster (or close) on this exact dual-3060 budget, the better quality is free. I'll add a same-day run and post the numbers here.
On the
custombranch A/B: I looked at your fork. It sits on top of a much newer codebase (0.2.0-era, lots of docs/GUI/packaging/feed commits relative to the 0.1.45 engine we run) rather than a small perf delta, so for us an A/B means a full local engine rebuild (our Xeon E5-2697 v2 is AVX-1 only, so we maintain a local AVX-floor build) plus adapting the config from your examples.Before I spend that time: what is concretely performance-relevant in
customfor a two-card 12 GB non-AVX2 setup? Which engine flags/patches are the intended win vs the 0.1.45 baseline we run? If the gain is in the NVFP4/IQ3_s path or prefill for small-VRAM cards specifically, I'll absolutely do a same-day A/B; if it's mostly new-app/GUI, I'll skip the rebuild and just note it.My full setup and baseline numbers are in the issue body. Happy to run whatever specific comparison is most useful.
sudoeste commentedon Oct 20, 2026
Hi! It's not my fork, It's just the fork I use.
From my understanding it uses a different way of splitting the cards, other than layer split.
From my tests I go from 850pp to 1250 and median 40 to 60tg.
The fork does the not have support for parallel requests
sudoeste commentedon Oct 20, 2026
fork on my 2x3060 iqs_3:
Results (median, range over the runs):
chat decode tok/s 50.10 (47.30 - 55.50, n=9)
chat first token (s) 0.58 (0.55 - 0.71, n=9)
4K prompt tok/s 1124.20 (1111.40 - 1124.70, n=3)
4K first token (s) 3.63 (3.63 - 3.69, n=3)
16K prompt tok/s 1490.00 (1467.80 - 1499.50, n=3)
16K first token (s) 10.84 (10.77 - 11.02, n=3)
agent cache hit % 69.42 (69.41 - 69.46, n=3)
agent re-read tokens/session 4.00 (3.00 - 5.00, n=3)
agent turn first token (s) 4.20 (2.53 - 4.74, n=15)
agent decode tok/s 46.00 (12.90 - 59.80, n=15)
mainline on my 2x3060 iqs_3:
Results (median, range over the runs):
chat decode tok/s 41.00 (36.90 - 50.40, n=9)
chat first token (s) 0.81 (0.71 - 0.91, n=9)
4K prompt tok/s 611.20 (609.90 - 612.20, n=3)
4K first token (s) 6.61 (6.60 - 6.62, n=3)
16K prompt tok/s 842.20 (833.20 - 869.90, n=3)
16K first token (s) 19.10 (18.50 - 19.31, n=3)
agent cache hit % 69.54 (69.35 - 69.55, n=3)
agent re-read tokens/session 3.00 (2.00 - 74.00, n=3)
agent turn first token (s) 5.82 (5.13 - 6.41, n=15)
agent decode tok/s 42.70 (35.20 - 57.10, n=15)
from my experience the fork is faster when generating longer responses.
btw, sorry for poluting you issue