Repository navigation
Pytorch gpu benchmarks - #1
Merged
Merged
Conversation
added 28 commits
October 2, 2026 13:58
The Windows venv (1,010 files of pip internals) was tracked because the './quantum_env' pattern never matches: gitignore does not accept a leading './'.
Both runs shared one Keras model, so the 'full' QNN was warm-started from the 500-example run and effectively trained for 6 epochs. Add build_qnn() so every run gets fresh weights, and evaluate against the [-1, 1] hinge labels the model was trained on.
- The model outputs logits (from_logits=True), but metrics=['accuracy'] resolves to binary accuracy at threshold 0.5. Logits in (0, 0.5) were counted as negatives. Use BinaryAccuracy(threshold=0.0). - Importing model.qnn built the TFQ circuit just to train an MLP, and the imported 'model' was immediately shadowed. - Rename Fair_CNN_model -> fair_nn_model: it has no convolutions.
- Plots titled 'Testing Accuracy/Loss' showed hand-copied *training* values. Read val_* curves from the History objects instead. - Hinge loss (QNN) and cross-entropy (NN) are on different scales; plot them on separate axes rather than as a head-to-head comparison. - Give each chart its own figure; barplots were drawn onto the axes of the following line plot. - Follow the fair_cnn -> fair_nn rename.
data_preprocess and qnn drew matplotlib figures and called IPython's display() at import time, so every training script needed IPython and produced stray plots. Move the visual checks under __main__ guards.
It listed stdlib modules (importlib, collections), the deprecated pkg_resources, cirq twice, and omitted numpy/matplotlib. Pin the stack verified to install and run on Python 3.11.
Reimplements the TFQ preprocessing (3-vs-6 filter, bilinear 4x4 downsample, contradiction removal, binarization) in NumPy/PyTorch. The download is checksum-verified, and a test pins the sample counts recorded in the original notebook so the two pipelines stay comparable.
- Each gate is one kernel over the whole batch: the (B, 2^n) state is reshaped as a view and contracted along the target axes. Diagonal gates (Z, CZ, ZZ, RZ) take a broadcast-multiply path with no matmul. - Gradients by autograd (memory grows with depth), adjoint differentiation (constant memory, ~2x gate applications) and the parameter-shift rule (forward-only, hardware compatible). - Tests check every gate unitary, random circuits and the 17-qubit QNN circuit against cirq.Simulator, all gradient methods against finite differences, and that adjoint saved memory is independent of depth.
Keeps the original code and notebook runnable for reproducibility and documents each bug fixed since the paper, together with the results of a verified end-to-end run of the fixed pipeline.
- QNN wraps the 32-parameter Farhi circuit with selectable gradient method and precision; FairMLP is the 37-parameter baseline. - LookupTable (majority label per binary pattern) is the Bayes-optimal classifier for this input encoding, so results can be read against the accuracy ceiling. - The training loop keeps metrics on device and syncs once per epoch, records throughput, and writes config + environment with each run.
The simulator is memory-bound (each gate sweeps the full B*2^n state for a few FLOPs per amplitude). Runs of diagonal gates now collapse into one batch-independent phase vector, and adjacent dense gates into one k-qubit unitary applied as a single GEMM. On the 17-qubit QNN with k=5 this takes 35 gate passes down to 6 and roughly halves forward and backward time on CPU. Fusion plugs into every gradient method's forward pass; tests check fused results and gradients against the unfused simulator.
- Harness: warmup, CUDA-event timing on GPU (host timers only measure launch overhead for async kernels), p50/p95, throughput, peak device memory, and autograd saved-tensor bytes so memory comparisons also work on CPU. Every result file records git SHA, torch/CUDA versions and device properties. - Suites: qubit scaling, batch size, fusion width, gradient method, complex64 vs complex128, eager vs torch.compile vs CUDA graphs, framework comparison (PennyLane, Cirq; outputs cross-checked), and a single-kernel study of complex vs split-real tensors under Inductor. - compare: matches cases between two result files and exits non-zero on median-latency regressions beyond a tolerance.
Re-runs the paper's protocol with independent models and fixed seeds, reports mean, std and range per model, and puts the lookup-table ceiling for the 4x4 binary encoding in the same table.
Each 2x2 patch is angle-encoded on 4 qubits, passed through a fixed random circuit and measured, giving a (4, 14, 14) feature map per image. Because the circuit is fixed and acts on 4 qubits, the fused path folds it into one 16x16 unitary, so a patch costs one GEMM row instead of one kernel per gate. The dataset pipeline ships uint8 images to the device (4x fewer bytes than float32), overlaps the next chunk's host-to-device copy with the current chunk's compute on a side CUDA stream (pinned memory, record_stream for allocator safety), stores float16 features and caches them on disk. Tests cross-check one patch end to end against Cirq.
… int8 - Same CNN head on three 2x2/stride-2 patch encoders: the fixed quantum filter, a fixed random classical conv (the control), and a learned conv, on full 10-class MNIST. - AMP picks bf16 on Ampere+ and fp16 with GradScaler on Turing (GTX 1650, T4); gradient accumulation skips DDP's all-reduce on all but the last micro-batch (no_sync); torchrun launches DDP over NCCL, or Gloo on CPU so the distributed path runs without GPUs. - Pinned-memory DataLoader with non-blocking copies; uint8 images are scaled on the device. - Post-training dynamic int8 quantization of the Linear layers, with accuracy, size and CPU latency measured against fp32.
Compares gate-by-gate simulation with the fused 16x16 unitary, sweeps chunk size, and on CUDA toggles overlap of host-to-device copies with compute.
Concurrent requests are queued and run as one batch (up to MAX_BATCH, or whatever arrives within MAX_WAIT_MS) on a single inference thread, so the event loop keeps accepting requests while kernels run. A failing batch fails only its own requests. /metrics exposes request counts and latency and batch-size quantiles in Prometheus text format. loadtest spawns one server per MAX_BATCH setting and drives it with closed-loop clients, reporting throughput and p50/p95/p99 latency.
- Dockerfile: CUDA PyTorch base, dependency layer cached separately from source, non-root user, gcc for torch.compile at runtime. - k8s: GPU benchmark Job (tolerates the GPU taint, optional GPU-type pinning for hardware comparisons), 2-GPU DDP Job with an in-memory /dev/shm for DataLoader workers, and a serving Deployment with readiness/liveness probes. Validated with kubeconform -strict. - CI: ruff, tests on Python 3.10 and 3.12 with CPU torch, quick benchmark smoke run, and a Docker build on main. - Drop the license field from pyproject: choosing one is up to the authors.
Runs in the legacy Python 3.11 environment and writes frameworks/grad results that line up with the PyTorch suites, so the original implementation is part of the framework comparison.
Figures use one y-axis per panel, direct labels plus a legend, and a fixed categorical color order. Result files are meant to be committed, so they now identify CPUs by model name and no longer record the machine's hostname.
QNN 88.9% +- 2.1, fair MLP 91.4% +- 0.3, against a 91.4% Bayes ceiling for the binarized 4x4 encoding. The MLP saturates the ceiling; the QNN does not, reversing the paper's single-run headline.
All nine suites on an otherwise idle CPU, plus the original TFQ (qsim) implementation measured on the same machine.
Colors follow each series by name across figures, the device moves to a subtitle so titles fit, and adjoint/param-shift memory (identical by construction) is drawn as one labelled line instead of two stacked.
Under load on a 4-core CPU, torch's intra-op threads competed with the asyncio event loop for every core, so requests reached the batcher late and batches stayed small. QNNBENCH_TORCH_THREADS (and loadtest --torch-threads) leaves cores for HTTP handling; at 64 clients it took batched throughput from 72 to 91 req/s.
Hybrid, 5 epochs on full MNIST: quantum filter 98.80%, random classical filter 98.93%, learned filter 98.97% (single seed); int8 keeps accuracy at 2.4x smaller weights but is slower at batch 1. Serving, 16 clients: micro-batching 178 vs 75 req/s and p50 88 vs 212 ms.
Leads with the performance and accuracy findings, including the optimizations that did not help and why, documents the design and measurement method, and records how the corrected results differ from the paper.
With python-version set, setup-uv already creates .venv, and current uv exits with code 2 rather than overwrite an existing environment, so every job that ran 'uv venv' failed at install. Use activate-environment instead, and install CPU torch explicitly from the PyTorch index so the resolver cannot pick the CUDA wheel from PyPI. Bump actions/checkout to v5 (Node 24).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.