A collection of experiments on model optimization, inference acceleration, benchmarking and serving.
| Category | Experiment | Description |
|---|---|---|
| Inference Acceleration | PyTorch → Torch-TensorRT [Notebook] | Compile RoBERTa-base with Torch-TensorRT (FP32/FP16), benchmark latency, and analyze output accuracy. |
| Inference Acceleration | PyTorch → ONNX [Notebook] | Export RoBERTa-base to ONNX and evaluate inference performance and numerical differences. |
| Inference Acceleration | PyTorch → ONNX → TensorRT [Notebook] | Convert a PyTorch model to ONNX and TensorRT, then benchmark speed and output fidelity. |
| Model Optimization | LLM Compressor (GPTQ W4A16) [Notebook] | Apply post-training quantization and compare compressed and full-precision models. |
| LLM Serving | vLLM Inference Server [Notebook] | Deploy a vLLM server and explore continuous batching, KV cache management, prefix caching, and serving metrics. |
| Benchmarking & Evaluation | GuideLLM and LM Evaluation [Notebook] | Benchmark vLLM deployment's latency and throughput using GuideLLM, evaluate model quality with LM Evaluation Harness, and analyze performance–quality trade-offs for deployment decisions. |
( Will grow over time )
- Post-Training Workflows
- Model Optimization & Quantization
- Inference Acceleration
- LLM Serving & Deployment
- Benchmarking & Profiling
- Throughput, Latency & Memory Trade-offs
- This is ongoing work—some things are half-finished
- Code here is for learning and experimentation, not necessarily production-ready
This project is licensed under the MIT License.