Agent Inference Dojo

Local LLM inference stack for learning. Replicates the patterns from gersonrpq/coding-agent-inference-infra (H100 + SGLang) but running locally on Apple Silicon.

Serves Qwen3-14B Q4_K_M through Ollama, with LiteLLM as an OpenAI-compatible proxy, Prometheus + Grafana for observability, and the Pi coding agent as a consumer.

---

Stack diagram

pi coding agent
  -> LiteLLM :4000  (proxy + metrics)
    -> Ollama /v1 :11434  (inference server)
      -> Qwen3-14B Q4_K_M  (model)

Prometheus scrapes /metrics/ from LiteLLM every 15s. Grafana visualizes the metrics.

---

Prerequisites

Requirement	Installation
Ollama	ollama.com — install and verify with ollama --version
Qwen3-14B model	ollama pull qwen3:14b (downloads ~9 GB, Q4_K_M quantization)
Docker	docker.com — required for Prometheus and Grafana
uv	docs.astral.sh/uv — Python package manager
LiteLLM	Installed with make install (via uv)
Pi coding agent	github.com/badlogic/pi-mono — clone and build

---

Quick start

1. Install LiteLLM

make install

This runs uv tool install 'litellm[proxy]' to install LiteLLM as a global tool.

2. Start Ollama

ollama serve

Verify that the model is available:

ollama list | grep qwen3

3. Start LiteLLM

# Start the proxy in the foreground (useful for viewing logs)
make start

This starts LiteLLM on port 4000 pointing to Ollama on port 11434.

4. Start monitoring

make monitoring-up

Starts Prometheus (:9090) and Grafana (:3000) with Docker Compose.

5. Open Grafana

Navigate to http://localhost:3000. Default credentials: admin / admin.

The "LiteLLM" dashboard will be automatically available thanks to provisioning.

6. Test with Pi

# Quick test: one-line response
make pi-test

# Full interactive session
make pi

7. Verify the stack

make test

Runs a test request against LiteLLM and displays the response.

---

Project structure

Path	Description
README.md	This file — project documentation
Makefile	Convenience targets for operating the stack
docker-compose.yaml	Prometheus and Grafana service definitions
config/litellm.yaml	LiteLLM configuration: models, parameters, routing to Ollama
prometheus/prometheus.yml	Prometheus configuration: scraping targets
grafana/provisioning/datasources/prometheus.yml	Automatic provisioning of the Prometheus datasource in Grafana
grafana/provisioning/dashboards/default.yml	Automatic provisioning of dashboards in Grafana
grafana/dashboards/litellm.json	Grafana dashboard with LiteLLM metrics
pi/agent/models.json	Model configuration for Pi: LiteLLM endpoint + compatibility flags
pi/agent/settings.json	General Pi configuration: default model
scripts/start.sh	Startup script for the full stack
scripts/stop.sh	Stack shutdown script
scripts/test.sh	Verification script: test request against LiteLLM

---

Lessons learned

Ollama has two distinct APIs

Ollama exposes two endpoints:

• /api/chat (native API): returns finish_reason: "stop" even when the model generates tool calls. Agents cannot detect that there are pending tools.
• /v1/chat/completions (OpenAI-compatible API): correctly returns finish_reason: "tool_calls". Always use /v1 when the consumer is an agent.

LiteLLM requires a special flag without a database

When LiteLLM runs without a database (simple proxy mode), it requires the environment variable:

LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY=true

Without this, the proxy rejects requests.

Qwen3's thinking mode generates thousands of extra tokens

Qwen3 has a "thinking" mode enabled by default that generates thousands of internal tokens before the actual response. This drastically increases latency and consumption.

Solution: use the /no_think tag in the system prompt or pass think: false in the model parameters to disable internal reasoning.

drop_params in LiteLLM for OpenAI compatibility

Some OpenAI clients send parameters that Ollama does not support (such as store). Setting drop_params: true in litellm.yaml makes LiteLLM silently ignore them instead of failing.

Pi needs supportsFinishReason: false

The Pi agent by default expects a reliable finish_reason from the model. With Ollama, this does not always work correctly. The supportsFinishReason: false flag in models.json makes Pi not depend on this field.

Q4_K_M quantization: 14B in 9 GB

Q4_K_M quantization allows loading a 14B parameter model in only ~9 GB of memory. On an Apple M5 Pro, this yields 29.6 tok/s generation — sufficient for interactive use with a coding agent.

---

Metrics available in Grafana

The "LiteLLM" dashboard includes the following panels:

Panel	Description
Total Requests	Total number of processed requests
Input Tokens	Input tokens consumed
Output Tokens	Output tokens generated
Failed Requests	Failed requests
Request Rate	Requests per second rate
Latency p50 / p95 / p99	Response latency percentiles
Token Rate	Tokens generated per second rate
LLM API Latency	LLM API latency (Ollama)

---

Reference

This project locally replicates the patterns from:

• gersonrpq/coding-agent-inference-infra — inference infrastructure for a coding agent on NVIDIA H100 with SGLang, LiteLLM, K3s, Prometheus, and Grafana.

The corresponding page in the wiki: [[coding-agent-inference-infra]].