Self-hosted inference on a single consumer GPU: 14 models on an RTX 3060

14 open-weight instruct models, 1,204 requests, 280k tokens, run entirely on a single RTX 3060 through Marigold's local Docker Compose stack.

Setup

One RTX 3060, run through the local Docker Compose stack described in the project README -- Postgres for the queue, a single worker process, no cloud component anywhere in the path. Each of the 14 models ran 86 sequential requests, split across five prompt groups: a fixed-prefix set (33 requests), a structured-output set (20), and three varying-context sets at short, medium, and long context length (11 each). Every request captured server-side inference time, instantaneous and peak VRAM, and instantaneous and peak GPU power draw directly from the worker process. max_tokens was capped at 256.

Throughput and parameters

Output tok/s vs parameter count. Parameter counts parsed from model names where unambiguous; phi-3.5-mini-instruct is omitted from this chart pending confirmation of its parameter count.

Per-request telemetry

Three views of the same 1,204 requests, in run order, each column a model. Click any model in the legend to hide or show it across all three charts.

The charts contain:

1. Mean power draw (watts) per request as reported by nvml.
2. Peak VRAM (GB) per request -- this includes the activations, KV-cache and static layers.
3. Server-side inference time (seconds) per request.

Observations

Output length determines duration. qwen3-8b and deepseek-r1-distill-qwen-1.5b both produce the longest average completions (251 and 255 tokens against a 256 cap) and are, respectively, the slowest and fourth-slowest model in the set by mean inference time -- despite deepseek-r1-distill-qwen-1.5b being one of the two smallest models in VRAM terms (1.7GB).

Silent failure. cognitivecomputations/dolphin-2.9-llama3-8b hit exactly 256 completion tokens on all 86 requests. This is likely a bug in the stop token setup.

Fastest model meta-llama/llama-3.2-1b-instruct reaches 48.78 tok/s and a 3.18s mean inference time.

Run totals

Across all 1,204 requests: 280,224 tokens (95,517 prompt, 184,707 completion). GPU was active for 2h 17m (the sum of per-request inference time); benchmark wall-time was 6h 1m end to end at the client. The roughly 3h 45m gap between those two figures is queue and poll overhead from the API's async design -- submit, then poll for a result. A further 53m 36s went to model load and switching between the 14 models.

Total electricity used while generating: 226.4 Wh. At the current Ofgem direct-debit cap (24.67p/kWh, April-June 2026), that is 5.6 pence for the entire 1,204-request run.

Conclusions

A single consumer GPU, with no cloud account and no managed infrastructure, ran 14 instruct models through 1,204 requests for five and a half pence in electricity. That is the headline number for anyone weighing self-hosting against an API bill: the marginal cost of generation on hardware you already own is close to zero for this size class of model.

Implementation notes

The raw benchmark harness output is a CSV of one row per request, with per-request power and VRAM sampled directly on the worker. Both data and the conversion script are published alongside the code on GitHub.