Local inference on a Mac is straightforward until several requests overlap. Then time to first token, memory growth, and admission control become serving problems rather than model-execution problems. vllm-metal brings vLLM’s scheduler, paged KV cache, and OpenAI-compatible server to Apple Silicon, with MLX and Metal handling execution.

v0.4.0 adds batched MTP, GGUF and hybrid-model support, and automatic prefill acceleration on M5. On SiliconBench’s agent split, vllm-metal keeps TTFT flatter as concurrency rises while serving from a fixed memory budget.

How vllm-metal fits into vLLM

vllm-metal plugs into upstream vLLM. vLLM provides the V1 scheduler, paged KV block management, chunked prefill, sampling, and the OpenAI-compatible frontend with streaming and tool-call parsing. mlx_lm provides the model implementations; MLX executes them.

At the model level, vllm-metal reuses mlx_lm’s weight loading, RMSNorm, linear, MoE, and MLP layers unchanged. Those layers process each token independently, so they run on a packed token axis without knowing request boundaries. Attention does need those boundaries, so vllm-metal replaces stock attention with a paged varlen Metal kernel. Most of the plugin’s model-specific code therefore sits in one layer.

Architecture overview: clients speak the OpenAI API to upstream vLLM's frontend and V1 scheduler, which hand the vllm-metal model runner a packed step plus block tables; the runner reuses mlx_lm's token-wise layers and adds custom Metal paths for paged varlen attention, MTP, and M5 NAX prefill, all executing through MLX and Metal on Apple Silicon unified memory

Start an OpenAI-compatible server

Install vllm-metal into its own virtual environment and activate it:

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
source ~/.venv-vllm-metal/bin/activate

The installer adds the plugin, vLLM core, and their dependencies to ~/.venv-vllm-metal.

Then launch a model:

# --gpu-memory-utilization caps vLLM's share of unified memory; see below.
vllm serve Qwen/Qwen3.5-0.8B --gpu-memory-utilization 0.5

# 64 GB Macs: the 27B hybrid
# vllm serve mlx-community/Qwen3.8-27B-8bit --gpu-memory-utilization 0.7

# Speculative decoding: Gemma 4 with its MTP assistant
# vllm serve google/gemma-4-E4B-it --gpu-memory-utilization 0.5 \
#   --max-model-len 16384 --no-async-scheduling \
#   --speculative-config '{"method":"mtp","model":"mlx-community/gemma-4-E4B-it-assistant-bf16","num_speculative_tokens":1}'

More models: model matrix. Speculative options: speculative decoding guide.

The server speaks the OpenAI API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3.5-0.8B",
       "messages": [{"role": "user", "content": "Say hi"}]}'

Anything that takes an OpenAI-compatible base URL can point at http://localhost:8000/v1, coding agents included; the vLLM docs cover Claude Code and Codex setup.

Set a predictable memory budget

vllm-metal reserves its KV cache at startup and serves every request from that fixed pool. If you are used to runtimes whose memory footprint moves with load, this fixed pool is the main mental-model change. --gpu-memory-utilization sets the share of the Mac’s GPU memory budget used for serving.

Apple Silicon has no separate VRAM. GPU allocations come from the same unified memory used by macOS, your browser, and your editor. --gpu-memory-utilization is a serving budget; process memory can exceed it. As a starting point, 0.5 keeps a laptop usable while it serves; a dedicated machine can go higher.

The scheduler tracks the available KV pages, packs requests against that budget, and queues requests that do not fit. Under a burst, queue depth grows while the KV pool stays fixed.

Before sizing the KV pool, vllm-metal accounts for model weights and temporary buffers, keeping memory predictable as batch shapes change (PR #268).

Concurrent serving under agent load

Coding agents create concurrency by fanning out tool calls. Each call carries a few thousand tokens of context and returns a short reply, with several in flight at once. Every round trip pays TTFT before work can continue; end-to-end request latency sets the duration of the turn.

Qwen3.8-27B

We measured the agent split with SiliconBench, our benchmark suite for LLM inference engines on Apple Silicon. The workload contains 100 requests averaging 4.6K input and 70 output tokens, run closed loop at concurrency 1, 2, and 4 against Qwen3.8-27B in 8-bit on a 64 GB M5 Pro. oMLX appears twice because its default mode spills KV cache to SSD without eviction, which no other engine here does; we report it both with that offload and in its bounded in-memory mode.

SiliconBench agent split on Qwen3.8-27B: TTFT, end-to-end request latency, and output token throughput versus concurrency for llama.cpp, vllm-metal, and oMLX with and without SSD offload

  • At concurrency 1, the three bounded engines sit within 0.7 s on roughly 10-second TTFTs; oMLX’s SSD-offload mode is fastest at 7.8 s.
  • As requests overlap, vllm-metal’s TTFT rises from 10.1 s to 14.7 s while completing all 100 requests at every level. llama.cpp also completes every request, about one second behind at concurrency 4. oMLX’s SSD mode reaches roughly 27 s, while bounded oMLX rejects 37 of 100 requests.
  • At concurrency 4, mean request latency is 52.7 s for vllm-metal, 47.9 s for oMLX’s SSD-offload arm, and 57.6 s for llama.cpp. Chunked prefill shares each engine step between new prefills and active decodes.

Gemma 4 E4B

Gemma 4 E4B is small enough to sweep through concurrency 16 on the same machine and agent split.

SiliconBench agent split on Gemma 4 E4B: TTFT, end-to-end request latency, and output token throughput versus concurrency for llama.cpp, vllm-metal with and without the MTP drafter, and oMLX in its default SSD-offload mode

At concurrency 16, vllm-metal averages 1.9 s TTFT and 63.6 output tokens per second. oMLX averages 14.7 s TTFT and 50.4 tokens per second; llama.cpp averages 28.3 s TTFT and 40.2 tokens per second. The dashed MTP arm raises vllm-metal to 71.6 output tokens per second and is analyzed below.

llama.cpp uses its default --parallel 4 configuration; --parallel 16 is not uniformly better (sensitivity results). At concurrency 16 it decodes four streams and queues the other twelve, producing 36.1 s mean request latency. oMLX ran only its default configuration, with SSD KV offload on; there is no bounded-memory arm for this model.

Packed queries and paged KV

mlx_lm’s stock attention runs on a contiguous cache of shape [B, H, T, D], with every sequence padded to the length of the longest and prefill and decode handled as separate phases. MLX’s scaled_dot_product_attention accepts no varlen argument.

vllm-metal flattens each step onto a single token axis, with cu_seqlens marking request boundaries. This is the packed-query layout used by vLLM’s unified Triton kernel on NVIDIA. KV remains in fixed-size pages located by per-request block tables, so admitted requests can grow without reshaping a padded cache. One kernel launch covers the variable-length query segments that the V1 scheduler packed into the step.

The packed axis applies to the whole forward pass, so MLP and MoE blocks compute only real tokens. A padded engine pushes padding rows through the network.

A padded rectangle versus vllm-metal's packed varlen step for a 30,000 + 5,000 + 10 token batch, with KV read from paged storage

A padded rectangle spends memory and attention time on the longest sequence in the batch whether or not the others need it. A concurrent mix that fits comfortably on a large machine can push a smaller one into macOS memory compression and slow down without an explicit error. Sizing one paged pool up front removes that failure mode.

Only vllm-metal pairs a packed query axis with paged KV storage among the Apple Silicon serving stacks we audited:

Engine Encoding Query axes KV
mlx_lm padding [B,T_max] contiguous
oMLX padding [B,T_max] contiguous
llama.cpp mask [total_q] fixed cells
vllm-metal cu_seqlens [total_q] paged

The Metal kernel ports vLLM’s unified Triton kernel, described in The Anatomy of a Triton Attention Kernel, to Apple GPUs, down to the binary search each threadgroup runs over cu_seqlens to find which request owns its query token.

What v0.4.0 adds

Batched MTP under concurrent load

Speculative decoding makes the query batch ragged. In one scheduler step, one request may contribute a single decode token, another its last token plus a request-specific number of drafts, and another a prefill chunk. A [B,T_max] query tensor must either pad those rows to a common width or split them across forwards. The audited versions split the work:

  • mlx_lm 0.32.0: With a draft model loaded, the server bypasses its batch generator and routes speculative requests through the single-request path.
  • oMLX 0.6.3rc2: Native MTP engages only for a single active sequence by default. Its opt-in multirow mode runs one backbone forward per row.

vllm-metal concatenates those windows on its varlen [total_q] axis and verifies them in one target-model forward. MTP drafting and verification stay inside the continuous-batching path. The dashed blue line in the Gemma 4 figure measures this path. Relative to the vllm-metal baseline:

Concurrency Wall Output tok/s TTFT avg Acceptance
1 +1% −3% +5% 73.4%
8 −21% +23% +15% 73.5%
16 −12% +13% +16% 73.3%

Single-stream, MTP is a wash on this model: the drafter’s cost roughly cancels the accepted tokens. At concurrency 8, it raises output throughput by 23% and cuts mean request latency from 13.7 s to 10.9 s; acceptance stays near 73% across the sweep. Today the Metal MTP path is limited to Gemma 4 and requires --no-async-scheduling; the quickstart above includes both. In this comparison, TTFT rises by about 15% under load. MTP is opt-in through --speculative-config, so prefill-dominated deployments leave it off.

Faster prefill on M5

On M5 Macs, vllm-metal automatically uses the NAX kernel for compatible prefill batches; pre-M5 Macs keep using the existing path.

NAX prefill kernel A/B benchmark: TTFT, throughput, and TPOT with the tensor units on and off

NAX cuts mean TTFT by 41% on the prefill-heavy split and 26% on the standard split, while total throughput rises 33% and 8%. It also lowers TPOT by 25% and 7% because faster chunked prefill returns time to active decode streams.

Reusing conversation history on hybrid models

Multi-turn agents resend most of their growing conversation on every turn. Prefix caching lets the next turn reuse blocks computed for earlier turns instead of prefilling the full history again.

For Qwen3.5-style hybrid models, vllm-metal supports vLLM’s align mode. It checkpoints GDN recurrent state at the same block boundaries as attention KV, so both parts of the model can resume from the same cached prefix. A small custom Metal scatter kernel updates only the GDN state rows that changed, in place, without copying the whole shared state pool (PR #634).

On an M5 Pro repeated-prefix workload, this made Qwen3.5-0.8B finish the 100-request run about one-fifth sooner; unrelated prompts stayed within run-to-run noise. vLLM 0.28 enables align-mode prefix caching by default for supported hybrid models. The Metal path remains experimental and cannot yet be combined with speculative decoding.

Models and serving features

v0.4.0 also ships:

  • LoRA adapters, structured outputs, and three speculative-decoding methods: Gemma 4 MTP, separate draft models, and prompt-lookup n-grams.
  • GGUF checkpoints, including Hugging Face config sources for local GGUF weights.
  • Hybrid-attention models: the Qwen3.5, Qwen3.6, Qwen3.8, and Qwen3-Next families alternate standard attention with gated-delta-net linear attention; v0.4.0 serves mlx-community/Qwen3.8-27B-8bit on a single Mac.
  • Pipeline parallelism across multiple Macs over the MLX ring backend.
  • Experimental vision-language models, text embeddings and reranking, and speech-to-text.

The supported-model matrix and feature guides are in the vllm-metal documentation.

The same stack on DGX Spark

NVIDIA’s DGX Spark is another unified-memory target for the same vLLM stack. It exposes a shared CPU/GPU memory pool and runs the same V1 scheduler, chunked prefill, and paged KV management. The execution layer changes from vllm-metal, MLX, and Metal to upstream vLLM and CUDA.

The two machines balance memory and compute differently:

  Apple M5 Pro DGX Spark (GB10)
Unified memory 64 GB LPDDR5X-9600 128 GB LPDDR5X-8533
Memory bandwidth 307 GB/s 273 GB/s
Nominal scalar shader width (rough) ≈2,560 lanes (20 cores × ≈128) 6,144 CUDA cores (48 SMs × 128)
Serving stack vllm-metal (MLX + Metal) upstream vLLM (CUDA)

Nominal scalar width gives a rough architectural comparison. It does not measure equivalent FLOPS. Apple publishes the 20-core GPU but not the M5 Pro’s lane count or absolute GPU throughput; the Apple value estimates 128 lanes per core from prior Apple GPU designs. NVIDIA publishes 6,144 CUDA cores. Clock rates, instruction issue, matrix accelerators, and kernel efficiency differ.

Neural Accelerators and Tensor Cores are omitted because Apple does not publish a comparable throughput figure.

SiliconBench agent split on Gemma 4 E4B and Qwen3.8-27B, vllm-metal on an M5 Pro against upstream vLLM on a DGX Spark: TTFT, end-to-end request latency, and output token throughput versus concurrency

On Gemma 4 E4B, the Mac leads at concurrency 1, with 21.1 output tokens per second to the Spark’s 15.7. At concurrency 16, the Spark leads 225.1 to 63.6. On Qwen3.8-27B, where the Mac uses an 8-bit MLX conversion and the Spark uses Qwen’s FP8 checkpoint, throughput at concurrency 4 is 5.3 on the Mac and 24.0 on the Spark, with TTFT at 14.7 s and 0.6 s, respectively.

At concurrency 16, Gemma 4 E4B averages 5.5 s per request on the Spark against 21.7 s on the Mac; on the 27B at concurrency 4, the gap is 11.2 s against 52.7 s.

Both expose the same OpenAI-compatible interface and V1 scheduling model, so the SiliconBench workload runs unchanged while the execution backend and hardware set the performance ceiling.

Reproducing the benchmarks

The cross-engine serving benchmarks use the SiliconBench agent split: 100 prompts averaging 4.6K input and 70 output tokens, run closed loop at fixed concurrency on an Apple M5 Pro with 64 GB running macOS 26.6. The NAX A/B instead uses two Sonnet configurations: a prefill-heavy split with 2,048 input and 32 output tokens, and a standard split with 1,024 input and 128 output tokens. Both run 100 prompts at request rate 10 and concurrency 32.

The DGX Spark comparison reuses the agent split and SiliconBench client unchanged on a GB10 box with 121 GB of usable unified memory.

Stats cover completed requests; an empty response counts as failed. The benchmark code and per-engine configurations live in the SiliconBench repo. The MTP arms ran vllm-metal 0.3.0.dev20260821152549 with the serve command from the quickstart.

Serving benchmark reproduction settings
  • llama.cpp: -ngl 99 --parallel 4 -c 49152. The context is divided across slots, giving 12,288 tokens per slot; the agent split’s longest prompt is 8.7K tokens.
  • vllm-metal (27B): VLLM_METAL_MEMORY_FRACTION=0.7 with --max-model-len 16384. Prefix caching was enabled explicitly with --enable-prefix-caching. These runs predate vLLM 0.28, which enables it by default for hybrid models. At the default fraction of 0.5, the available KV cache held 36,408 tokens and preemption began at concurrency 4.
  • oMLX: --paged-ssd-cache-dir <fresh-empty-dir> and --hot-cache-max-size 0, with a restart and a new directory before each concurrency level. Its default 100 GB prefix cache persists in ~/.omlx/cache, while CLI values persist in ~/.omlx/settings.json. The model directory contained only the target checkpoint because oMLX auto-discovers every entry and returns them in ASCII order. At concurrency 4, its bounded-memory admission guard rejected 37 of 100 requests, so the main chart reports a failure count rather than survivor-only latency.
  • vllm-metal MTP: --no-async-scheduling with "num_speculative_tokens":1. Without the scheduling flag, server health and /v1/models succeed, but inference returns HTTP 500. Values above 1 are ignored rather than rejected, so they do not test a wider speculation window.
  • DGX Spark: upstream vLLM 0.27.2rc1.dev568+gf25c580af built from source against torch 2.13.0+cu130 and flashinfer 0.6.17, on GB10 with driver 580.159.03, CUDA 13.0, Ubuntu 24.04 and kernel 6.17. It uses the same SiliconBench client, agent split, and 100 prompts, with --max-model-len 16384 and prefix caching on by default. vLLM selects its FLASH_ATTN backend on this GPU; FlashInfer is installed but unused. The 27B arm serves Qwen/Qwen3.8-27B-FP8, a different checkpoint from the Mac’s 8-bit MLX conversion.
  • --gpu-memory-utilization on the Spark: the 27B needs an explicit 0.7, matching the fraction the Mac uses for the same model. GB10 shares one 121 GB pool between the GPU and the operating system, so vLLM’s default 0.9 reserves around 109 GB of it. Small models absorb that (Qwen3.5-0.8B took a 102 GiB KV cache and served fine), but 28.5 GiB of 27B weights on top of it exhausted the machine, the driver returned NV_ERR_NO_MEMORY, and the desktop session died. At 0.7 the KV pool holds 619,557 tokens, 37x the 16K budget of a single request, so nothing is preempted at concurrency 4.
llama.cpp server-slot sensitivity

The main figures use llama.cpp’s default four server slots. A 16-slot sensitivity run improves output throughput at concurrency 16 but regresses at concurrency 8:

Split Concurrency --parallel 4 --parallel 16 Change
Chat 1 22.4 tok/s 24.2 tok/s +8%
Chat 8 77.0 tok/s 42.7 tok/s −45%
Chat 16 81.5 tok/s 104.7 tok/s +28%
Agent 1 18.3 tok/s 19.0 tok/s +4%
Agent 8 44.2 tok/s 26.7 tok/s −40%
Agent 16 40.2 tok/s 49.7 tok/s +24%

Acknowledgments

vllm-metal builds on MLX and mlx_lm from Apple’s MLX team, mlx-vlm for the vision-language paths, and vLLM’s engine and hardware-plugin interface. Thanks to the upstream vLLM maintainers for review and support along the way, and to everyone who filed issues and shared benchmarks against the v0.2 and v0.3 releases.