Warm Prompts¶

Pre-populate the prefix cache at server startup so the first request an agent sends hits a warm cache instead of paying the full prefill for its multi-kilobyte system prompt.

When to use this¶

Agent workloads — proxies to coding/reasoning assistants, MCP servers, multi-agent orchestrators — always send the same system prompt. Today the first request from a cold server pays the full prefill for that system. On a multi-billion-parameter model that is several seconds of TTFT, landing exactly when a user is waiting for their new agent to respond for the first time.

If you already know your agents' system prompts at deploy time, write them to a JSON file and point --warm-prompts at it. The server runs a max_tokens=1 chat completion for each at startup, the KV state lands in the prefix cache, and the first real request matches via strict-prefix.

Requires --continuous-batching (the prefix cache lives there).

Quick example¶

# Write the agents you care about once
cat > ~/.config/vllm-mlx/agents.json <<'JSON'
[
  [{"role": "system", "content": "You are a code assistant..."}]
]
JSON

# Point the server at it
vllm-mlx serve mlx-community/Qwen3-4B-4bit \
  --continuous-batching \
  --warm-prompts ~/.config/vllm-mlx/agents.json

On start you'll see:

[lifespan] Warm-up done (strict-prefix): 1 completed, 0 skipped,
           1431 prompt tokens in 0.2s

The first real request that shares the warmed system prompt hits the cache with tokens_saved close to the warm-up prompt length.

File format¶

A top-level JSON list. Each entry is itself a list of chat messages — same shape as messages in /v1/chat/completions.

[
  [
    {"role": "system", "content": "You are a code assistant..."}
  ],
  [
    {"role": "system", "content": "You are a senior code reviewer..."}
  ],
  [
    {"role": "system", "content": "You are a planner..."},
    {"role": "user", "content": "hi"},
    {"role": "assistant", "content": "Hello, what are we planning?"}
  ]
]

Single-message system prompts are the common case. Multi-turn histories are supported for scenarios where you want to warm a specific conversation start (few-shot examples, a running assistant persona).

Sizing¶

Warm-up prompts are processed concurrently via asyncio.gather, so N entries fire N concurrent prefills at startup. Each prefill allocates KV cache for its prompt length.

Recommended: 1–3 entries. That covers the hot paths for typical agent deployments (one persona per entry). A very large warm-prompts file on a memory-tight model can exhaust headroom at boot.

If you need to warm dozens of personas, open an issue with your workload and we can add a --warm-prompts-concurrency=N cap.

Benchmarks¶

Setup. M4 Max, 128 GB unified memory. Two separate servers per measurement (cold vs warm), isolated cold start. long prompt set (~2.5k user tokens) prepended with a ~1.7k-token system prompt to match the warm-up prompt. max_tokens=128. bench-serve with --skip-preflight-token-count so the count_prompt_tokens preflight does not pollute the cache.

Model	conc	cold TTFT	warm TTFT	Speedup
Qwen3-0.6B-8bit	1	563 ms	419 ms	1.34x
Qwen3-0.6B-8bit	4	1 723 ms	1 282 ms	1.34x
Qwen3-0.6B-8bit	8	3 708 ms	2 661 ms	1.39x
Llama-3.2-3B-Instruct-4bit	1	1 754 ms	1 060 ms	1.65x
Llama-3.2-3B-Instruct-4bit	4	5 926 ms	3 945 ms	1.50x
Llama-3.2-3B-Instruct-4bit	8	15 161 ms	9 820 ms	1.54x
Qwen3-4B-4bit	1	4 937 ms	2 191 ms	2.25x
Qwen3-4B-4bit	4	12 535 ms	9 623 ms	1.30x
Qwen3-4B-4bit	8	38 148 ms	23 878 ms	1.60x
Qwen3.6-35B-A3B-4bit (MoE/hybrid)	1	2 400 ms	1 603 ms	1.50x
Qwen3.6-35B-A3B-4bit	4	8 735 ms	6 054 ms	1.44x
Qwen3.6-35B-A3B-4bit	8	22 419 ms	14 409 ms	1.56x

All 12 configurations improve. TTFT savings are largest when the prompt-to-total ratio is highest (conc=1, long system prompt) and still meaningful under concurrent load.

Generation tok/s is neutral (within ±5%) for the dense models. Qwen3.6-35B-A3B (MoE) shows a 20–35% decode drop at conc ≥ 4 that appears to be MoE routing interaction with batched scheduling. TTFT savings still dominate end-to-end latency on agent workloads, but note this if your workflow is heavily decode-bound at high concurrency.

How it works¶

The naive warm-up — render the chat template with a placeholder user message and cache the tokens — does not work for hybrid SSM+attention models (Qwen3.5-MoE, Qwen3.6-MoE). Their cache layers include SSM state that cannot be trimmed, so memory_cache.py disables LCP matching. The placeholder user content diverges from real user content, and a tokens-level cached entry is no longer a strict prefix of any real request.

The warmer here renders the chat template twice with two distinct user contents ("__PROBE_A__" and "__PROBE_B__"), finds the character position where the two strings diverge, and truncates the first rendering at that boundary. That truncated string — everything up to the point where user content gets inserted — is what goes to the engine.

Because the engine's real-request path also renders the template with tokenize=False and then lets the tokenizer encode the result, the warm-up's tokens are guaranteed to be a strict prefix of any real request with a matching system and empty chat history. Strict prefix matches work on every cache layer type, including the hybrid paths where LCP is disabled.

Admin¶

Clear the in-memory prefix cache¶

curl -X DELETE http://localhost:8000/v1/cache/prefix

If the server was started with --warm-prompts, the warm-up re-runs in the background after clear. The response returns immediately without waiting for re-warm.

Response:

{"status": "cleared", "rewarm_scheduled": true}

Inspect cache state¶

curl http://localhost:8000/v1/status | jq '.cache'

After startup with warm-prompts you will see entry_count > 0 before the first user request.

Benchmarking your own setup¶

To measure the impact on your model and prompts, use bench-serve:

# Cold: no warm-prompts
vllm-mlx serve MODEL --continuous-batching &
vllm-mlx bench-serve --prompts long --concurrency 1,4 \
  --system-prompt-file my-system.txt --tag cold \
  --output cold.csv --format csv

# Warm: same server config + --warm-prompts
vllm-mlx serve MODEL --continuous-batching \
  --warm-prompts ~/.config/vllm-mlx/agents.json &
vllm-mlx bench-serve --prompts long --concurrency 1,4 \
  --system-prompt-file my-system.txt --tag warm \
  --output warm.csv --format csv

--skip-preflight-token-count is auto-enabled when --system-prompt-file is set, so the count_prompt_tokens preflight does not pollute the cache. Compare cold.csv and warm.csv for your workload.